Text Preprocessing
4 projects
Showing 4 of 4 projects
SentencePieceC++
An unsupervised text tokenizer and detokenizer for neural network-based text generation systems with subword units.
#machine-translation#neural-machine-translation#neural-networks
Stars11.8k
Forks1.3k
Last commit2 days ago
tokenizersRust
Fast, state-of-the-art tokenizers for training and tokenization, optimized for both research and production.
#nlp-library#natural-language-understanding#unigram
Stars10.7k
Forks1.1k
Last commit4 days ago
trafilaturaPython
A Python library and CLI tool for web crawling, scraping, and extracting main text, metadata, and comments from web pages.
#text-extraction#readability#article-extractor
Stars5.8k
Forks359
Last commit7 months ago
textacyPython
A Python NLP library built on spaCy for text preprocessing, feature extraction, and analysis tasks.
#nlp-library#computational-linguistics#spacy
Stars2.2k
Forks249
Last commit2 years ago
Related Tags
Community-curated · Updated weekly · 100% open source
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.