Tokenization
Showing 36 of 46 projects
A Python library for Chinese text segmentation, offering multiple modes, custom dictionaries, and keyword extraction.
A comprehensive Python library for natural language processing, providing modules, datasets, and tutorials for NLP research and development.
A lightweight JavaScript library for natural language processing that transforms text into structured data with a modest, pragmatic approach.
Fast, state-of-the-art tokenizers for training and tokenization, optimized for both research and production.
A comprehensive Node.js library offering a wide range of natural language processing facilities.
A Python NLP library from Stanford for tokenization, sentence segmentation, NER, and dependency parsing across 60+ languages.
A Go library for efficient multilingual text segmentation and NLP, supporting English, Chinese, Japanese, and more.
A spaCy pipeline and models specifically designed for processing scientific and biomedical documents.
A fast, straightforward, reliable tool for performing massive, automated code refactoring using custom Python patterns.
A self-hosted, GDPR-compliant Go-based vault for secure tokenization and storage of PII, PHI, PCI, and KYC records.
A self-hosted, GDPR-compliant Go tool for secure tokenization and encrypted storage of PII, PHI, PCI, and KYC records.
Open-source supply chain security scanner that automatically detects vulnerabilities like Log4Shell in dependencies and notifies via GitHub pull requests.
A C# parser combinator library with high-quality error reporting and token-driven parsing.
A curated collection of Ruby libraries, tools, and resources for Natural Language Processing (NLP).
A curated list of awesome resources, libraries, and tools for natural language processing (NLP) in Ruby.
A self-contained Japanese morphological analyzer written in pure Go, tokenizing text into words and analyzing parts of speech.
An R package for the quantitative analysis of textual data, providing comprehensive tools for natural language processing and text management.
A .NET wrapper for Stanford CoreNLP providing natural language processing capabilities including tokenization, parsing, and named entity recognition.
A comprehensive Natural Language Processing (NLP) library for the Crystal programming language.
A Ruby natural language processor for tokenizing and analyzing text with flexible filtering and custom regex support.
A React Native library for integrating Braintree v.zero SDK to accept credit card and PayPal payments.
A React Native bridge for integrating Google Pay into Android apps to accept payments.
An iOS SDK that tokenizes card payments and provides customizable payment UI components for Checkout.com's payment infrastructure.
A Python library providing German language support for TextBlob, enabling NLP tasks like tokenization, POS tagging, and sentiment analysis.
An open-source .NET library that adds efficient full-text search capabilities to the ZoneTree storage engine.
A Julia package providing high-performance, configurable tokenizers and sentence splitters for natural language processing.
A multilingual Ruby gem for splitting strings into tokens with extensive language support and configurable options.
Ruby bindings to the OpenNLP Java toolkit for natural language processing tasks like tokenization, POS tagging, and named entity recognition.
A Ruby port of the NLTK Punkt algorithm for unsupervised, language-independent sentence boundary detection.
A rule-based Unicode tokenizer that separates words from punctuation and splits sentences for NLP preprocessing.
An Elixir natural language processor for tokenization, counting, and string similarity analysis.
A fast and accurate rule-based sentence segmentation tool for Ruby that uses placeholder replacement.
A state-based lexer for Elixir that uses macros to generate reusable lexers with regular expression rules.
A Ruby gem for customizable text tokenization, useful for web crawling and natural language processing.
A Rust crate for generating n-grams from token sequences with Unicode padding support.
A Haxe library for linguistical analysis and natural language processing with tokenization, stemming, classification, and dictionary features.
Related Tags
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.