Tokenization
Showing 31 of 31 projects
A Python library for Chinese text segmentation, offering multiple modes, custom dictionaries, and keyword extraction.
A comprehensive Python library for natural language processing, providing modules, datasets, and tutorials for NLP research and development.
A lightweight JavaScript library for natural language processing that transforms text into structured data with a modest, pragmatic approach.
A comprehensive Node.js library offering a wide range of natural language processing facilities.
Fast, state-of-the-art tokenizers for training and tokenization, optimized for both research and production.
A Python NLP library from Stanford for tokenization, sentence segmentation, NER, and dependency parsing across 60+ languages.
A Go library for efficient multilingual text segmentation and NLP, supporting English, Chinese, Japanese, and more.
A spaCy pipeline and models specifically designed for processing scientific and biomedical documents.
A fast, straightforward, reliable tool for performing massive, automated code refactoring using custom Python patterns.
Open-source supply chain security scanner that automatically detects vulnerabilities like Log4Shell in dependencies and notifies via GitHub pull requests.
A self-hosted, GDPR-compliant Go tool for secure tokenization and encrypted storage of PII, PHI, PCI, and KYC records.
A self-hosted, GDPR-compliant Go-based vault for secure tokenization and storage of PII, PHI, PCI, and KYC records.
A C# parser combinator library with high-quality error reporting and token-driven parsing.
A curated collection of Ruby libraries, tools, and resources for Natural Language Processing (NLP).
A curated list of awesome resources, libraries, and tools for natural language processing (NLP) in Ruby.
A self-contained Japanese morphological analyzer written in pure Go, tokenizing text into words and analyzing parts of speech.
An R package for the quantitative analysis of textual data, providing comprehensive tools for natural language processing and text management.
A .NET wrapper for Stanford CoreNLP providing natural language processing capabilities including tokenization, parsing, and named entity recognition.
A comprehensive Natural Language Processing (NLP) library for the Crystal programming language.
A Ruby natural language processor for tokenizing and analyzing text with flexible filtering and custom regex support.
A React Native library for integrating Braintree v.zero SDK to accept credit card and PayPal payments.
A React Native bridge for integrating Google Pay into Android apps to accept payments.
An iOS SDK that tokenizes card payments and provides customizable payment UI components for Checkout.com's payment infrastructure.
A Python library providing German language support for TextBlob, enabling NLP tasks like tokenization, POS tagging, and sentiment analysis.
A Julia package providing high-performance, configurable tokenizers and sentence splitters for natural language processing.
An open-source .NET library that adds efficient full-text search capabilities to the ZoneTree storage engine.
A multilingual Ruby gem for splitting strings into tokens with extensive language support and configurable options.
Ruby bindings to the OpenNLP Java toolkit for natural language processing tasks like tokenization, POS tagging, and named entity recognition.
A Ruby port of the NLTK Punkt algorithm for unsupervised, language-independent sentence boundary detection.
A rule-based Unicode tokenizer that separates words from punctuation and splits sentences for NLP preprocessing.
An Elixir natural language processor for tokenization, counting, and string similarity analysis.
Related Tags
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.