Tokenization
Showing 10 of 46 projects
A Ruby toolset for accurate fuzzy document searching using tuned TF-IDF with hybrid tokenization.
A lightweight compression algorithm for Lua source code using Huffman coding and LZ77 on language tokens.
A Common Lisp DSL for string parsing and tokenization inspired by the REXX PARSE construct.
A full tokenization framework for ACTUS-compliant debt instruments on the Algorand blockchain.
A JRuby wrapper for Apache OpenNLP that provides natural language processing tools like tokenization, POS tagging, and entity extraction.
A Common Lisp lexical analyzer for shell-like syntaxes, ported from Python's shlex module.
A VS Code extension for viewing large datasets (JSONL/Parquet/CSV) instantly without crashes, with 16 production LLM tokenizers for accurate token counting.
A TypeScript library providing text vectorization, image processing, and ML metrics utilities for machine learning workflows.
A Ruby wrapper for basic NLP tools including sentence detection, tokenization, and POS tagging using OpenNLP and Berkeley Parser.
A Go library for programmatically analyzing entities and their sentiment in non-fictional texts using dependency parsing.
Related Tags
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.