Bpe

SentencePieceC++

An unsupervised text tokenizer and detokenizer for neural network-based text generation systems with subword units.

tokenizersRust

Fast, state-of-the-art tokenizers for training and tokenization, optimized for both research and production.

tiktoken-rsRust

A Rust implementation of OpenAI's tiktoken tokenizer for working with GPT models and token counting.