Bpe

SentencePieceC++

An unsupervised text tokenizer and detokenizer for neural network-based text generation systems with subword units.

tokenizersRust

Fast, state-of-the-art tokenizers for training and tokenization, optimized for both research and production.

tiktoken-rsRust

A Rust implementation of OpenAI's tiktoken tokenizer for working with GPT models and token counting.

formErlang

A declarative document and form system for business applications, generating web forms from Erlang specifications.

formsErlang

A declarative document and form system for business workflows, generating web applications from Erlang specifications.