Tokenizer
4 projects
Showing 4 of 4 projects
SentencePieceC++
An unsupervised text tokenizer and detokenizer for neural network-based text generation systems with subword units.
#machine-translation#neural-machine-translation#neural-networks
Stars11.8k
Forks1.3k
Last commit2 days ago
JsmnC
A minimalistic, single-header JSON tokenizer/parser in C for resource-limited and embedded systems.
#parsing#c-library#json-string
Stars4.1k
Forks803
Last commit1 year ago
ChevrotainTypeScript
A blazing fast and feature-rich parser building toolkit for JavaScript, supporting LL(K) and LL(*) grammars.
#parsing#llk-grammars#compiler-construction
Stars2.8k
Forks218
Last commit3 days ago
html5everRust
A high-performance, browser-grade HTML5 parser written in Rust, developed as part of the Servo project.
#servo#browser-engine#web-standards
Stars2.6k
Forks271
Last commit4 days ago
Community-curated · Updated weekly · 100% open source
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.