sentences
A multilingual command-line sentence tokenizer written in Go, ported from NLTK's Punkt system.
What is sentences?
Sentences is a Go library and command-line tool for sentence tokenization—splitting text into individual sentences. It solves the problem of accurately identifying sentence boundaries across multiple languages, especially in cases involving abbreviations or ambiguous punctuation. The tool is based on the Punkt unsupervised learning algorithm, originally from NLTK.
Developers and researchers working with text processing, NLP pipelines, or multilingual applications who need reliable sentence segmentation without heavy dependencies.
It offers a fast, dependency-free implementation of a proven algorithm with support for 13 languages, making it a lightweight yet accurate alternative to heavier NLP libraries.
Overview
A multilingual command line sentence tokenizer in Golang
Use Cases
Best For
- Preprocessing text for NLP tasks like named entity recognition or sentiment analysis
- Building multilingual content analysis tools or chatbots
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
