pragmatic_tokenizer

MITRuby

A multilingual Ruby gem for splitting strings into tokens with extensive language support and configurable options.

93 stars11 forks0 contributors

Overview

A multilingual tokenizer to split a string into tokens

Community-curated · Updated weekly · 100% open source

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

1 year ago

CreatedSince 2016

Auto-fetched 23 hours ago

Pragmatic Segmenter is a rule-based sentence boundary detection gem that works out-of-the-box across many languages.

Ruby port of the NLTK Punkt sentence segmentation algorithm