textoken

MITRuby1.2.1

A Ruby gem for customizable text tokenization, useful for web crawling and natural language processing.

31 stars3 forks0 contributors

Overview

Simple and customizable text tokenization gem.

Community-curated · Updated weekly · 100% open source

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

4 years ago

CreatedSince 2015

Auto-fetched 20 hours ago

Pragmatic Segmenter is a rule-based sentence boundary detection gem that works out-of-the-box across many languages.

A multilingual tokenizer to split a string into tokens

Ruby port of the NLTK Punkt sentence segmentation algorithm

Accurate Bayesian sentence tokenizer in Ruby.