python-ucto

Cythonv0.6.10

32 stars5 forks0 contributors

Overview

This is a Python binding to the tokenizer Ucto. Tokenisation is one of the first step in almost any Natural Language Processing task, yet it is not always as trivial a task as it appears to be. This binding makes the power of the ucto tokeniser available to Python. Ucto itself is regular-expression based, extensible, and advanced tokeniser written in C++ (http://ilk.uvt.nl/ucto).

Quick Stats

Stars32

Forks5

Contributors0

Open Issues

Related Projects

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

Submit a project Star on GitHub

Last commit5 months ago

CreatedSince 2014

Included in

Machine Learning72.2k

Auto-fetched 18 hours ago

HuggingFace Transformers

🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

Stars162,903

Forks34,009

Last commit17 hours ago

jieba

结巴中文分词

Stars35,079

Forks6,693

Last commit1 year ago

spacy

💫 Industrial-strength Natural Language Processing (NLP) in Python

Stars33,768

Forks4,699

Last commit2 months ago

Haystack

Open-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search, and conversational systems.

Stars25,993

Forks2,946

Last commit19 hours ago