jieba
A Python library for Chinese text segmentation, offering multiple modes, custom dictionaries, and keyword extraction.
What is jieba?
Jieba is a Python library for Chinese text segmentation, which splits Chinese text into individual words. It solves the problem of word boundary detection in Chinese, where spaces are not used to separate words, making it essential for text analysis, search indexing, and NLP tasks.
Developers and researchers working with Chinese text processing, including those in natural language processing, search engine development, data mining, and academic research.
Developers choose Jieba for its high accuracy, multiple segmentation modes, ease of use, and extensive features like custom dictionaries and keyword extraction, all under an MIT license.
Overview
结巴中文分词
Use Cases
Best For
- Building search engines that require Chinese text indexing
- Performing text analysis on Chinese corpora for research
- Extracting keywords from Chinese documents using TF-IDF or TextRank
- Adding part-of-speech tagging to Chinese NLP pipelines
- Processing large volumes of Chinese text with parallel segmentation
- Integrating Chinese segmentation into Python-based applications
Not Ideal For
- Applications requiring real-time, streaming text segmentation with sub-millisecond latency
- Teams integrated with modern deep learning frameworks like TensorFlow or PyTorch seeking end-to-end neural segmentation models
- Projects running on Windows that need parallel processing for speed optimization
- Environments where multilingual text segmentation beyond Chinese is a primary requirement
Pros & Cons
Pros
Offers four modes—precise, full, search engine, and paddle—allowing developers to trade off accuracy and speed for different use cases, as shown in the code examples for text analysis and search indexing.
Supports custom dictionaries and dynamic word frequency adjustment via methods like add_word and suggest_freq, enabling domain-specific tuning to improve segmentation accuracy, detailed in the user dictionary section.
Includes keyword extraction using TF-IDF and TextRank algorithms, plus part-of-speech tagging compatible with ictclas standards, reducing the need for additional libraries in text processing pipelines.
Implements parallel processing for faster segmentation on large texts (though not on Windows) and efficient prefix dictionary scanning, achieving speeds up to 1.5 MB/s in full mode per the benchmark.
Cons
Parallel processing is not supported on Windows, which significantly hampers performance gains for teams using that OS for large-scale text analysis, as noted in the README.
The paddle mode relies on an old version of PaddlePaddle (v1.6.1), which may conflict with newer installations and lacks updates, limiting its usefulness for cutting-edge neural segmentation tasks.
The README presents Chinese documentation first with English below, which can inconvenience non-Chinese speakers despite translations, potentially slowing onboarding for international teams.
Frequently Asked Questions
Related Projects
HuggingFace Transformers🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
spacy💫 Industrial-strength Natural Language Processing (NLP) in Python
HaystackOpen-source AI orchestration framework for building context-engineered, production-ready LLM applications. Design modular pipelines and agent workflows with explicit control over retrieval, routing, memory, and generation. Built for scalable agents, RAG, multimodal applications, semantic search, and conversational systems.
Rasa💬 Open source machine learning framework to automate text- and voice-based conversations: NLU, dialogue management, connect to Slack, Facebook, and more - Create chatbots and voice assistants
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.