decancer
A library that removes common Unicode confusables, homoglyphs, and diacritics from strings to normalize text.
What is decancer?
decancer is a library that removes common Unicode confusables, homoglyphs, and diacritics from strings. It solves the problem of text obfuscation used in spam, evasion, or visual confusion by normalizing input to a clean, readable form. The library is built for performance and accuracy, supporting multiple programming languages.
Developers building applications that require text sanitization, such as chat platforms, content moderation systems, or security tools that need to handle user-generated input safely.
Unlike other text filtering libraries, decancer is unicode bidirectional-aware and handles a wide range of confusables efficiently. Its Rust core ensures high speed, and its multi-language bindings make it accessible across different tech stacks.
Overview
A library that removes common unicode confusables/homoglyphs from strings.
Use Cases
Best For
- Filtering homoglyph attacks in usernames or domains
- Cleaning Zalgo text from user inputs in forums or chats
- Normalizing leetspeak in content moderation pipelines
- Sanitizing strings for database storage to prevent encoding issues
- Improving search accuracy by removing diacritics and confusables
- Securing applications against Unicode-based evasion techniques
Not Ideal For
- Applications that must preserve original Unicode characters for linguistic or cultural accuracy, such as multilingual text processing where diacritics are essential.
- Simple string manipulation tasks without security concerns, where basic trim or case functions would suffice without the overhead.
- Environments with strict dependency constraints that cannot accommodate native code bindings or Rust toolchains.
- Projects needing context-aware text normalization beyond character-level, like semantic analysis or natural language understanding.
Pros & Cons
Pros
Core is written in Rust using binary search, ensuring fast processing for filtering over 222,557 Unicode codepoints, as highlighted in the README.
Correctly interprets right-to-left characters as rendered, unlike other packages, improving accuracy in security and moderation contexts.
Filters homoglyphs, diacritics, leetspeak, Zalgo text, and emojis, covering a wide range of evasion techniques.
Available in Rust, JavaScript, Java, C/C++, Go, and Python, making it accessible across diverse tech stacks for integration.
Highly configurable behavior allows tailoring to specific needs, as emphasized in the philosophy and examples.
Cons
Installation for Go requires Rust and elevated permissions, and C/C++ involves platform-specific downloads, adding deployment complexity.
Python bindings are listed as unofficial, which may lead to inconsistent updates or limited maintenance compared to core languages.
Output is a CuredString object that shouldn't be coerced to regular strings, complicating integration with existing string processing code.
Aggressive filtering of many codepoints might inadvertently remove legitimate characters in niche use cases, despite customizability.
Frequently Asked Questions
Related Projects
ripgrep-allrga: ripgrep, but also search in PDFs, E-Books, Office documents, zip, tar.gz, etc.
KreuzbergPolyglot document intelligence with a Rust core: extract text, metadata, images, tables, and structured data from 106 formats across 140 file extensions, plus code intelligence for 371 languages. Fifteen bindings, with CLI, REST API, and MCP server.
grexA command-line tool and Rust library with Python bindings for generating regular expressions from user-provided test cases
MelodyMelody is a language that compiles to regular expressions and aims to be more readable and maintainable
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.