duplicut
A fast, memory-optimized C tool to remove duplicates from massive wordlists while preserving order, designed for password cracking.
What is duplicut?
Duplicut is a high-performance, memory-optimized command-line tool written in C that removes duplicate lines from massive wordlists without sorting them. It solves the specific problem in password cracking where wordlist order must be preserved to keep the most probable passwords at the front for efficient cracking, while still handling files larger than available RAM.
Security researchers, penetration testers, and password cracking enthusiasts who work with large, combined wordlists and need efficient deduplication without losing the strategic order of passwords.
Developers choose Duplicut because it uniquely combines order preservation with the ability to process wordlists exceeding system memory, using optimized C code and multithreading for speed—addressing a gap left by general-purpose deduplication tools.
Overview
Remove duplicates from MASSIVE wordlist, without sorting it (for dictionary-based password cracking)
Use Cases
Best For
- Deduplicating combined password wordlists for hashcat or John the Ripper
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
