How do I use Duplicut to prepare a wordlist for hashcat?

Compile Duplicut, then run it on your combined wordlist with options like --lowercase to normalize case and --o to output a clean file. This preserves order, which hashcat benefits from for efficient cracking.

Is Duplicut faster than 'sort -u' for removing duplicates?

For massive files where order must be preserved, Duplicut is faster because it avoids sorting. However, for small files or when sorting is acceptable, 'sort -u' is simpler and may suffice.

How to compile Duplicut on Windows?

You'll need a C compiler like GCC via MinGW or use WSL. Clone the repository and run 'make', but Windows support isn't officially documented and might require troubleshooting.

Can Duplicut handle binary files or UTF-8 text?

No, Duplicut is designed for ASCII printable text, as indicated by the --printable filter. It may not correctly process binary data or multi-byte encodings like UTF-8.

How to filter out long lines while deduplicating with Duplicut?

Use the --line-max-size option to set a maximum character limit. Lines exceeding this will be filtered out during the deduplication process, as shown in the help menu.

What's the best way to install Duplicut on Kali Linux?

Since Duplicut is packaged in BlackArch, you can install it via the BlackArch repository if enabled, or compile from source by cloning the GitHub repo and running 'make'.

Duplicut or uniq for deduplicating password lists?

Duplicut is better for large, order-sensitive wordlists because it preserves order and handles massive files. Uniq requires sorted input and may struggle with memory on huge files.

Open-Awesome

duplicut

GPL-3.0C++v2.4

A fast, memory-optimized C tool to remove duplicates from massive wordlists while preserving order, designed for password cracking.

GitHub

968 stars96 forks0 contributors

What is duplicut?

Duplicut is a high-performance, memory-optimized command-line tool written in C that removes duplicate lines from massive wordlists without sorting them. It solves the specific problem in password cracking where wordlist order must be preserved to keep the most probable passwords at the front for efficient cracking, while still handling files larger than available RAM.

Target Audience

Security researchers, penetration testers, and password cracking enthusiasts who work with large, combined wordlists and need efficient deduplication without losing the strategic order of passwords.

Value Proposition

Developers choose Duplicut because it uniquely combines order preservation with the ability to process wordlists exceeding system memory, using optimized C code and multithreading for speed—addressing a gap left by general-purpose deduplication tools.

Overview

Remove duplicates from MASSIVE wordlist, without sorting it (for dictionary-based password cracking)

Use Cases

Best For

Deduplicating combined password wordlists for hashcat or John the Ripper
Processing multi-gigabyte wordlists on systems with limited RAM
Maintaining password probability order in cracking dictionaries
Filtering wordlists by line length or printable characters during deduplication
Security research workflows involving large text corpus cleaning
Pre-processing wordlists for brute-force or dictionary attacks

Not Ideal For

General data cleaning tasks where sorting the output is acceptable or desired
Processing small files where the overhead of compilation and optimization isn't necessary
Scenarios requiring deduplication based on fuzzy matching or semantic similarity
Environments where easy installation via package managers is preferred over compiling from source

Pros & Cons

Pros

Massive File Handling

Can process wordlists larger than available RAM by splitting them into virtual chunks, enabling handling of multi-gigabyte files without system limits.

Order Preservation

Deduplicates without altering line order, which is critical for password cracking efficiency where probable passwords must remain at the front.

Memory Optimization

Uses compressed hashmap items and tagged pointers to minimize memory footprint, as detailed in the technical implementation for efficient large-scale processing.

Multithreading Support

Leverages multiple threads for faster processing on modern hardware, speeding up deduplication for performance-intensive workflows.

Additional Filtering

Includes options to filter by line length, ASCII printable characters, and case conversion, adding utility beyond basic deduplication for tailored wordlist preparation.

Cons

Line Size Limitation

The --line-max-size option cannot exceed 4095 characters, which may be restrictive for use cases with very long lines or non-standard data.

Performance Trade-off with Dupfile

Using the --dupfile option to save duplicates slows down processing, as admitted in the README, making it less ideal for time-sensitive operations.

Niche ASCII Focus

Optimized for ASCII text and password cracking; lacks built-in support for non-ASCII encodings or binary files, limiting general-purpose applicability.

Compilation Barrier

Requires compilation from C source, which can be a hurdle for users without development tools or those seeking quick, package-based installation.

Frequently Asked Questions

Related Projects

CUPP

Common User Passwords Profiler (CUPP)

Stars5,903

Forks1,968

Last commit4 months ago

StringZilla

Up to 100x faster strings for C, C++, CUDA, Python, Rust, Swift, JS, & Go, leveraging NEON, AVX2, AVX-512, SVE, GPGPU, & SWAR to accelerate search, hashing, sorting, edit distances, sketches, and memory ops 🦖

Stars3,444

Forks123

Last commit1 month ago

Mentalist

Mentalist is a graphical tool for custom wordlist generation. It utilizes common human paradigms for constructing passwords and can output the full wordlist as well as rules compatible with Hashcat and John the Ripper.

Stars1,974

Forks258

Last commit19 days ago

bopscrk

Generate smart and powerful wordlists

Stars1,076