runiq
An efficient command-line tool and library for filtering duplicate lines from textual input, optimized for speed and memory usage.
What is runiq?
Runiq is a Rust-based command-line utility and library that filters duplicate lines from textual input with high efficiency in both time and memory. It solves the problem of deduplicating large datasets by offering multiple filtering algorithms optimized for different trade-offs between speed, memory usage, and accuracy. It serves as a versatile alternative to traditional Unix tools like `uniq` and `sort -u`.
System administrators, data engineers, and developers processing large log files, datasets, or streams who need efficient deduplication with control over performance characteristics. It is also suitable for Rust developers seeking a programmatic deduplication library.
Developers choose Runiq for its configurable filters that allow precise tuning for specific use cases, such as the memory-efficient `compact` filter using a Bloom Filter or the high-speed `sorted` filter for pre-sorted data. It offers competitive or superior performance and lower memory usage compared to alternatives like `uniq`, `sort -u`, `uq`, and `huniq`.
Overview
An efficient way to filter duplicate lines from input, à la uniq.
Use Cases
Best For
- Processing unsorted log files with a need for balanced speed and memory efficiency using the `quick` filter.
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
