crabz
A fast, cross-platform, multi-threaded compression and decompression CLI tool written in Rust.
What is crabz?
Crabz is a command-line compression and decompression tool written in Rust that provides parallel processing capabilities for handling large files efficiently. It serves as a faster, cross-platform alternative to pigz, supporting multiple compression formats including Gzip, BGZF, and Mgzip. The tool addresses the need for high-speed data compression in workflows where performance and format flexibility are critical.
System administrators, data engineers, and developers working with large datasets who need efficient compression and decompression tools for data processing pipelines or storage optimization.
Developers choose Crabz for its superior multi-threaded performance, cross-platform support, and additional format options compared to pigz. Its Rust implementation offers memory safety advantages while maintaining competitive speed through optimized backends like zlib-ng.
Overview
Like pigz, but rust
Use Cases
Best For
- Compressing large log files for storage efficiency
- Parallel decompression of genomic data in BGZF format
- High-speed data pipeline preprocessing with multi-threaded compression
- Cross-platform compression tasks on mixed Linux/Windows environments
- Optimizing compression performance with thread pinning on multi-core systems
- Batch processing of files with in-place compression to save disk space
Not Ideal For
- Users needing GUI-based compression tools with drag-and-drop interfaces
- Environments with single-core CPUs where multi-threading overhead reduces efficiency
- Workflows requiring compression formats not supported, such as bzip2 or xz
- Scripts relying on standard gzip command compatibility without extra dependencies
Pros & Cons
Pros
Benchmarks show crabz with zlib-ng backend compresses up to 50% faster than pigz, making it ideal for large datasets like logs or genomic data.
Explicitly designed to run on Linux, macOS, and Windows, providing consistent performance across different operating systems without recompilation.
Supports zlib, zlib-ng, and a pure Rust backend, allowing users to choose between speed, compatibility, and memory safety as evidenced in benchmark tables.
Handles BGZF and Mgzip formats with optimized decompression, outperforming pigz in block data benchmarks, useful for bioinformatics workflows.
Cons
The README lists this as a TODO, requiring users to manually specify formats with -f flag, which can be inconvenient for mixed-format batches.
Thread pinning requires specifying physical cores with -P flag, which can conflict with other processes and needs careful management in multi-instance environments.
The -I flag deletes the input file upon completion, posing a data loss risk if not used cautiously, with no built-in backup option.
The -d flag is noted to 'may change in future releases', indicating potential breaking changes that could disrupt existing scripts.
Open Source Alternative To
crabz is an open-source alternative to the following products:
Frequently Asked Questions
Related Projects
BatA cat(1) clone with wings.
fdA simple, fast and user-friendly alternative to 'find'
nushellA new type of shell
zoxideA smarter cd command. Supports all major shells.
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.