A Rust crate for multi-threaded compression and decompression, supporting Gzip, Zlib, Snappy, BGZF, and Mgzip formats.
gzp is a Rust crate that enables multi-threaded compression and decompression for various formats like Gzip, Zlib, Snappy, BGZF, and Mgzip. It solves the problem of slow compression by parallelizing data chunk processing while preserving output order. The library acts as a drop-in replacement for writers, making it easy to integrate into existing Rust applications.
Rust developers working with large datasets or high-throughput applications that require fast compression, such as bioinformatics (BGZF), log processing, or data pipeline optimization.
Developers choose gzp for its significant speed improvements over single-threaded compression, support for multiple formats including specialized block formats, and seamless integration via the standard `Write` trait. Its performance scales with thread count, making it ideal for modern multi-core systems.
Multi-threaded Compression
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Multi-threaded compression scales performance with thread count, significantly reducing compression time for large datasets, as shown in benchmarks with a 550Mb file.
Handles Gzip, Zlib, Raw Deflate, Snappy, BGZF, and Mgzip, making it versatile for bioinformatics, log processing, and general compression tasks.
Implements the `Write` trait, allowing easy replacement in existing Rust code with minimal changes, as demonstrated in the simple example.
Offers choice between libdeflate and Rust-native backends via feature flags, enabling optimization for performance or compatibility.
Missing auto-generated indexes for BGZF/Mgzip and adler implementation, as noted in future todos, which limits functionality for specialized use cases.
Known compilation issues with libdeflate on i686-pc-windows-msvc targets, requiring manual workarounds as mentioned in the README.
Requires careful feature selection (e.g., deflate_default vs deflate_rust) which can be confusing and error-prone for users.
Only provides multi-threaded decompression for Mgzip and BGZF formats; other formats rely on single-threaded or external decompressors.
gzp is an open-source alternative to the following products: