Rapidgzip
A parallel gzip decompressor with fast random access, utilizing multi-core CPUs for high-speed decompression of standard gzip files.
What is Rapidgzip?
Rapidgzip is a parallel gzip decompression tool and library that accelerates decompression of standard gzip files on multi-core machines. It solves the problem of slow single-threaded gzip decompression by utilizing all available CPU cores, and adds fast random access capabilities for seeking within compressed files without full decompression.
Developers and data engineers working with large gzip-compressed datasets, such as genomic FASTQ files, log archives, or scientific data, who need faster decompression or random access to portions of compressed files.
Unlike tools like bgzip or pigz that only parallelize their own formats, Rapidgzip works with any standard gzip file, offering significant speedups (up to 55x with 128 cores) and random access without requiring re-compression.
Overview
Gzip Decompression and Random Access for Modern Multi-Core Machines
Use Cases
Best For
- Decompressing large gzip files faster by using all CPU cores
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
