A high-speed entropy coding library featuring Huff0 (modern Huffman) and FSE (ANS-based) codecs for fast compression and decompression.
FiniteStateEntropy is a library implementing two high-speed entropy coders: Huff0, a modern Huffman codec optimized for CPU out-of-order execution, and FSE, an Asymmetric Numeral Systems (ANS) based encoder that offers arithmetic coding-like accuracy at much higher speeds. It solves the problem of achieving fast compression and decompression while maintaining high compression ratios, especially for skewed data distributions.
Developers and engineers working on performance-critical applications requiring fast lossless data compression, such as real-time data processing, game development, or embedded systems.
It provides a significant speed advantage over traditional entropy coders like zlib's Huffman encoder, with Huff0 offering extreme decompression speeds and FSE delivering near-optimal compression accuracy without sacrificing performance.
New generation entropy codecs : Finite State Entropy and Huff0
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Huff0 achieves decompression speeds up to 1350 MB/s, massively outperforming traditional implementations like zlibh (300 MB/s), as shown in benchmarks on an Intel Core i7-5600U.
FSE provides near-arithmetic coding ratios, e.g., 8.84 vs Huff0's 6.38 on skewed data (Proba80), breaking the 1-bit-per-symbol Huffman limit for better efficiency.
Both codecs are designed for out-of-order execution and multiple ALUs, leveraging modern architectures to bridge the gap between speed and accuracy.
Offers flexibility with Huff0 for raw speed on less compressible data and FSE for high ratios on skewed distributions, allowing tailored performance.
Users must analyze their data to choose between Huff0 and FSE, adding complexity compared to libraries with auto-adaptive or unified interfaces.
Focuses solely on entropy coding without built-in support for common compression streams, containers, or higher-level APIs, requiring additional integration work.
The README is highly technical and assumes prior knowledge of entropy coding theories, lacking beginner-friendly guides or comprehensive API examples.
FiniteStateEntropy is an open-source alternative to the following products: