DataSketches
A Java library of stochastic streaming algorithms (sketches) for approximate analysis of massive datasets.
What is DataSketches?
Apache DataSketches Core is a Java library implementing stochastic streaming algorithms called sketches for approximate analysis of massive datasets. It solves the problem of processing data streams and large datasets where exact computation would require prohibitive memory resources. The library provides algorithms for distinct counting, quantile estimation, frequency analysis, and set operations with mathematically proven error bounds.
Data engineers and scientists working with streaming data or large-scale datasets who need memory-efficient approximate analytics. Developers building data processing systems in Hadoop, Spark, or other big data ecosystems will find this library particularly valuable.
Developers choose DataSketches for its rigorous mathematical foundations, production-quality implementations, and cross-language consistency. The library provides reliable, memory-efficient algorithms with predictable error bounds, making it suitable for mission-critical data processing tasks where traditional exact methods are impractical.
Overview
A software library of stochastic streaming algorithms, a.k.a. sketches.
Use Cases
Best For
- Counting unique users in high-volume web traffic streams
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
