skizze
A probabilistic data structure service and storage for efficient frequency, cardinality, and membership queries on large datasets.
Overview
Skizze is a sketch data store designed to handle counting and sketching problems using probabilistic data structures. Unlike a key-value store, it appends values to defined sketches, enabling near O(1) time queries for frequency, cardinality, and membership with minimal memory footprint, making it suitable for real-time analytics on massive datasets.
Key Features
- Probabilistic Data Structures — Uses sketches to estimate metrics like cardinality and frequency, trading precision for reduced memory consumption.
- Fire-and-Forget Service — Operates as a standalone service for easy integration into data processing pipelines.
- Multi-Type Sketches — Supports distinct sketch types for cardinality (CARD), membership (MEMB), frequency (FREQ), and rankings (RANK).
- Domain Management — Organizes sketches into domains (collections) for structured data handling.
- CLI and Bindings — Includes a command-line interface for testing and bindings for Go and Node.js.
Philosophy
Skizze aims to replace heavyweight distributed processing for certain analytics tasks by providing a compact, efficient service that summarizes data streams using probabilistic approximations, enabling real-time insights without storing raw values.
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
