Engine
27 projects
Apache ApexA unified platform for big data stream and batch processing on Hadoop YARN with enterprise-grade operability.
Apache BallistaA distributed query execution engine that extends Apache DataFusion to run SQL queries in parallel across multiple nodes.
Apache Heron (incubating)Apache Heron is a real-time, distributed, fault-tolerant stream processing engine developed by Twitter.
Apache SamzaA distributed stream processing framework built on Apache Kafka and Apache Hadoop YARN for fault-tolerant, stateful processing.
ArkFlowA high-performance Rust stream processing engine with integrated AI capabilities for real-time data processing and intelligent analysis.
ArroyoA distributed stream processing engine in Rust that performs stateful computations on real-time data with subsecond results.
AthenaXSQL-based streaming analytics platform that scales to process hundreds of billions of real-time events daily.
BytewaxA Python framework and Rust-based distributed processing engine for stateful event and stream processing.
CocoIndexAn ultra-performant data transformation framework for AI, with incremental processing and data lineage built-in.
GearpumpA lightweight real-time big data streaming engine built on Akka for high-throughput, low-latency data processing.
Hazelcast JetAn open-source, in-memory, distributed batch and stream processing engine for Java applications.
hailstormA distributed stream processing system written in Haskell that guarantees exactly-once semantics.
mantisA platform for building realtime, cost-effective, operations-focused applications.
mupd8(muppet)A MapReduce-style framework for processing fast/streaming data, implementing the MapUpdate model.
NebulaStreamAn end-to-end data management system for IoT, optimizing stream processing across cloud, edge, and sensor deployments.
NumaflowA Kubernetes-native, serverless platform for running massively parallel data and streaming jobs with exactly-once semantics.
OnyxA masterless, cloud-scale, fault-tolerant distributed computation system for batch and stream processing written in Clojure.
PathwayA Python ETL framework for stream processing, real-time analytics, and building live LLM/RAG pipelines, powered by a scalable Rust engine.
Scramjet Cloud PlatformA runtime supervisor for deploying and running data processing programs called Sequences on Linux servers, Docker, and Kubernetes clusters.
tigonAn open-source real-time stream processing framework combining high-throughput event processing with low-latency SQL-like streaming queries.
TrillA high-performance one-pass in-memory streaming analytics engine for temporal and streaming data.
WallarooA fast, resilient distributed stream processing framework that simplifies real-time data applications with high performance and easy scaling.
LightSaberA multi-core stream processing engine for high-throughput window aggregation with optional exactly-once fault tolerance.
HStreamDBAn open-source, cloud-native streaming database designed for real-time data processing and IoT applications.
KuiperA lightweight IoT data analytics and stream processing engine for resource-constrained edge devices.
RisingWaveAn enterprise-grade event streaming platform that ingests, processes, and manages real-time event data with PostgreSQL compatibility and Apache Iceberg™ integration.
Library
13 projectsApache Kafka StreamsA distributed event streaming platform for building high-performance data pipelines, streaming analytics, and data integration.
StreamizA .NET stream processing library for Apache Kafka, providing a Kafka Streams-like API for building real-time applications.
Akka StreamsA platform for building highly responsive, resilient, and scalable distributed systems using the actor model.
BenthosA high-performance, resilient stream processor that connects various sources and sinks, performs data transformations, and guarantees at-least-once delivery.
FS2(prev. 'Scalaz-Stream')A purely functional, effectful, and polymorphic stream processing library for Scala built on Cats and Cats-Effect.
FastStreamAn asynchronous Python framework for building services that interact with Apache Kafka, RabbitMQ, NATS, and Redis event streams.
monixA high-performance Scala library for composing asynchronous, event-based programs with strong functional programming influences.
Quix StreamsA Python framework for building real-time data pipelines and event-driven microservices on Apache Kafka using a Streaming DataFrame API.
StreamlineA visual development platform for building, deploying, and managing streaming analytics applications with multiple engine bindings.
SubstationA serverless toolkit for routing, normalizing, and enriching security event and audit logs in AWS.
TributaryA Python library for constructing reactive dataflow graphs and streaming computations as data models.
YoMoAn open-source LLM function calling framework for building scalable, low-latency AI agents with geo-distributed edge infrastructure.
MediaPipeCross-platform framework for building customizable on-device machine learning pipelines for live and streaming media.
Application
3 projects
strawA scalable real-time search platform for streaming data using Apache Storm, Kafka, and Lucene.
storm-crawlerA scalable, mature, and versatile web crawler built on Apache Storm for building low-latency, distributed crawling systems.
ZillaA stateless, multi-protocol proxy that bridges web apps, IoT devices, and microservices directly to Apache Kafka via declarative APIs.
IoT
3 projects
sensorbeeA lightweight stream processing engine designed specifically for IoT data processing and analytics.
Apache EdgentAn open source programming model and runtime for analyzing data and events on edge devices, reducing data transmission and storage costs.
Apache StreamPipesA self-service IoT toolbox enabling non-technical users to connect, analyze, and explore industrial IoT data streams.
DSL
2 projects
EsperA Java/.NET component for complex event processing (CEP), streaming SQL, and event series analysis.
summingbirdA library for writing MapReduce programs that execute on distributed platforms like Storm and Scalding using Scala/Java collection-like syntax.