fluvio
A distributed data streaming engine with stateful stream processing for building responsive data-intensive applications.
What is fluvio?
Fluvio is a distributed data streaming engine written in Rust, combined with the Stateful DataFlow stream processing framework. It provides a unified, composable paradigm for collecting, transforming, and processing real-time data streams to power responsive applications.
Developers and data engineers building scalable, real-time data pipelines and stream processing applications, particularly those working with IoT, financial data, or event-driven architectures.
Developers choose Fluvio for its lean, unified approach that combines streaming and processing into a single system, offering high performance through Rust, extensibility via WASM-based Smart Modules and connectors, and multi-language client support for flexibility.
Overview
🦀 event stream processing for developers to collect and transform data in motion to power responsive data intensive applications.
Use Cases
Best For
- Building real-time data pipelines that require high performance and scalability, such as IoT sensor data ingestion.
- Implementing stateful stream processing workflows with custom logic using the Stateful DataFlow framework.
- Integrating with diverse systems (e.g., HTTP, MQTT, Kafka, SQL) using native or community-built connectors.
- Developing reusable data transformation functions as WASM-based Smart Modules for stream processing.
- Creating responsive applications that need to process data in motion across multiple programming languages (Rust, Python, JavaScript, Go, etc.).
- Setting up local or distributed streaming clusters quickly with a declarative and composable model for developer productivity.
Not Ideal For
- Enterprises requiring fully-managed, vendor-supported streaming services with SLAs
- Projects needing stable, production-ready connectors for specific databases like DuckDB or Redis (listed as experimental)
- Teams heavily invested in Java ecosystems without tolerance for community-maintained client libraries
- Environments where native Windows support is mandatory without using WSL2
Pros & Cons
Pros
Combines distributed data streaming with stateful processing in a single framework, reducing system complexity and enabling cohesive real-time workflows.
Built in Rust for efficient resource usage and scalability, making it suitable for high-throughput data pipelines like IoT or financial streaming.
Uses WebAssembly-based Smart Modules for safe, reusable data transformations, allowing custom logic without compromising performance.
Offers native and community-built connectors for systems like HTTP, MQTT, Kafka, and SQL, facilitating easy integration with diverse data sources and sinks.
Provides official clients in Rust, Python, JavaScript, and community-maintained ones in Go, Java, Elixir, enhancing accessibility across tech stacks.
Cons
Connectors for DuckDB, Redis, S3, and Graphite are labeled as experimental, meaning they may lack stability or full feature sets for production use.
The installation script currently defaults to a dev version due to community transition, leading to potential setup inconsistencies or version management issues.
Officially recommends WSL2 for Windows, indicating poor compatibility with native Windows environments, which can hinder adoption in certain teams.
Clients for languages like Go, Java, and Elixir rely on community maintenance, risking uneven support, slower updates, and potential integration gaps.
Open Source Alternative To
fluvio is an open-source alternative to the following products:
Frequently Asked Questions
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.