A distributed stream processing engine in Rust that performs stateful computations on real-time data with subsecond results.
Arroyo is a distributed stream processing engine written in Rust that performs stateful computations on streams of data. It enables users to ask complex questions of high-volume real-time data with subsecond results, operating on both bounded and unbounded data sources.
Data engineers and developers building real-time data pipelines who need scalable, fault-tolerant stream processing with SQL interfaces.
Developers choose Arroyo for its serverless operations in cloud environments, high-performance SQL as a first-class concern, and accessibility to non-experts while maintaining expert-level capabilities.
Distributed stream processing engine in Rust
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
SQL is treated as a first-class concern with consistently excellent performance, enabling complex queries without deep streaming expertise, as highlighted in the README's value proposition.
Designed for modern cloud environments, Arroyo supports serverless operations with seamless scaling, recovery, and rescheduling, making it ideal for scalable deployments.
Features state checkpointing for pipeline recovery, ensuring reliability in distributed settings, which is key for real-time data pipelines.
Offers a variety of connectors including Kafka and Iceberg, facilitating integration with common data sources and sinks, as listed in the features.
As a newer project, Arroyo has a smaller ecosystem and community compared to incumbents like Apache Flink, which may limit third-party integrations and support.
The managed Cloudflare Pipelines service is currently in beta and only supports stateless pipelines, restricting its use for stateful operations in cloud deployments.
Built in Rust, which may require teams to learn a new language or deal with Rust's learning curve, unlike JVM-based alternatives, potentially increasing onboarding time.
Arroyo is an open-source alternative to the following products: