Build concurrent, multi-stage data ingestion and processing pipelines with Elixir, supporting back-pressure, batching, and fault tolerance.
Broadway is an Elixir library for building concurrent, multi-stage data ingestion and processing pipelines. It simplifies consuming data from sources like Amazon SQS, Apache Kafka, and RabbitMQ by providing built-in back-pressure, batching, and fault tolerance, enabling robust and efficient event-driven architectures.
Elixir developers building data-intensive applications that require reliable ingestion from message queues or streaming services, such as real-time analytics, event processing, or microservices communication.
Developers choose Broadway for its operational focus—it abstracts complex GenStage topologies into a simple API with automatic acknowledgements, metrics, and failure handling, reducing boilerplate while ensuring high concurrency and resilience in continuous pipelines.
Concurrent and multi-stage data ingestion and data processing with Elixir
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Built-in back-pressure manages message flow to prevent downstream overload, ensuring stable processing under variable loads, as highlighted in the key features.
Provides fault tolerance with graceful shutdown and custom failure handling, making pipelines robust for long-lived, event-driven applications, per the philosophy section.
Abstracts complex GenStage topologies into a concise setup, demonstrated by the SQS integration example requiring only 20 lines of code in the README.
Offers dedicated utilities for testing pipeline components, reducing development overhead and ensuring reliability, as listed in the built-in features.
Lacks built-in support for data transformations like joins or aggregations, which the README admits are better handled by Flow, making it less suitable for complex analytics.
Tightly coupled to Elixir and the Erlang VM, limiting use in polyglot projects or teams not invested in this stack, despite broad producer support.
Integrating with unsupported data sources requires implementing custom producers, which can be non-trivial and time-consuming, as inferred from the producer documentation.