A high-performance, resilient stream processor that connects various sources and sinks, performs data transformations, and guarantees at-least-once delivery.
Redpanda Connect is a stream processor that connects various data sources and sinks, transforms payloads using a mapping language, and guarantees at-least-once delivery. It simplifies building resilient data pipelines with declarative configuration and cloud-native deployment options.
Developers and data engineers building real-time data pipelines, ETL processes, or event-driven architectures who need reliable integration between multiple systems.
It offers a unique combination of high performance, operational simplicity, and strong delivery guarantees without requiring complex state management, making it easier to deploy and scale than many alternatives.
Fancy stream processing made operationally mundane
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Entire pipelines are defined in a single YAML file, enabling easy versioning and deployment, as shown in the README example config.
Uses an in-process transaction model without disk persistence to ensure reliable message processing even during crashes, simplifying resilience.
Integrates with AWS, GCP, Kafka, databases, and many others, reducing the effort to connect diverse data sources and sinks.
Runs as a Docker image or binary with built-in health checks and metrics, making it easy to deploy in Kubernetes or similar environments.
Adding extra plugins requires specific build tags like 'x_benthos_extra' and can lead to linking errors, complicating custom extensions.
The custom mapping language, while powerful, requires learning a new syntax, which can slow initial adoption compared to using standard languages like Python.
Even basic data transfers involve writing YAML, which might be overkill compared to lightweight scripts or simpler tools.