A high-performance, declarative stream processor that connects various sources and sinks with built-in data transformation capabilities.
Redpanda Connect is a declarative stream processor that connects various data sources and sinks while performing real-time transformations, enrichment, and filtering on payloads. It solves the problem of building resilient data pipelines between different systems with guaranteed delivery and minimal operational overhead.
Platform engineers, data engineers, and DevOps teams building real-time data pipelines, ETL processes, or event-driven architectures that require reliable integration between multiple data sources and destinations.
Developers choose Redpanda Connect for its declarative configuration model, at-least-once delivery guarantees without disk persistence, and extensive connector ecosystem that simplifies building production-ready stream processing pipelines with minimal code.
Fancy stream processing made operationally mundane
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Allows defining entire stream pipelines in a single YAML file, enabling easy version control and deployment without writing code, as shown in the example config.
Provides at-least-once delivery by default using an in-process transaction model without disk persistence, ensuring resilience during crashes or server faults.
Supports integration with dozens of sources and sinks including AWS, GCP, Kafka, and SQL databases, reducing the need for custom integration code.
Includes built-in health checks, metrics, and tracing, making it easy to deploy and monitor in containerized environments like Kubernetes.
Some plugins require external libraries and specific build tags (e.g., x_benthos_extra), which can complicate custom deployments and lead to linking errors.
The README directs users to external documentation sites, potentially making initial onboarding and troubleshooting less centralized and accessible.
The custom Bloblang mapping language, while powerful, requires learning a new DSL, which can be a barrier compared to using standard languages like SQL or Python.