Kestra
An open-source, event-driven orchestration platform for building reliable scheduled and real-time workflows using declarative YAML.
What is Kestra?
Kestra is an open-source, event-driven orchestration platform that automates both scheduled and real-time workflows using declarative YAML. It solves the complexity of building reliable data pipelines, process automation, and microservice orchestration by applying Infrastructure as Code principles. Developers can define workflows visually or in code, with built-in scalability and a rich plugin ecosystem for extensibility.
Data engineers, DevOps teams, and platform engineers who need to orchestrate complex data pipelines, automate business processes, or manage microservice workflows. It's ideal for organizations running mission-critical applications requiring high availability and fault tolerance.
Developers choose Kestra for its declarative YAML approach combined with a powerful UI, enabling both code-first and visual workflow development. Its rich plugin ecosystem supports any language or cloud service, while built-in scalability and Git integration make it enterprise-ready.
Overview
Event Driven Orchestration & Scheduling Platform for Mission Critical Applications
Use Cases
Best For
- Orchestrating complex ETL and data pipelines across multiple cloud services
- Building event-driven automation that reacts to real-time triggers like Kafka messages or file uploads
- Managing scheduled cron jobs and batch processes with built-in error handling and retries
- Implementing Infrastructure as Code practices for workflow deployment and version control
- Creating scalable microservice orchestration with parallel and sequential task execution
- Developing custom plugins to extend orchestration capabilities for niche tools or internal systems
Not Ideal For
- Projects requiring simple, one-off script execution without workflow dependencies or monitoring
- Real-time streaming applications that need continuous data processing with sub-second latency
- Teams already deeply invested in Apache Airflow or similar tools with extensive customizations
Pros & Cons
Pros
Enables Infrastructure as Code for workflows with real-time syntax validation and auto-completion, ensuring consistency whether edited via UI or code.
Hundreds of built-in plugins for databases, cloud services, and multi-language scripting (e.g., Python, Node.js) reduce integration effort and extend functionality.
Designed to handle millions of workflows with high availability and fault tolerance, featuring retries, timeouts, and parallel execution for enterprise-scale deployments.
Combines a drag-and-drop interface with direct Git integration, allowing visual workflow building while maintaining version control and CI/CD pipelines.
Cons
Requires setting up Docker, Kubernetes, or cloud infrastructure (e.g., AWS CloudFormation), which can be cumbersome for quick prototyping or small teams.
With a vast plugin ecosystem, ensuring compatibility and updates adds maintenance burden; custom plugin development requires Java knowledge, as per the developer guide.
Complex workflows can lead to lengthy YAML configurations that are hard to manage manually, potentially relying heavily on the UI for readability.
Frequently Asked Questions
Related Projects
AirflowApache Airflow - A platform to programmatically author, schedule, and monitor workflows
TemporalTemporal service
LuigiLuigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
DagsterAn orchestration platform for the development, production, and observation of data assets.
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.