An open-source MLOps framework for defining and deploying machine learning and LLM workloads across any cloud infrastructure.
Aqueduct is an open-source MLOps framework that enables data scientists and engineers to define machine learning and LLM workflows in vanilla Python and deploy them on any cloud infrastructure, such as Kubernetes, Spark, or AWS Lambda. It addresses the fragmented nature of MLOps by providing a unified interface to existing tools while ensuring centralized visibility into pipeline execution. The framework allows seamless movement of code between different cloud layers without requiring a rip-and-replace approach.
Data scientists and ML engineers who need to deploy and manage machine learning or LLM workflows across diverse cloud infrastructure like Kubernetes, Spark, Airflow, or AWS Lambda. It is suited for teams dealing with the complexity of siloed MLOps tools and seeking a Python-native solution.
Developers choose Aqueduct because it offers a Python-native API without DSLs or YAML configurations, enabling quick production deployment. Its unique selling point is the ability to integrate with and orchestrate workflows across multiple existing cloud infrastructure systems while providing centralized visibility into code, data, and metadata for reliability and debugging.
Aqueduct is no longer being maintained. Aqueduct allows you to run LLM and ML workloads on any cloud infrastructure.
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Enables defining ML and LLM workflows in vanilla Python without DSLs or YAML, as shown in the quickstart guide for rapid deployment and ease of use.
Integrates seamlessly with diverse infrastructures like Kubernetes, Spark, and AWS Lambda, allowing tasks to run across different systems without replacing existing tooling.
Provides a unified UI to monitor code, data, metrics, and metadata from each workflow run, enhancing reliability and debugging capabilities as illustrated in the README screenshot.
Runs entirely within the user's cloud environment, ensuring data and code security without relying on external services, aligning with the emphasis on security in the features list.
Lacks native features for data versioning and experiment tracking, requiring users to integrate and manage additional tools for a complete MLOps stack, which adds complexity.
Configuring and managing connections to various cloud infrastructures like Kubernetes and AWS Lambda can be challenging, especially for teams without extensive DevOps expertise.
As a newer framework, it has fewer pre-built operators, integrations, and community support compared to established platforms like Airflow or MLflow, potentially slowing adoption.