Ploomber
The fastest way to build data pipelines with iterative development and deployment anywhere.
What is Ploomber?
Ploomber is a framework for building data pipelines that allows developers to work interactively in editors like Jupyter, VSCode, or PyCharm and deploy without code changes to platforms like Kubernetes, Airflow, AWS Batch, or SLURM. It solves the problem of transitioning from exploratory notebooks to production pipelines by providing automated refactoring and flexible deployment options.
Data scientists, data engineers, and ML engineers who need to build, maintain, and deploy data or machine learning pipelines, especially those working with Jupyter notebooks and requiring production deployment.
Developers choose Ploomber for its fast pipeline development, seamless integration with interactive editors, and ability to deploy anywhere without rewriting code, along with automated migration of legacy notebooks into modular pipelines.
Overview
The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️
Use Cases
Best For
- Converting exploratory Jupyter notebooks into production data pipelines
- Building machine learning pipelines with incremental computation and caching
- Deploying data workflows to Kubernetes, Airflow, or AWS Batch
- Refactoring monolithic notebooks into modular, maintainable code
- Developing data pipelines interactively in VSCode or PyCharm
- Managing ETL processes with flexible deployment options
Not Ideal For
- Teams processing data in non-Python languages like R or Scala
- Organizations with deep customizations in established orchestrators like Apache Airflow
- Projects with simple, linear scripts that don't require pipeline orchestration or incremental computation
- Environments requiring battle-tested, enterprise-grade tools with extensive third-party integrations
Pros & Cons
Pros
Offers a simple YAML API for quick pipeline definition, enabling fast onboarding without extensive coding, as shown in the 'Get started quickly' feature video.
Integrates seamlessly with Jupyter, VSCode, and PyCharm, allowing developers to build and test pipelines interactively in their preferred environment, bridging exploration and production.
Automatically caches pipeline results and incrementally recomputes only changed tasks, reducing development time and resource usage, demonstrated in the development cycles video.
Supports deployment to Kubernetes, Airflow, AWS Batch, and SLURM without code changes, facilitating easy transition from development to production across various platforms.
Cons
Setting up distributed deployment on platforms like Kubernetes requires additional tools like soopervisor, which can add complexity compared to native orchestrator setups.
Limited to Python pipelines, making it unsuitable for polyglot data teams that need to integrate code from multiple programming languages.
As a newer framework, Ploomber has a smaller community and fewer pre-built integrations compared to established tools, which might affect support and extensibility.
Frequently Asked Questions
Related Projects
JupytextJupyter Notebooks as Markdown Documents, Julia, Python or R scripts
papermill📚 Parameterize, execute, and analyze notebooks
VoilaVoilà turns Jupyter notebooks into standalone web applications
nbdevCreate delightful software with Jupyter Notebooks
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.