How do I pass a list or dictionary as a parameter in Papermill?

Use YAML input via the -y, -f, or -b flags for structured data. For example, -y allows nested YAML syntax to define arrays and dictionaries directly in the command line or files.

Papermill vs. nbconvert: which should I use for automating notebooks?

Papermill is for parameterizing and executing notebooks with dynamic inputs, ideal for data pipelines. nbconvert focuses on converting notebooks to static formats like HTML or PDF; use it for reporting without parameter changes.

Can Papermill run notebooks in different Python environments or with conda?

Papermill executes notebooks using the current Jupyter kernel, so you must pre-configure kernels for each environment. It doesn't manage conda environments natively, requiring manual kernel setup.

How to handle errors when a notebook fails during Papermill execution?

Papermill raises exceptions on failures, but for robust error handling, wrap execute_notebook in try-catch blocks or use orchestration tools like Apache Airflow to manage retries, logging, and alerts.

Does Papermill work with JupyterHub or cloud notebook platforms?

Yes, but you need to ensure notebook paths and kernels are accessible. Papermill can execute via API in JupyterHub, though configuration might require additional setup for authentication and file storage.

How to schedule Papermill notebook runs automatically?

Use cron jobs for CLI execution or integrate the Python API into workflow schedulers like Apache Airflow, Prefect, or Luigi, leveraging their scheduling features to trigger parameterized runs.

Open-Awesome

papermill

BSD-3-ClausePython

A Python tool for parameterizing, executing, and analyzing Jupyter Notebooks at scale.

Visit Website GitHub

6.4k stars449 forks0 contributors

What is papermill?

Papermill is a Python tool that allows users to parameterize, execute, and analyze Jupyter Notebooks programmatically. It solves the problem of manually running notebooks with different inputs by enabling automation and integration into larger data workflows.

Target Audience

Data scientists, data engineers, and analysts who use Jupyter Notebooks for reproducible research, reporting, or pipeline tasks and need to automate execution with varying parameters.

Value Proposition

Developers choose Papermill for its straightforward approach to notebook automation, support for multiple cloud storage backends, and its ability to turn notebooks into reusable, parameterized components within scalable data pipelines.

Overview

📚 Parameterize, execute, and analyze notebooks

Use Cases

Best For

Automating financial or operational reports that need to run with different date ranges or values
Building data pipelines where notebooks are executed sequentially based on previous results
Running batch experiments in machine learning with varying hyperparameters
Generating parameterized dashboards or visualizations from notebook templates
Scheduling notebook executions in production workflows
Testing notebook code with different input datasets programmatically

Not Ideal For

Real-time applications requiring sub-second processing latencies
Teams emphasizing pure Python scripts for better version control and unit testing
Environments where Jupyter Notebooks are prohibited or unsupported, such as lightweight CI/CD pipelines

Pros & Cons

Pros

Seamless Parameter Injection

Uses Jupyter cell tags to designate a 'parameters' cell, automatically injecting and tracking runtime inputs in an 'injected-parameters' cell for easy reuse and reproducibility.

Flexible Input Methods

Supports parameters via CLI flags, YAML files, raw strings, and base64-encoded YAML, accommodating various workflow needs from simple commands to complex structured data.

Multi-Cloud Storage Support

Integrates with AWS S3, Azure Blob/DataLake, Google Cloud Storage, and local/HTTP paths, enabling scalable data pipeline execution across diverse environments.

Programmatic Execution Options

Offers both Python API and CLI interfaces, allowing seamless integration into automated workflows, schedulers, or custom Python applications.

Cons

Jupyter Ecosystem Dependency

Tightly coupled to Jupyter Notebooks and kernels, which can be resource-intensive, slow to start, and less suitable for production deployments preferring lightweight scripts.

Optional Dependency Overhead

Full cloud storage support requires installing optional bundles like 'all', adding setup complexity and potential version conflicts in constrained environments.

Limited Advanced Orchestration

Focuses on parameterization and execution but lacks built-in features for complex error handling, monitoring, or dependency management, often needing external tools like Airflow.

Frequently Asked Questions

Related Projects

Excalidraw

Virtual whiteboard for sketching hand-drawn like diagrams

Stars121,745

Forks13,394

Last commit2 days ago

tesseract

Tesseract Open Source OCR Engine (main repository)

Stars73,670

Forks10,609

Last commit3 days ago

Pandoc

Universal markup converter

Stars43,611

Forks3,827