A pytest plugin that validates Jupyter notebooks by executing cells and comparing outputs to ensure consistency.
nbval is a pytest plugin that validates Jupyter notebooks by executing their code cells and comparing the outputs with the stored results. It ensures notebooks run without errors and produce consistent outputs, which is crucial for maintaining reliable documentation, tutorials, and reproducible research. The plugin integrates seamlessly into pytest workflows, allowing developers to treat notebooks as testable artifacts.
Data scientists, researchers, and developers who use Jupyter notebooks for documentation, tutorials, or reproducible research and need to ensure their notebooks execute correctly and produce consistent outputs over time.
nbval provides a specialized testing framework for Jupyter notebooks, offering flexible validation modes, output sanitization, and integration with pytest ecosystems like coverage and parallel execution. It addresses the unique challenge of testing notebooks as executable documents, ensuring reliability without manual checks.
A py.test plugin to validate Jupyter notebooks
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Offers both strict (--nbval) and lax (--nbval-lax) modes, allowing users to choose between full output comparison or checking only marked cells, as detailed in the execution section.
Provides regex-based sanitization via a configuration file to ignore variable data like timestamps or random values, essential for reproducible testing without false positives.
Integrates directly into pytest workflows, enabling notebook tests to run alongside traditional Python tests and leverage pytest's ecosystem for coverage and parallel execution.
Supports running notebooks with their specified kernels or overriding with custom kernels, including the current environment, offering flexibility in execution contexts.
Setting up output sanitization requires creating and maintaining a ConfigParser-style file with regex patterns, which can be error-prone and time-consuming.
The plugin is deeply integrated with pytest, making it unsuitable for projects using alternative testing frameworks, limiting adoption in diverse environments.
Validating notebooks by executing all cells can be computationally expensive, and while parallel execution is supported, it requires extra configuration with pytest-xdist.