deequ
A library built on Apache Spark for defining unit tests to measure data quality in large datasets.
What is deequ?
Deequ is an open-source library built on Apache Spark that allows data teams to define and run unit tests for data quality at scale. It helps validate assumptions about large datasets—such as completeness, uniqueness, and value constraints—before the data is used in analytics or machine learning pipelines. By catching data errors early, it prevents downstream issues and ensures reliable data products.
Data engineers, data scientists, and analytics engineers working with large-scale data pipelines on Apache Spark who need to ensure data quality and reliability.
Developers choose Deequ because it provides a scalable, programmatic way to enforce data quality checks directly within Spark workflows, reducing manual validation efforts and catching data issues before they impact business decisions or models.
Overview
Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.
Use Cases
Best For
- Validating data completeness and uniqueness in large ETL pipelines
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.