python-deequ

Leverages Apache Spark to compute data quality metrics like completeness and uniqueness on massive datasets, as shown in the Analyzers and Profile examples that process distributed DataFrames.

Uses the ConstraintSuggestionRunner to automatically generate data quality rules based on dataset analysis, reducing manual effort in defining checks, as highlighted in the quickstart.

The Metrics Repository allows persisting and querying past data quality runs via FileSystemMetricsRepository, enabling trend analysis and monitoring over time, as demonstrated in the repository example.

Built as a Python wrapper for Deequ, it integrates directly with PySpark sessions and AWS Glue workflows, making it easy to add data quality checks to existing big data pipelines, per the blogpost references.