datacompy
A Python library for comparing Pandas, Polars, Spark, and Snowpark DataFrames with detailed reporting and flexible matching.
What is datacompy?
DataComPy is a Python library for comparing DataFrames across multiple backends like Pandas, Polars, Spark, and Snowpark. It solves the problem of validating data consistency and quality by providing detailed mismatch reports and configurable comparison logic, going beyond basic equality checks. It serves as a modern, open-source alternative to proprietary data comparison tools.
Data engineers, data scientists, and analysts working with Python data ecosystems who need to validate, reconcile, or audit datasets across different processing frameworks or environments.
Developers choose DataComPy for its multi-backend support, detailed reporting, and flexibility in matching criteria, offering a more informative and adaptable solution than native DataFrame equality methods or limited proprietary alternatives.
Overview
Pandas, Polars, Spark, and Snowpark DataFrame comparison for humans and more!
Use Cases
Best For
- Validating ETL pipeline outputs between different processing engines
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
