data-diff
Fast tool for comparing datasets within or across SQL databases to identify differences.
What is data-diff?
Data-diff is a command-line tool and Python library for efficiently comparing datasets across different SQL databases or within the same database. It helps data engineers and analysts identify discrepancies, validate data migrations, and ensure data consistency between systems by using hashing and segmentation to quickly pinpoint row-level differences in large datasets.
Data engineers and analysts working with large datasets across SQL databases like PostgreSQL, MySQL, Snowflake, or BigQuery, who need to validate data migrations, monitor data quality, or ensure consistency between systems.
Developers choose data-diff for its cross-database comparison capabilities and efficient diff algorithms optimized for performance at scale, allowing fast and accurate identification of differences in massive datasets across heterogeneous data environments.
Overview
Compare tables within or across databases
Use Cases
Best For
- Validating data migrations between different SQL database systems to ensure no data loss or corruption.
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.