A tool to copy PostgreSQL databases between servers with parallel data streaming and concurrent index building, bypassing intermediate files.
pgcopydb is a command-line tool for copying PostgreSQL databases between servers. It enhances the standard `pg_dump` and `pg_restore` utilities by enabling parallel data streaming and concurrent index building, solving performance bottlenecks in large database migrations. It also supports online migrations through Change Data Capture (CDC) using PostgreSQL's logical decoding.
Database administrators, DevOps engineers, and developers who need to migrate, replicate, or clone PostgreSQL databases efficiently, especially in production environments with minimal downtime.
Developers choose pgcopydb because it eliminates the need for intermediate files, allows highly parallel operations, and provides both offline and online migration capabilities, resulting in significantly faster and more reliable database copies compared to vanilla pg_dump/pg_restore.
Copy a Postgres database to a target Postgres server (pg_dump | pg_restore on steroids)
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Streams table data directly between servers using multiple processes, bypassing intermediate file writes and significantly speeding up large data transfers, as shown in the example output with 8 concurrent COPY jobs.
Creates all indexes for each table simultaneously after data copy, optimizing performance by reducing lock contention, implemented via CREATE UNIQUE INDEX followed by ALTER TABLE commands.
Supports Change Data Capture via PostgreSQL's logical decoding and wal2json, enabling zero-downtime migrations with the `pgcopydb clone --follow` command for replaying changes.
Uses PostgreSQL snapshots to ensure data consistency during the copy process, preventing issues from concurrent modifications, as demonstrated in the logs with exported snapshot IDs.
Requires configuration of logical decoding and the wal2json plugin on the source database, which the README admits involves 'a lot more complexities' and can be error-prone for online migrations.
Depends on pg_dump and pg_restore binaries matching the target PostgreSQL version, adding maintenance overhead and potential compatibility issues in heterogeneous environments.
Lacks a graphical interface or web API, which may hinder adoption in teams accustomed to GUI-based database management tools, relying solely on terminal commands.