db2lake

db2lake is a small framework designed for building ETL (Extract, Transform, Load) pipelines that move data from databases to data lakes and warehouses. It provides a stable core API with pluggable source and destination drivers, enabling efficient and resumable data transfers.

Key Features

• Driver-Based Architecture — Uses separate packages for sources (like MySQL, Postgres) and destinations (like BigQuery, Snowflake) for modularity.
• Cursor-Based Scheduling — Supports resumable transfers by tracking cursor information, ensuring only new data is moved on subsequent runs.
• Lightweight Core — Offers a minimal, stable API that keeps the framework simple and focused.
• TypeScript Support — Includes complete TypeScript examples for building type-safe pipelines.
• Custom Transformers — Allows row transformation between source and destination to adapt data schemas.

Philosophy

db2lake emphasizes a minimal core API to keep drivers simple and the framework lightweight, prioritizing stability and ease of extension.