databus
A source-agnostic distributed change data capture system for reliably capturing and streaming primary data changes.
What is databus?
Databus is a source-agnostic distributed change data capture system built by LinkedIn to reliably capture and stream primary data changes from databases like Oracle and MySQL. It solves the problem of keeping secondary data stores, caches, and indexes consistent with primary data mutations by mining database transaction logs, avoiding the consistency issues of application-driven dual writes.
Data engineers and architects at technology companies building complex data ecosystems with multiple specialized data stores (e.g., relational databases, NoSQL, caches, search indexes) that need reliable, low-latency data change propagation.
Developers choose Databus for its source-agnostic approach that prevents technology lock-in, its strong consistency guarantees by using database logs as a single source of truth, and its ability to handle high throughput with low latency while supporting infinite data lookback.
Overview
Source-agnostic distributed change data capture system
Use Cases
Best For
- Building reliable data pipelines between primary OLTP databases and secondary stores like search indexes or caches
- Keeping social graph or people search indexes consistent with member profile data changes
- Implementing change data capture for Oracle or MySQL databases without proprietary replication
- Avoiding consistency issues in application-driven dual-write architectures
- Enabling infinite lookback capabilities for data change replay in downstream applications
- Creating scalable event streaming systems for data change propagation in microservices architectures
Not Ideal For
- Environments using databases not supported by Databus, like PostgreSQL or MongoDB, without custom log mining adapters
- Small-scale projects where application-driven dual writes with eventual consistency are acceptable and simpler to implement
- Teams lacking expertise in distributed systems or database transaction log mining, as setup and maintenance are complex
- Organizations seeking a cloud-managed, zero-dependency CDC service with minimal operational overhead
Pros & Cons
Pros
Extracts changes from database transaction logs like Oracle and MySQL without relying on proprietary replication, avoiding technology lock-in, as highlighted in the Key Features.
Solves consistency issues by mining database logs as a single source of truth, unlike error-prone application-driven dual writes, per the Philosophy and Use-cases sections.
Delivers transport layer latencies in low milliseconds and handles thousands of events per second per server, ensuring scalable data flow for high-throughput environments.
Supports replaying historical data changes from any point in time, enabling flexible subscription and recovery capabilities for downstream applications.
Cons
Requires downloading Oracle JDBC driver with specific licensing and version, and uses gradle for build, making initial setup non-trivial and prone to errors, as warned in the How to build? section.
Primarily designed for Oracle and MySQL sources; adapting to other databases requires custom log mining implementations, which may not be well-documented or supported.
Based on a 2012 paper and older technology, the project might have less active development and community support compared to modern CDC tools like Debezium, risking obsolescence.
Frequently Asked Questions
Related Projects
Apache Kafka StreamsApache Kafka - A distributed event streaming platform
nsqA realtime distributed messaging platform
Apache RocketMQApache RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications.
Apache PulsarApache Pulsar - distributed pub-sub messaging system
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.