Gobblin
A distributed data integration framework for big data ecosystems, handling ingestion, replication, organization, and lifecycle management for both streaming and batch data.
What is Gobblin?
Apache Gobblin is a distributed data integration framework designed to simplify big data integration tasks such as data ingestion, replication, organization, and lifecycle management. It handles both streaming and batch data ecosystems, providing a scalable solution for managing structured and byte-oriented data across heterogeneous environments. The framework is optimized for ELT patterns with inline transformations and is battle-tested at petabyte scale in production environments.
Data engineers and architects working with large-scale data ecosystems who need reliable ingestion, replication, and lifecycle management across diverse data sources and sinks. Organizations with complex data integration requirements across Hadoop, cloud storage, and external APIs will benefit most.
Developers choose Apache Gobblin for its proven scalability in production environments, comprehensive feature set for data management, and flexibility in supporting both stream and batch execution modes. Its ability to handle petabyte-scale workflows while providing fault tolerance, data quality checking, and compliance management makes it a robust alternative to building custom integration solutions.
Overview
A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.
Use Cases
Best For
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
