Gobblin from LinkedIn
A distributed data integration framework for big data ecosystems, handling ingestion, replication, organization, and lifecycle management for both streaming and batch data.
What is Gobblin from LinkedIn?
Apache Gobblin is a distributed data integration framework that simplifies big data integration tasks such as data ingestion, replication, organization, and lifecycle management. It supports both streaming and batch data ecosystems, enabling seamless data movement across heterogeneous sources and sinks like HDFS, S3, Kafka, and external APIs. The framework is optimized for ELT patterns and handles petabyte-scale data workflows in production environments.
Data engineers and architects working in big data ecosystems who need to manage data integration, replication, and lifecycle tasks across diverse data sources and storage systems. It is particularly suited for organizations with large-scale data lakes and compliance requirements.
Developers choose Apache Gobblin for its battle-tested scalability, support for both stream and batch execution, and comprehensive feature set including fault tolerance, data quality checking, and a control plane for orchestration. It integrates with existing data systems without replacing them, focusing specifically on data integration and management tasks.
Overview
A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.
Use Cases
Best For
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
