Data Pipeline
Showing 11 of 47 projects
A fully asynchronous, futures-based Apache Kafka client library for Rust built on librdkafka.
A fault-tolerant service that persists Kafka log data to cloud storage like S3, GCS, Azure Blob Storage, and OpenStack Swift.
A parallel bulk data loader that transfers data between various storages, databases, NoSQL, and cloud services via plugins.
A Ruby framework for writing reliable, concise, and maintainable ETL (Extract-Transform-Load) data processing jobs.
A federated Big Data orchestration service that simplifies job execution across distributed clusters by abstracting infrastructure complexity.
A command-line utility for processing JSON and JavaScript data, inspired by Perl and Unix tools like sed and awk.
A Java library for declarative JSON-to-JSON transformations using JSON-based specifications.
An open-source Reverse ETL platform for syncing data from warehouses to business tools like Salesforce, HubSpot, and Slack.
A library enabling MongoDB to serve as input source or output destination for Hadoop MapReduce tasks and ecosystem tools.
A unified data pipeline tool for ingestion, transformation with SQL/Python/R, and data quality checks across major platforms.
A high-performance CSV ingestion and generation library for Ruby with C acceleration, designed for real-world data with intelligent defaults.
Related Tags
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.