Data Ingestion
Showing 36 of 65 projects
An open-source framework for building LLM-powered applications with data ingestion, indexing, and retrieval capabilities.
A scalable time series database optimized for real-time metrics, events, and analytics with fast query response.
A realtime distributed messaging platform designed to operate at scale, handling billions of messages per day.
An open-source time-series database for high-speed ingestion and low-latency SQL queries.
An enterprise-grade event streaming platform that ingests, processes, and manages real-time event data with PostgreSQL compatibility and Apache Iceberg™ integration.
Enterprise-grade event streaming platform that continuously ingests, processes, and serves real-time data with Apache Iceberg™ integration.
A CLI tool to copy data between any databases and platforms with a single command, no code required.
Build concurrent, multi-stage data ingestion and processing pipelines with Elixir, supporting back-pressure, batching, and fault tolerance.
A distributed service for efficiently collecting, aggregating, and moving large amounts of log-like data.
A real-time distributed analytical database built entirely on bitmaps for low-latency queries on fresh data.
A distributed data integration framework for big data ecosystems, handling ingestion, replication, organization, and lifecycle management for both streaming and batch data.
A distributed data integration framework for big data ecosystems, handling ingestion, replication, organization, and lifecycle management for both streaming and batch data.
A unified data pipeline tool for ingestion, transformation with SQL/Python/R, and data quality checks across major platforms.
A high-performance CSV ingestion and generation library for Ruby with C acceleration, designed for real-world data with intelligent defaults.
A one-stop, full-scenario integration framework for massive data, supporting data ingestion, synchronization, and subscription.
Official Java client library for InfluxDB 1.x, enabling Java applications to write and query time series data.
A C++ library for parallel text file reading with CSV support and Python bindings.
LinkedIn's previous generation Kafka to HDFS pipeline for batch data ingestion.
A high-performance, zero-dependency JavaScript client library for InfluxDB v1.x, compatible with Node.js and browsers.
A data pipeline engine for security teams to collect, transform, enrich, and route telemetry data at scale.
A fast, low-overhead metric database written in pure Erlang, optimized for time-series data storage and querying.
An AWS Lambda function that automatically loads files from S3 into Amazon Redshift clusters with zero server administration.
A simple collector that batches many small ClickHouse inserts into larger bulk inserts for improved performance.
A PHP client library for reading from and writing to InfluxDB 1.x time series databases.
A Java library for building efficient and reliable producer applications for Amazon Kinesis Data Streams.
A proxy service that offloads event processing, normalization, and ingestion from Sentry SDKs and server.
Sample AWS Lambda functions for streaming data from S3 and Kinesis into Amazon Elasticsearch Service.
An open-source toolkit for building a Unified Namespace to ingest, contextualize, and store factory data for Industrial IoT platforms.
Official Ruby client library for InfluxDB 1.x, providing data writing, querying, and administrative capabilities.
A unified platform for big data stream and batch processing on Hadoop YARN with enterprise-grade operability.
A lightweight time-series database written in Rust, deployable as an embedded library, standalone server, or scalable cluster.
A serverless reference architecture for building an IoT backend using AWS Lambda and IoT Core to ingest, process, and alert on sensor data.
An Elixir driver for InfluxDB supporting both v1.x and v2.x versions with Flux and InfluxQL query capabilities.
A high-performance Kafka consumer written in Python that reads metrics from Kafka and writes them to InfluxDB.
A .NET library for efficiently sending time series data to InfluxDB 1.x using the Line Protocol.
Go client library for connecting to InfluxDB 1.x time series databases.
Related Tags
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.