Data Pipeline
Showing 36 of 131 projects
A Java library for building data pipelines that connect Amazon Kinesis streams to AWS and non-AWS services like DynamoDB, Redshift, S3, and Elasticsearch.
A Kotlin library for extracting path-based code representations and ASTs from multiple languages to prepare code for machine learning models.
A curated reference hub of tools and real-world examples for designing effective threat detection and response pipelines.
A Go library for declarative JSON-to-JSON transformations using JSON specifications.
An idiomatic Clojure machine learning library providing a unified interface for classification, regression, and unsupervised models.
Docker image for Logstash 1.4.5 with optional Elasticsearch 1.7.0 and Kibana 3.1.2 integration.
A Go-based toolkit for fast ETL and feature extraction on Hadoop, optimized for rapid development and execution.
An R package providing a toolbox of pipeline-friendly functions for manipulating and querying non-tabular data stored in list objects.
A monolith codebase that powers the Iowa Environmental Mesonet's environmental data ingest, processing, and web services.
An R package providing multiple pipeline styles (operator, object, function) for readable function chaining and data transformation.
A CLI tool to send GeoJSON files to geojson.io for instant visualization and editing.
A Go library for building data processing workflows and pipelines with functional operations, cycles, and fan-out capabilities.
A visual development platform for building, deploying, and managing streaming analytics applications with multiple engine bindings.
Serverless data pipeline for crawling PDFs from the web and extracting structured data using AWS Textract.
Python utilities for parallel uploads and downloads to Amazon S3 using multipart uploads and range requests.
Collect, validate, and send ROS 2 data to build APIs and dashboards with reliable data pipelines.
A universal data converter that translates JSON, BSON, YAML, CSV, XML, and MT940 to any format using Go templates.
An open-source data pipeline that aggregates and standardizes heterogeneous public COVID-19 data from multiple global sources.
A Python framework for building and deploying serverless data and ML pipelines on AWS using AWS CDK.
A modern analytics pipeline for tracking and analyzing GitHub contributions across repositories with AI-powered summaries and leaderboards.
A buffered output plugin for Fluentd that sends time-series data to InfluxDB.
An open-source framework for developing large-scale anomaly detection models using Apache Spark.
A Logstash input plugin that reads data from DynamoDB tables via table scans and DynamoDB Streams for near real-time data processing.
A Docker container providing a complete streaming environment for experimenting with Kafka, Spark Streaming, and Cassandra.
A Flume NG sink that writes event data to MongoDB with flexible configuration for collection mapping and data transformation.
A multi-threaded Python example demonstrating how to produce and consume records from Amazon Kinesis streams.
fill-pg is a tool that populates PostgreSQL with blockchain data from EOSIO's State History Plugin for monitoring applications.
A Log4J appender for publishing Java application logs to Amazon Kinesis streams with buffering and retry capabilities.
Streams PostgreSQL database events to Kafka using logical replication and can also consume events from Kafka.
A Go package providing a simplistic implementation of pipelines using goroutines for concurrent data processing.
A Logstash output plugin that sends metrics from Logstash pipelines to InfluxDB time-series databases.
A Node.js module for instrumenting streams to provide debugging, monitoring, and provenance tracking in data pipelines.
A blazing-fast, fully-automated tool for loading large CSV files into MySQL or PostgreSQL databases with parallel processing.
A Python framework for building serverless data pipelines on AWS with Airflow-like elegance.
A Go library for concurrent processing that returns output in the same order as the input via channels.
A modular framework for ingesting and processing Algorand blockchain data into external applications.
Related Tags
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.