Data Pipeline
Showing 36 of 131 projects
A distributed event streaming platform for building high-performance data pipelines, streaming analytics, and data integration.
A high-performance, end-to-end observability data pipeline for collecting, transforming, and routing logs and metrics.
A high-performance, end-to-end observability data pipeline for collecting, transforming, and routing logs and metrics.
A curated list of awesome open-source libraries for deploying, monitoring, versioning, and scaling production machine learning systems.
A curated list of awesome open-source libraries for deploying, monitoring, versioning, and scaling production machine learning systems.
A plugin-driven agent for collecting, processing, aggregating, and writing metrics, logs, and arbitrary data.
A server-side data processing pipeline that ingests, transforms, and ships logs and events from multiple sources.
A curated list of awesome big data frameworks, resources, and tools across various categories.
An open-source log collector that unifies logging infrastructure by collecting events from various sources and routing them to multiple destinations.
A low-latency platform for change data capture (CDC) that streams row-level changes from databases to applications.
A practical booklet covering the four main steps of designing machine learning systems with 27 interview questions.
A polyglot document intelligence framework with a Rust core for extracting text, metadata, and structured data from 91+ file formats.
A high-performance, resilient stream processor that connects various sources and sinks, performs data transformations, and guarantees at-least-once delivery.
A high-performance, declarative stream processor that connects various sources and sinks with built-in data transformation capabilities.
An open-source ETL (Extract, Transform, Load) tool for data integration and migration.
Open-source customer data infrastructure that collects, validates, and enriches behavioral event data for AI and analytics.
Open-source data pipelines to sync cloud infrastructure metadata from AWS, Azure, GCP, and 70+ sources into your data warehouse.
Open-source data pipelines for cloud asset inventory, CSPM, FinOps, and vulnerability management across AWS, Azure, GCP, and 70+ sources.
An easy-to-use, powerful, and reliable system to process and distribute data across cybersecurity, observability, and AI pipelines.
A lightweight command-line tool for producing, consuming, and inspecting Apache Kafka messages, similar to netcat for Kafka.
A lightweight, non-JVM command-line tool for producing, consuming, and inspecting Apache Kafka messages.
A distributed data streaming engine with stateful stream processing for building responsive data-intensive applications.
A lightweight, fast, and beautiful database management tool with AI-powered chat interface for PostgreSQL, MySQL, SQLite, MongoDB, Redis, and more.
An open-source, privacy-focused customer data platform (CDP) that collects, processes, and routes event data to warehouses and tools.
A lightweight, efficient, and fast high-level web crawling and scraping framework for .NET.
A CLI tool to copy data between any databases and platforms with a single command, no code required.
Self-hostable workflow orchestrator for teams whose main work isn't orchestration. Declarative YAML over your scripts, SSH commands, containers, etc; keep workflows separate from business logic. One binary, no database, runs on limited H/W resources. Alternative to Airflow / Cron / Job Scheduler.
A local-first, single-binary workflow orchestration engine that runs declarative DAGs from laptop to distributed cluster.
A fast, embeddable scripting language for Go applications, compiled to bytecode and executed on a stack-based VM.
A source-agnostic distributed change data capture system for reliably capturing and streaming primary data changes.
An AI-native modular infrastructure for quantitative trading, featuring a weight-centric architecture for building, testing, and deploying algorithmic strategies.
A curated list of awesome streaming frameworks, applications, readings, and resources for stream processing.
A Kubernetes-native, serverless platform for running massively parallel data and streaming jobs with exactly-once semantics.
Build concurrent, multi-stage data ingestion and processing pipelines with Elixir, supporting back-pressure, batching, and fault tolerance.
A Scala API for Apache Beam and Google Cloud Dataflow, enabling unified batch and streaming data processing.
A single C++ binary SQL engine for high-performance stream processing, analytics, observability, and AI/ML pipelines.
Related Tags
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.