Batch Processing
Showing 36 of 74 projects
LinkedIn's previous generation Kafka to HDFS pipeline for batch data ingestion.
A self-hosted web platform for distributed video encoding using HandBrake across multiple headless devices.
A Go library providing limited and unlimited goroutine pools for easier concurrent task handling and cancellation.
A command-line batch file renaming utility with regex support, sorting, and mapping capabilities for developers.
A visual GUI tool for packing and managing texture atlases for the libGDX game framework.
Convert CSV files into Markdown tables with customizable padding and delimiter support.
A unified resource scheduler for co-scheduling batch, stateless, and stateful workloads in a single cluster to maximize resource utilization.
A simple, lightweight batch processing framework for Java designed for ETL jobs.
An AWS Lambda function that automatically loads files from S3 into Amazon Redshift clusters with zero server administration.
A command-line tool combining xargs and awk with regex pattern matching for batch processing.
A powerful, feature-rich queue library for Node.js with persistent storage, batch processing, prioritization, and task management.
A simple collector that batches many small ClickHouse inserts into larger bulk inserts for improved performance.
A Grunt plugin that automates image optimization using ImageOptim, ImageAlpha, and JPEGmini for Mac.
A Node.js image thumbnailing service using AWS SQS/S3 and ImageMagick for scalable background processing.
A PowerShell function for simplified parallel execution using runspaces to speed up scripts.
A Neo4j-based library for building high-performance recommendation engines with real-time and pre-computed capabilities.
A RESTful engine for orchestrating sequential Docker container workflows, marshaling data between steps.
A command-line tool for quick and simple image processing operations like convert, crop, resize, and watermark.
A unified platform for big data stream and batch processing on Hadoop YARN with enterprise-grade operability.
A Java library for building data pipelines that connect Amazon Kinesis streams to AWS and non-AWS services like DynamoDB, Redshift, S3, and Elasticsearch.
A Bash command-line tool that converts filenames and directory names to web-friendly slugs.
A powerful, reliable background job processing library for Clojure with pluggable message brokers and a management console.
A desktop tool that converts C code to Delphi/Pascal in real-time with syntax checking and execution capabilities.
A simple tool for API load testing and batch jobs with configurable rate, concurrent workers, and protocol-agnostic design.
A Python library for building lazy data processing and machine learning workflows that handle datasets larger than memory.
A Go-based toolset for data extraction, transformation, and loading, providing powerful data synchronization capabilities.
An image processing framework for batch operations using OpenFX plugins, with tools for movies and file sequences.
A command-line tool for parsing, searching, and analyzing Windows Registry hives with batch processing and forensic capabilities.
A cross-platform CLI tool for batch renaming files with variable replacement, templating, and undo capabilities.
An opinionated Elixir wrapper around the Brod Kafka client with built-in support for Heroku Kafka and encrypted connections.
A Ruby gem providing ActiveRecord batch methods for plucking columns, reducing SQL queries and memory usage by up to 50%.
A CLI tool that wraps FFmpeg to simplify merging multiple video and audio files with automatic chapter creation.
A collection of robust and fast Python tools for parsing, extracting, and analyzing web archive data, including a high-performance WARC parser.
An open-source enterprise job scheduling system for automating and managing batch jobs and workflows.
A tool that calculates a GitHub repository's quality score based on engineering best practices using metadata and source analysis.
Immutable key/value store with efficient space utilization and fast reads, ideal for batch-built tables shipped to multiple servers.
Related Tags
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.