Batch Processing
Showing 36 of 120 projects
LinkedIn's previous generation Kafka to HDFS pipeline for batch data ingestion.
A self-hosted web platform for distributed video encoding using HandBrake across multiple headless devices.
A Go library providing limited and unlimited goroutine pools for easier concurrent task handling and cancellation.
A command-line batch file renaming utility with regex support, sorting, and mapping capabilities for developers.
A visual GUI tool for packing and managing texture atlases for the libGDX game framework.
Convert CSV files into Markdown tables with customizable padding and delimiter support.
A unified resource scheduler for co-scheduling batch, stateless, and stateful workloads in a single cluster to maximize resource utilization.
A simple, lightweight batch processing framework for Java designed for ETL jobs.
An AWS Lambda function that automatically loads files from S3 into Amazon Redshift clusters with zero server administration.
A command-line tool combining xargs and awk with regex pattern matching for batch processing.
A powerful, feature-rich queue library for Node.js with persistent storage, batch processing, prioritization, and task management.
A simple collector that batches many small ClickHouse inserts into larger bulk inserts for improved performance.
A Grunt plugin that automates image optimization using ImageOptim, ImageAlpha, and JPEGmini for Mac.
A Node.js image thumbnailing service using AWS SQS/S3 and ImageMagick for scalable background processing.
A PowerShell function for simplified parallel execution using runspaces to speed up scripts.
A Neo4j-based library for building high-performance recommendation engines with real-time and pre-computed capabilities.
A RESTful engine for orchestrating sequential Docker container workflows, marshaling data between steps.
A command-line tool for quick and simple image processing operations like convert, crop, resize, and watermark.
A unified platform for big data stream and batch processing on Hadoop YARN with enterprise-grade operability.
A Java library for building data pipelines that connect Amazon Kinesis streams to AWS and non-AWS services like DynamoDB, Redshift, S3, and Elasticsearch.
A Bash command-line tool that converts filenames and directory names to web-friendly slugs.
A powerful, reliable background job processing library for Clojure with pluggable message brokers and a management console.
A desktop tool that converts C code to Delphi/Pascal in real-time with syntax checking and execution capabilities.
A simple tool for API load testing and batch jobs with configurable rate, concurrent workers, and protocol-agnostic design.
A Python library for building lazy data processing and machine learning workflows that handle datasets larger than memory.
An image processing framework for batch operations using OpenFX plugins, with tools for movies and file sequences.
A Go-based toolset for data extraction, transformation, and loading, providing powerful data synchronization capabilities.
A command-line tool for parsing, searching, and analyzing Windows Registry hives with batch processing and forensic capabilities.
A cross-platform CLI tool for batch renaming files with variable replacement, templating, and undo capabilities.
An opinionated Elixir wrapper around the Brod Kafka client with built-in support for Heroku Kafka and encrypted connections.
A Ruby gem providing ActiveRecord batch methods for plucking columns, reducing SQL queries and memory usage by up to 50%.
A CLI tool that wraps FFmpeg to simplify merging multiple video and audio files with automatic chapter creation.
A collection of robust and fast Python tools for parsing, extracting, and analyzing web archive data, including a high-performance WARC parser.
An open-source enterprise job scheduling system for automating and managing batch jobs and workflows.
A tool that calculates a GitHub repository's quality score based on engineering best practices using metadata and source analysis.
Immutable key/value store with efficient space utilization and fast reads, ideal for batch-built tables shipped to multiple servers.
Related Tags
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.