Data Processing
Showing 36 of 361 projects
A tiny wrapper around Node.js streams.Transform to simplify stream creation without subclassing.
A curated list of awesome Apache Spark packages, libraries, and resources for data engineers and scientists.
A high-performance Python package for fast, multi-threaded manipulation of large tabular datasets, inspired by R's data.table.
A C++ library for reading, writing, creating, and modifying Microsoft Excel .xlsx files.
A Ruby framework for writing reliable, concise, and maintainable ETL (Extract-Transform-Load) data processing jobs.
A fast, lightweight JSON Query Language CLI tool built with Rust for querying and transforming JSON data.
A distributed map-reduce framework for parallel computations over large datasets on unreliable computer clusters.
A computational parallel flow library for Elixir built on top of GenStage for parallel processing of collections.
A collection of small, chainable command-line utilities for advanced password cracking operations.
A Python library for agile data preparation workflows that works with Pandas, Dask, cuDF, Dask-cuDF, Vaex, and PySpark.
A high-performance streaming CSV parser for Node.js that converts CSV to JSON at up to 90,000 rows per second.
Python library providing clean, chainable functions for data cleaning and manipulation with pandas DataFrames.
A comprehensive benchmark suite for evaluating speed, throughput, and resource utilization of big data frameworks like Hadoop, Spark, and streaming engines.
A suite of high-performance command line tools for filtering, summarizing, joining, and manipulating large tabular data files.
A fast, resilient distributed stream processing framework that simplifies real-time data applications with high performance and easy scaling.
A modern, minimal, and high-performance .NET library for reading and writing CSV/TSV files with zero allocations and SIMD-accelerated parsing.
A high-performance Rust JSON parser porting simdjson's SIMD techniques, with Serde compatibility.
A Python framework for processing spatio-temporal satellite imagery and extracting features for machine learning applications.
A pure Python library for reading and writing ESRI Shapefiles, the popular GIS vector data format.
A Ruby library for reading, writing, and modifying Microsoft Excel-compatible spreadsheet documents (XLS format).
A lean and fast C++ library for 3D point cloud data processing with efficient implementations of common operations.
A high-performance, fully-featured CSV parser and serializer for modern C++ with streaming, random access, and robust format handling.
A Rust library providing streaming compression and decompression for DEFLATE, zlib, and gzip formats with multiple backend options.
A curated list of amazingly awesome Hadoop and Hadoop ecosystem resources.
A Ruby gem for normalizing, formatting, and splitting E164 international phone numbers.
A collection of extra nodes for Node-RED, extending its capabilities with hardware I/O, social APIs, data parsing, and utility functions.
A collection of extra nodes for Node-RED, extending its capabilities with hardware, I/O, social, storage, and utility functions.
A header-only C++11 CSV parser library with easy-to-use API for reading and writing CSV files.
Run lambda functions over S3 objects with concurrency control for data pipelining and analytics.
Run lambda functions over S3 objects with concurrency control for data pipelining and analytics.
A C++ library for parallel text file reading with CSV support and Python bindings.
A fast and friendly R package for reading rectangular data from delimited files like CSV and TSV.
A fast, idiomatic, and dependency-free Go library for mapping between CSV and Go values.
A high-performance C++ JSON serializing and deserializing library accelerated by SIMD instructions.
A REST interface for interacting with Apache Spark from anywhere, enabling remote code execution and job submissions.
A high-performance JSON parser and toolkit for Go, optimized for large and variable datasets.
Related Tags
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.