Crunch
A Go-based toolkit for fast ETL and feature extraction on Hadoop, optimized for rapid development and execution.
What is Crunch?
Crunch is a Go-based toolkit for building ETL (Extract, Transform, Load) and feature extraction pipelines on Hadoop. It allows developers to define data transformations and feature computations using a simple API, then generates Hadoop-compatible scripts and binaries for processing semi-structured data like JSON logs at scale.
Data engineers and developers working with Hadoop ecosystems who need to build or optimize ETL pipelines for processing large volumes of semi-structured data efficiently.
Developers choose Crunch for its rapid development cycle, seamless Hadoop integration, and the ability to embed custom Go code directly into data workflows, reducing the complexity typically associated with big data processing.
Overview
A fast to develop, fast to run, Go based toolkit for ETL and feature extraction on Hadoop.
Use Cases
Best For
- Processing semi-structured JSON logs in Hadoop environments
- Building custom feature extraction pipelines for machine learning data preparation
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
