Apache DataFu
A collection of libraries for large-scale data processing in Hadoop ecosystems, including Spark, Pig, and incremental MapReduce.
What is Apache DataFu?
Apache DataFu is a collection of libraries for working with large-scale data in Hadoop ecosystems. It provides utilities and frameworks for data processing, including extensions for Apache Spark and Pig, as well as an incremental processing framework for MapReduce. The project solves the need for stable, well-tested tools for data mining and statistics in big data environments.
Data engineers and scientists working with Hadoop, Spark, or Pig who require reliable libraries for data transformation, mining, and incremental processing. It is particularly useful for teams building data pipelines or analytical applications at scale.
Developers choose Apache DataFu for its Apache-licensed, production-ready libraries that are specifically designed for Hadoop-based workflows. Its unique selling point is the combination of Spark and Pig utilities with an incremental processing framework, all backed by rigorous testing and community support.
Overview
Mirror of Apache DataFu
Use Cases
Best For
- Extending Apache Spark with custom user-defined functions and utilities
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
