summingbird
A library for writing MapReduce programs that execute on distributed platforms like Storm and Scalding using Scala/Java collection-like syntax.
What is summingbird?
Summingbird is a library that lets developers write MapReduce programs using Scala or Java collection-like syntax, which can then be executed on distributed platforms like Storm (for real-time processing) and Scalding (for batch processing). It solves the problem of maintaining separate codebases for batch and streaming data pipelines by providing a unified API.
Data engineers and Scala/Java developers building large-scale, fault-tolerant data processing systems that require both batch and real-time capabilities.
Developers choose Summingbird because it abstracts away the complexities of distributed platforms, allows code reuse across batch and streaming contexts, and provides strong fault-tolerance guarantees through its hybrid execution mode.
Overview
Streaming MapReduce with Scalding and Storm
Use Cases
Best For
- Building unified batch and streaming data pipelines with a single codebase
- Aggregating real-time data (e.g., Twitter streams) with Storm
- Processing large-scale batch datasets with Scalding
- Implementing fault-tolerant MapReduce jobs that require hybrid execution
- Simplifying distributed data processing for teams familiar with Scala collections
- Creating production systems that need rock-solid data consistency guarantees
Not Ideal For
- Projects using non-JVM languages like Python or R for data processing
- Teams requiring active maintenance, frequent updates, and modern community support
- Data pipelines needing integration with contemporary frameworks like Apache Spark or Flink
- Small to medium-scale applications where the overhead of distributed systems (e.g., Storm setup) is unjustified
Pros & Cons
Pros
Enables writing data processing logic once and running it on both batch (Scalding) and real-time (Storm) engines, as demonstrated in the word count example where the same code works for multiple modes.
Uses Scala/Java collection transformations like flatMap and sumByKey, making code intuitive for developers; the README shows how Summingbird code closely mirrors native Scala collections.
Supports hybrid execution that combines batch and real-time processing for robust data consistency, offering production-ready primitives to build fault-tolerant systems as highlighted in the features.
Abstracts underlying platforms such as Storm and Scalding, allowing business logic to remain decoupled from execution engines, which simplifies code reuse across different processing contexts.
Cons
Marked as 'retired' in the README badge, indicating no active development or support, which poses significant risks for long-term project viability and bug fixes.
Getting started requires installing and configuring multiple external services like Memcached and Storm, as detailed in the example setup, adding overhead compared to more integrated solutions.
Only integrates with Storm and Scalding, excluding newer and more widely adopted frameworks like Apache Spark or Apache Flink, which limits flexibility and ecosystem benefits.
Exclusively supports Scala and Java, with no support for other programming languages, restricting adoption for teams using diverse tech stacks.
Frequently Asked Questions
Related Projects
Apache SupersetApache Superset is a Data Visualization and Data Exploration Platform
PlotlyData Apps & Dashboards for Python. No JavaScript Required.
bokehInteractive Data Visualization in the browser, from Python
ziplineZipline, a Pythonic Algorithmic Trading Library
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.