amazon-kinesis-aggregators
A Java framework for creating real-time time series aggregations from Amazon Kinesis streams.
What is amazon-kinesis-aggregators?
Amazon Kinesis Aggregators is a Java framework that automatically creates real-time time series aggregations from data streaming through Amazon Kinesis. It solves the problem of manually building complex aggregation logic by allowing developers to simply configure the data format, time granularity, and grouping criteria, while the framework handles all calculations and persistence.
Java developers and data engineers working with Amazon Kinesis streams who need to generate real-time aggregated metrics and time series data without building custom aggregation pipelines.
Developers choose this framework because it significantly reduces the complexity of implementing real-time aggregations, provides built-in integration with AWS services like DynamoDB and CloudWatch, and offers extensibility for custom data formats and storage backends.
Overview
Amazon Kinesis Aggregators provides a simple way to create real time aggregations of data on Amazon Kinesis.
Use Cases
Best For
- Generating real-time per-second or per-minute metrics from Kinesis streams
- Creating aggregated dashboards in CloudWatch from streaming data
- Building time series databases in DynamoDB from event streams
- Analyzing streaming data with Hive on Amazon EMR or Amazon Redshift
- Extending aggregation logic with custom data extractors or storage backends
- Integrating aggregation capabilities into existing Kinesis consumer applications
Not Ideal For
- New projects requiring active maintenance and future feature development
- Teams operating in multi-cloud or on-premises environments without AWS services
- Developers using programming languages other than Java for stream processing
- Simple aggregation tasks that could be handled with serverless functions like AWS Lambda
Pros & Cons
Pros
Supports CSV, JSON, regex-extracted strings, and object-serialized data, allowing aggregation from diverse stream sources without custom parsing, as detailed in the configuration options.
Built-in persistence to DynamoDB and metrics emission to CloudWatch, streamlining setup for AWS-centric applications and enabling real-time dashboards.
Allows custom implementations for data extraction, data stores, and metrics emitters, as described in the 'Extending Aggregators' section, facilitating adaptation to specific needs.
Aggregates data by second, minute, hour, day, month, year, or custom grouped periods, enabling precise time series analysis with minimal code.
Cons
The framework is no longer actively developed, with only security updates, making it unsuitable for new systems that require long-term support and improvements.
Heavily reliant on AWS services like Kinesis, DynamoDB, and CloudWatch, limiting portability and increasing dependency on a single cloud provider.
Requires detailed JSON configuration files and understanding of AWS deployment options, such as Elastic Beanstalk setup, which can be cumbersome compared to simpler tools.
Frequently Asked Questions
Related Projects
amazon-kinesis-clientClient library for Amazon Kinesis
amazon-kinesis-producerAmazon Kinesis Producer Library
amazon-kinesis-client-pythonAmazon Kinesis Client Library for Python
amazon-kinesis-scaling-utilsThe Kinesis Scaling Utility is designed to give you the ability to scale Amazon Kinesis Streams in the same way that you scale EC2 Auto Scaling groups – up or down by a count or as a percentage of the total fleet. You can also simply scale to an exact number of Shards. There is no requirement for you to manage the allocation of the keyspace to Shards when using this API, as it is done automatically.
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.