amazon-kinesis-scaling-utils
A utility for scaling Amazon Kinesis Streams manually or automatically, similar to EC2 Auto Scaling.
What is amazon-kinesis-scaling-utils?
Amazon Kinesis Scaling Utility is an open-source tool that enables dynamic scaling of Amazon Kinesis Streams. It allows users to manually adjust shard counts or deploy an auto-scaling service that monitors stream metrics and automatically scales capacity up or down based on demand. It solves the problem of manually managing Kinesis stream sharding to match fluctuating data ingestion or consumption rates.
AWS developers and data engineers who manage Kinesis Streams with variable throughput and need to optimize cost and performance through automated scaling.
Developers choose this utility because it provides a programmatic, auto-scaling approach to Kinesis stream management—similar to EC2 Auto Scaling—reducing operational overhead and ensuring streams have adequate capacity without manual intervention.
Overview
The Kinesis Scaling Utility is designed to give you the ability to scale Amazon Kinesis Streams in the same way that you scale EC2 Auto Scaling groups – up or down by a count or as a percentage of the total fleet. You can also simply scale to an exact number of Shards. There is no requirement for you to manage the allocation of the keyspace to Shards when using this API, as it is done automatically.
Use Cases
Best For
- Automatically scaling Kinesis Streams based on PUT or GET rate thresholds
- Reducing Kinesis costs by scaling down underutilized streams
- Handling sudden spikes in data ingestion to Kinesis
- Managing multiple Kinesis Streams with centralized auto-scaling policies
- Simplifying shard management without manual keyspace allocation
- Integrating Kinesis scaling with SNS for operational alerts
Not Ideal For
- Streams with static, predictable throughput where manual scaling via AWS Console is sufficient
- Organizations mandating AWS-native managed services for compliance or support reasons
- Teams using infrastructure-as-code tools like Terraform that prefer integrated, declarative scaling modules
- Applications requiring sub-minute scaling latency, as checks are interval-based (e.g., every 300 seconds)
Pros & Cons
Pros
Supports manual scaling by shard count or percentage and automatic scaling with configurable thresholds, cool-off periods, and SNS notifications via JSON, similar to EC2 Auto Scaling patterns.
Handles shard splitting and merging automatically during scaling operations, eliminating the need for manual partitioning and reducing operational complexity.
Scales based on both PUT and GET utilization rates, with configurable logic for combined metrics, ensuring comprehensive capacity adjustment for varied workloads.
Provides pre-built WAR files deployable on Elastic Beanstalk or any Java app server, with region-specific S3 URLs for quick setup and reduced manual effort.
Cons
Requires deploying a Java web application and managing detailed JSON configuration files, which is more involved than using AWS native tools or simple scripts.
Scaling decisions rely on periodic metric checks (e.g., configurable intervals like 300 seconds), so it may not instantly respond to rapid traffic spikes.
The README highlights specific behaviors in version .9.8.8, such as scalePct logic, which could lead to confusion and require careful testing and validation.
Frequently Asked Questions
Related Projects
amazon-kinesis-clientClient library for Amazon Kinesis
amazon-kinesis-producerAmazon Kinesis Producer Library
amazon-kinesis-client-pythonAmazon Kinesis Client Library for Python
amazon-kinesis-connectorsThe Amazon Kinesis Connector Library is a Java framework that simplifies the integration of Amazon Kinesis data streams with various storage and analytics services. It provides a structured pipeline for processing, transforming, and emitting streaming data to destinations such as DynamoDB, Redshift, S3, and Elasticsearch, enabling real-time data workflows. ## Key Features - **Modular Pipeline** — Implements interfaces for transformation, filtering, buffering, and emission to define custom data flows. - **Pre-built Connectors** — Includes ready-to-use connectors for AWS DynamoDB, Redshift, S3, and Elasticsearch. - **Batch Processing** — Buffers records based on configurable thresholds (count, size, time) for efficient batch writes. - **Custom Transformations** — Supports user-defined data models and serializers via the ITransformer interface. - **Sample Implementations** — Provides complete sample applications with Ant/Maven build files for each connector type. ## Philosophy The library emphasizes a decoupled, extensible architecture where developers can plug in custom logic for each stage of the data pipeline, promoting flexibility and reuse in stream processing applications.
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.