How does KPL compare to using the AWS SDK for Kinesis directly?

KPL abstracts low-level complexities like batching, retries, and shard discovery, which you'd need to implement manually with the SDK. It's better for high-throughput, reliable producers, but adds more dependency and configuration overhead.

How to implement back-pressure in Amazon Kinesis Producer Library?

The README provides a sample using `kpl.getOutstandingRecordsCount()` to check buffer levels and sleep threads when limits are exceeded. You must customize this based on your record sizes, rates, and host memory to prevent crashes.

Is KPL compatible with Kinesis Data Firehose?

No, KPL is designed specifically for Kinesis Data Streams, not Firehose. Firehose has its own producer APIs, so you'd need to use different libraries or direct integrations for that service.

What are the recommended settings for large Kinesis streams?

For streams over 800 shards, the README suggests using ThreadingModel='POOLED', MetricsGranularity='stream', and ThreadPoolSize=128 to optimize CPU usage and reduce delays, but testing is advised due to varying use cases.

How to migrate from KPL 0.x to 1.x without changing data logic?

AWS claims migration doesn't require data processing changes, but you must update dependencies and configurations, such as moving to AWS SDK for Java 2.x, and test thoroughly, as per the migration guide linked in the README.

Does KPL support schema validation or registry?

Yes, version 0.14.2 added support for Glue Schema Registry, allowing schema serialization, compression, and auto-registration, but this adds complexity and dependency on AWS Glue.

amazon-kinesis-producer

Apache-2.0C++v1.0.7

A Java library for building efficient and reliable producer applications for Amazon Kinesis Data Streams.

GitHub

What is amazon-kinesis-producer?

The Amazon Kinesis Producer Library (KPL) is a Java library that simplifies building high-performance, reliable data producers for Amazon Kinesis Data Streams. It abstracts low-level streaming complexities by handling tasks like automatic batching, retry logic, shard discovery, and back-pressure management, allowing developers to focus on application logic.

Target Audience

Java developers building data ingestion pipelines for Amazon Kinesis Data Streams, particularly those needing high throughput, reliability, and operational simplicity. It is suited for teams managing streaming data from applications, IoT devices, or logs into KDS.

Value Proposition

Developers choose KPL over manual implementations because it provides a robust, scalable foundation with built-in optimizations like batching for cost efficiency, configurable retries for reliability, and automatic shard discovery for stream scalability, reducing development time and operational overhead.

Overview

Amazon Kinesis Producer Library

Use Cases

Best For

Building high-throughput data ingestion pipelines into Amazon Kinesis Data Streams with automatic batching and retries.

Related Projects

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

Submit a project Star on GitHub

414 stars343 forks0 contributors

Java applications requiring reliable streaming producers that handle back-pressure to prevent memory exhaustion under high load.

Scaling Kinesis Data Streams beyond 800 shards with optimized configurations for efficient CPU usage and record writing.

Migrating from KPL 0.x to 1.x without changes to data processing logic, as per AWS end-of-support timelines.

Producers needing visibility into performance via CloudWatch metrics for monitoring buffer health and throughput.

Applications leveraging Kinesis features like StreamARN for enhanced availability or cross-account access without code modifications.

Not Ideal For

Non-Java development ecosystems where producers are built in languages like Python or Go
Multi-cloud or vendor-agnostic architectures needing to stream data to non-AWS services
Low-volume data ingestion where the overhead of batching and retry logic is unnecessary compared to direct API calls
Projects requiring Windows deployment, as support was dropped from version 0.14.0 onwards

Pros & Cons

Pros

Efficient Automatic Batching

Aggregates multiple user records into fewer Kinesis records, reducing costs and improving throughput, as highlighted in the key features for optimizing data ingestion.

Robust Retry Logic

Implements configurable retries with backoff for failed PutRecords requests, enhancing reliability without manual intervention, as specified in the README's features.

Scalable Shard Discovery

Uses the ListShards API to efficiently adapt to stream scaling events, supporting 100 TPS per stream compared to 10 TPS with DescribeStream, per the version 0.14.0 update notes.

Comprehensive Monitoring

Emits CloudWatch metrics for visibility into producer performance and buffer health, enabling proactive management of streaming pipelines.

Cons

Manual Back-pressure Implementation

Requires developers to add custom back-pressure logic to prevent memory exhaustion and crashes, as the README warns that the KPL can consume enough memory to crash itself without proper safeguards.

Forced Version Migration

KPL 0.x is reaching end-of-support in 2026, forcing users to migrate to 1.x, which adds operational burden despite claims of backward compatibility, as noted in the important notice.

Platform Limitations

Windows support was discontinued from version 0.14.0, restricting deployment options for teams using Windows-based infrastructure.

Frequently Asked Questions

Home

Amazon Web Services

amazon-kinesis-client

Client library for Amazon Kinesis

Stars659

Forks483

Last commit3 days ago

amazon-kinesis-client-python

Amazon Kinesis Client Library for Python

Stars376

Forks228

Last commit3 days ago

amazon-kinesis-scaling-utils

The Kinesis Scaling Utility is designed to give you the ability to scale Amazon Kinesis Streams in the same way that you scale EC2 Auto Scaling groups – up or down by a count or as a percentage of the total fleet. You can also simply scale to an exact number of Shards. There is no requirement for you to manage the allocation of the keyspace to Shards when using this API, as it is done automatically.

Stars336

Forks85

Last commit2 years ago

amazon-kinesis-connectors

The Amazon Kinesis Connector Library is a Java framework that simplifies the integration of Amazon Kinesis data streams with various storage and analytics services. It provides a structured pipeline for processing, transforming, and emitting streaming data to destinations such as DynamoDB, Redshift, S3, and Elasticsearch, enabling real-time data workflows. ## Key Features - **Modular Pipeline** — Implements interfaces for transformation, filtering, buffering, and emission to define custom data flows. - **Pre-built Connectors** — Includes ready-to-use connectors for AWS DynamoDB, Redshift, S3, and Elasticsearch. - **Batch Processing** — Buffers records based on configurable thresholds (count, size, time) for efficient batch writes. - **Custom Transformations** — Supports user-defined data models and serializers via the ITransformer interface. - **Sample Implementations** — Provides complete sample applications with Ant/Maven build files for each connector type. ## Philosophy The library emphasizes a decoupled, extensible architecture where developers can plug in custom logic for each stage of the data pipeline, promoting flexibility and reuse in stream processing applications.

Stars328

Forks187

Last commit5 years ago

#java-library

#real-time-processing

Amazon Web Services14.0k