A Java client library for building scalable, fault-tolerant applications that consume and process data from Amazon Kinesis Data Streams.
Amazon Kinesis Client Library (KCL) is a Java library for building applications that consume and process data from Amazon Kinesis Data Streams. It solves the problem of handling the operational complexities of distributed stream processing, such as load balancing, fault tolerance, and checkpointing, allowing developers to focus on their data processing logic.
Java developers building real-time data processing applications on AWS, particularly those who need to consume high-volume, streaming data from Kinesis Data Streams with reliability and scalability.
Developers choose KCL because it provides a managed, production-ready framework for stream consumption that handles scaling, fault tolerance, and checkpointing automatically, reducing the need to build and maintain custom distributed systems infrastructure.
Client library for Amazon Kinesis
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Dynamically scales processing across workers, supporting manual or auto-scaling without load redistribution, as highlighted in the Scalability feature section of the README.
Provides at-least-once delivery and continuous processing during worker failures, with built-in mechanisms ensuring reliability even in distributed failures.
Handles shard splits and merges seamlessly, maintaining ordering by processing child shards only after parent completion, as described in the Stream-Level Change Handling feature.
Supports Java natively and enables other languages through MultiLangDaemon, allowing diverse development teams to write processors in languages like Python without Java expertise.
KCL 1.x is nearing end-of-support, requiring non-trivial interface and credential provider updates to migrate to 3.x, as warned in the IMPORTANT notice, adding maintenance overhead.
Specific AWS SDK for Java versions (2.27.19-2.27.23) cause DynamoDB exceptions with KCL 3.x, necessitating careful version management and updates, as highlighted in the warning box.
Tightly integrated with Amazon Kinesis and AWS ecosystem, making it unsuitable for multi-cloud or on-premises deployments without significant rework.