How to set up Amazon KCL for Node.js from scratch?

Install the npm package, ensure Java is installed, and use the kcl-bootstrap script with a properties file. The README provides step-by-step samples, including configuring credentials and running producers and consumers.

Amazon KCL Node.js vs AWS SDK for Kinesis – which one should I use?

Use KCL for distributed, fault-tolerant processing with automatic scaling and checkpointing; it handles shard management. Use the SDK for simpler, single-instance consumers where you need direct control over API calls.

How to handle errors in the KCL Node.js record processor?

Implement error handling in callbacks and ensure idempotency, as exceptions should not be thrown to the KCL. The README example shows checkpointing with error callbacks in processRecords to continue processing on failures.

What IAM permissions are needed for KCL Node.js 3.x?

It requires permissions for Kinesis, DynamoDB (including new worker metrics and coordinator tables), and CloudWatch. Refer to AWS documentation for detailed policies, as migration to KCL 3.x introduces new resources.

Can I use KCL Node.js with Apache Kafka?

No, it's designed specifically for Amazon Kinesis. For Kafka, consider other Node.js libraries like kafka-node or the official Kafka client, as KCL relies on AWS-native APIs and services.

How to scale KCL applications horizontally?

Deploy multiple consumer instances; KCL automatically balances shards based on load and CPU utilization, as per the managed scalability feature and KCL 3.x's new lease assignment algorithm.

amazon-kinesis-client-nodejs

Apache-2.0JavaScriptv3.1.1

A Node.js interface to the Amazon Kinesis Client Library for building distributed applications that process streaming data at scale.

GitHub

What is amazon-kinesis-client-nodejs?

The Amazon Kinesis Client Library (KCL) for Node.js is a managed interface to the KCL MultiLangDaemon that enables developers to build robust, distributed applications for processing streaming data from Amazon Kinesis. It abstracts the complexities of distributed computing, such as load balancing, checkpointing, and shard management, allowing developers to focus on implementing custom record processing logic. The library leverages a Java-based daemon to handle heavy lifting while providing a simple Node.js API for processing records.

Target Audience

Node.js developers building distributed, real-time data processing applications on AWS Kinesis Data Streams who need reliable, scalable stream consumption with managed fault tolerance. It is suited for teams that want to avoid implementing low-level distributed system concerns like shard coordination and checkpointing themselves.

Value Proposition

Developers choose this library because it provides a production-ready, AWS-managed solution for consuming Kinesis streams with automatic scaling, built-in checkpointing, and graceful handling of failures and shard rebalancing. Its unique selling point is the language-agnostic MultiLangDaemon architecture, which offers identical features and a consistent parallel processing model as other KCL language implementations (Java, Python, Ruby), while exposing a simple Node.js interface.

Overview

Amazon Kinesis Client Library for Node.js

Use Cases

Best For

Related Projects

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

Submit a project Star on GitHub

306 stars200 forks0 contributors

Building scalable real-time analytics pipelines that process high-volume streaming data from Kinesis with guaranteed exactly-once processing semantics.
Implementing fault-tolerant consumer applications that require automatic recovery from instance failures and dynamic scaling with stream volume changes.
Developing distributed data processors where you need managed shard lifecycle handling, including automatic adaptation to shard splits and merges.
Creating Node.js microservices that consume Kinesis streams and need built-in checkpointing to ensure reliable record processing without data loss.
Migrating existing KCL applications from other languages (Java, Python, Ruby) to Node.js while maintaining the same processing model and operational guarantees.
Teams that want to leverage AWS best practices for stream processing without building custom coordination logic for load balancing and lease management.

Not Ideal For

Environments where Java is prohibited or unavailable, due to the mandatory Java-based MultiLangDaemon dependency.
Low-volume streaming use cases where the overhead of distributed coordination is unnecessary; direct AWS SDK calls are simpler.
Teams requiring fine-grained control over processing lifecycle without the KCL's abstraction layer for shard management and checkpointing.
Applications consuming data from multiple cloud providers or non-AWS streaming platforms like Apache Kafka.

Pros & Cons

Pros

Managed Scalability and Fault Tolerance

Automatically handles load balancing across instances and responds to failures, as described in the Key Features for managed scalability and graceful shutdown, minimizing data reprocessing.

Built-in Checkpointing for Reliability

Provides checkpointing mechanisms to ensure exactly-once processing, with example code in the README showing how to checkpoint sequence numbers in the record processor.

Automatic Shard Management

Manages Kinesis shard lifecycle including splits and merges, ensuring continuous record processing without manual intervention, as highlighted in the shard management feature.

Cross-Language Consistency

Leverages the MultiLangDaemon for identical features and parallel processing model across languages, per the philosophy section, easing migration from other KCL implementations.

Cons

Java Runtime Dependency

Requires Java 1.8 or higher installed, adding complexity to deployment and environment setup, as noted in the prerequisites, which can be a barrier in pure Node.js stacks.

Complex Bootstrap and Configuration

Involves running a custom bootstrap script with properties files and managing I/O protocols, making setup more cumbersome than simpler libraries, as seen in the running instructions.

Debugging and Logging Challenges

Application logs must be redirected to avoid mingling with MultiLangDaemon stdout/stderr, complicating debugging, as warned in the notes on running the data processor.

Vendor Lock-in to AWS

Tightly coupled with Amazon Kinesis and AWS services like DynamoDB and CloudWatch, limiting portability to other cloud providers or streaming platforms.

Frequently Asked Questions

Home

Amazon Web Services

amazon-kinesis-client

Client library for Amazon Kinesis

Stars660

Forks483

Last commit2 days ago

amazon-kinesis-producer

Amazon Kinesis Producer Library

Stars414

Forks343

Last commit2 months ago

amazon-kinesis-client-python

Amazon Kinesis Client Library for Python

Stars376

Forks228

Last commit17 days ago

amazon-kinesis-scaling-utils

The Kinesis Scaling Utility is designed to give you the ability to scale Amazon Kinesis Streams in the same way that you scale EC2 Auto Scaling groups – up or down by a count or as a percentage of the total fleet. You can also simply scale to an exact number of Shards. There is no requirement for you to manage the allocation of the keyspace to Shards when using this API, as it is done automatically.

Stars336

Forks84

Last commit2 years ago

Amazon Web Services14.0k