How to set up SSL connections with Kafka-node?

Configure the sslOptions in KafkaClient with TLS settings, such as { rejectUnauthorized: false }, and for SASL/PLAIN, use the sasl object with username and password. Details are in the KafkaClient options section.

Kafka-node vs kafkajs: which is better for Node.js?

Kafka-node is more mature and feature-rich with administrative APIs and stream support, but kafkajs is newer, written in pure JavaScript, and often preferred for simpler setups and TypeScript. Choose based on feature needs versus modernization.

How to handle consumer group rebalancing in Kafka-node?

Use ConsumerGroup with sessionTimeout and protocol options, and the onRebalance callback for custom logic before rebalance finishes. The README provides examples and configuration tips for smooth transitions.

Troubleshooting snappy installation errors in Kafka-node?

Install with --no-optional flag to skip snappy, or ensure system dependencies for node-gyp are met. Snappy is optional but required for snappy compression; errors are common on Windows, as noted in the FAQ.

How to produce and consume binary data in Kafka-node?

Set encoding to 'buffer' in consumer options, and for producers, pass messages as Buffer objects. Avoid TypedArrays; convert them to Buffer, as explained in the troubleshooting section for binary data.

Best practices for error handling in Kafka-node consumers?

Always listen to the 'error' event on consumers and clients. Use the debug module with DEBUG=kafka-node:* for logs, and refer to the troubleshooting guide for issues like offsetOutOfRange or stuck partitions.

Open-Awesome

kafka-node

MITJavaScript

A Node.js client for Apache Kafka 0.9 and later, providing producers, consumers, and administrative APIs.

GitHub

2.7k stars612 forks0 contributors

What is kafka-node?

Kafka-node is a Node.js client library for Apache Kafka that allows JavaScript applications to interact with Kafka clusters. It provides APIs for producing messages to Kafka topics, consuming messages with various consumer models, and performing administrative tasks like topic management. It solves the problem of integrating Kafka's distributed streaming platform into Node.js-based microservices and data processing applications.

Target Audience

Node.js developers building real-time data pipelines, event-driven microservices, or stream processing applications that need to integrate with Apache Kafka. It's also suitable for teams migrating Java-based Kafka clients to Node.js environments.

Value Proposition

Developers choose Kafka-node because it offers a comprehensive, production-ready Node.js interface to Kafka with support for modern Kafka features (0.9+), multiple consumption patterns, and built-in stream integration. Its active maintenance and detailed troubleshooting guide make it a reliable choice for Node.js Kafka integration.

Overview

Node.js client for Apache Kafka 0.8 and later.

Use Cases

Best For

Building real-time data pipelines with Node.js and Apache Kafka
Creating event-driven microservices that publish/subscribe to Kafka topics
Migrating Java-based Kafka clients to Node.js environments
Developing stream processing applications using Node.js streams and Kafka
Implementing consumer groups for scalable message processing in Node.js
Adding administrative Kafka operations (topic creation, group management) to Node.js tools

Not Ideal For

Projects using Apache Kafka versions older than 0.9, as it only supports 0.9 and later.
Teams preferring a TypeScript-first Kafka client with built-in type definitions and modern tooling.
Applications requiring native integration with Kafka Streams API for complex stateful stream processing.
Environments with strict dependency constraints where optional modules like snappy cause installation issues.

Pros & Cons

Pros

Comprehensive API Coverage

Supports multiple producer types (Producer, HighLevelProducer, ProducerStream) and consumer models (Consumer, ConsumerGroup, ConsumerStream), offering flexibility for various use cases as detailed in the API section.

Stream Integration

Provides ProducerStream and ConsumerGroupStream for seamless integration with Node.js streams, enabling real-time data pipelines without manual buffer management, with examples in the README.

Administrative Capabilities

Includes Admin APIs for topic creation, group management, and configuration inspection directly from Node.js, reducing reliance on external tools, as shown in the createTopics and describeConfigs methods.

Security and Compression

Supports SSL connections, SASL/PLAIN authentication, and message compression (GZip, snappy), ensuring secure and efficient communication, with configuration options documented in KafkaClient.

Cons

Dependency Installation Issues

Optional dependencies like snappy can cause node-gyp errors during npm install, often requiring the --no-optional flag or manual fixes, as highlighted in the troubleshooting FAQ.

Configuration Complexity

Numerous options for clients, producers, and consumers (e.g., fetchMaxBytes, autoCommit) make setup error-prone and require careful tuning to avoid issues like stuck partitions.

Limited to Node.js Ecosystem

As a Node.js-specific library, it's unsuitable for projects using other runtimes or languages, and lacks built-in TypeScript support compared to newer alternatives.

Frequently Asked Questions

Related Projects

kafka-manager

CMAK is a tool for managing Apache Kafka clusters

Stars11,925

Forks2,474

Last commit3 years ago

Kreuzberg

Polyglot document intelligence with a Rust core: extract text, metadata, images, tables, and structured data from 106 formats across 140 file extensions, plus code intelligence for 371 languages. Fifteen bindings, with CLI, REST API, and MCP server.

Stars9,275

Forks582

Last commit13 hours ago

kafka-docker

Dockerfile for Apache Kafka

Stars6,958

Forks2,655

Last commit2 years ago

kafkacat

Generic command line non-JVM Apache Kafka producer and consumer

Stars5,779

Forks501