How does librdkafka compare to the Java Kafka client?

librdkafka often outperforms the Java client in raw throughput due to its C implementation, but it requires more low-level coding and lacks the Java ecosystem's integrated tooling. Choose librdkafka for performance-critical C/C++ apps, and Java for ease of use in JVM environments.

How to install librdkafka on Ubuntu?

On Ubuntu, add the Confluent APT repository as per their documentation, then run `apt install librdkafka-dev` for prebuilt packages. Alternatively, build from source using the provided configure and make steps, but ensure dependencies like libssl-dev are installed.

Does librdkafka support OAuth authentication?

Yes, it supports SASL OAUTHBEARER for OAuth, as mentioned in the features. However, building from source requires libcurl-dev for OIDC support, and configuration details are in the wiki for setup with Kafka brokers.

What compression codecs are best for performance in librdkafka?

librdkafka supports snappy, gzip, lz4, and zstd compression. Snappy and lz4 offer good balance between speed and ratio, while zstd provides high compression with moderate CPU use; enable static linking of ZStd as noted in the README for optimal consumer performance.

How to handle message delivery failures in librdkafka?

Use the producer's error callbacks and configuration properties like `message.timeout.ms` to manage retries and failures. The examples directory shows code patterns for idempotent and transactional producers to ensure reliability.

Is librdkafka good for Windows development with Visual Studio?

Yes, it's portable to Windows via NuGet packages like librdkafka.redist or by building from source with Visual Studio using README.win32 instructions, but the process is more involved than on Unix-like systems.

librdkafka — High-Performance Kafka Client

What is librdkafka?

librdkafka is a robust C library implementation of the Apache Kafka protocol, providing high-performance Producer, Consumer, and Admin clients. It is designed for reliable message delivery in distributed streaming applications, capable of handling over 1 million messages per second for producers and 3 million for consumers. The library serves as the foundational client for many programming language bindings and is widely used in production systems requiring efficient Kafka communication.

Target Audience

System and application developers building high-throughput, reliable data streaming applications in C or C++, or those creating language bindings for Kafka in other programming languages. It is also suitable for DevOps and platform engineers integrating Kafka into performance-critical infrastructure.

Value Proposition

Developers choose librdkafka for its exceptional performance, proven reliability in production, and full support for Kafka features like Exactly-Once-Semantics (EOS) and security protocols. Its cross-platform portability and guaranteed API/ABI stability make it a dependable foundation for both direct C/C++ development and as the core for higher-level Kafka clients.

The Apache Kafka C/C++ library

Use Cases

Best For

Building high-performance Kafka producers and consumers in C or C++ applications.
Developing language bindings or SDKs for Apache Kafka in other programming languages (e.g., Python, Go, .NET).
Implementing transactional messaging with Exactly-Once-Semantics (EOS) in Kafka clients.
Creating cross-platform Kafka applications that need to run on Linux, Windows, macOS, and various Unix-like systems.
Integrating Kafka with systems requiring robust security features like SSL and multiple SASL mechanisms (Kerberos, OAuth, etc.).
Optimizing for maximum throughput and low latency in data streaming pipelines, such as log aggregation or real-time event processing.

Not Ideal For

Developers building simple Kafka integrations in high-level languages like Python or Java who prefer native clients without C dependencies
Projects requiring rapid prototyping or minimal setup where ease of use outweighs raw performance
Teams with limited C/C++ expertise or those prioritizing memory-safe languages for client development
Environments where containerized deployments benefit from lightweight, pure-language clients to reduce image size and complexity

Pros & Cons

Pros

Exceptional Performance

Benchmarks cited in the README show it can handle over 1 million messages per second for producers and 3 million for consumers, making it ideal for high-throughput systems.

Full EOS Support

Provides complete Exactly-Once-Semantics for transactional message delivery, ensuring data integrity in critical applications as highlighted in the features.

Cross-Platform Portability

Runs on a wide range of platforms including Linux, macOS, Windows, Solaris, and FreeBSD, as listed, enabling deployment in diverse environments.

API and ABI Stability

Guaranteed stability for C and C++ APIs with ABI safety for C, offering reliability for long-term projects without breaking changes.

Comprehensive Security Features

Supports SSL and multiple SASL mechanisms like Kerberos, OAuth, and SCRAM, enabling secure communication in enterprise setups as documented in the wiki.

Cons

Complex Build Process

Building from source requires managing dependencies like zlib, OpenSSL, and SASL libraries, and Windows builds need separate instructions from README.win32, adding setup overhead.

Low-Level C Interface

As a C library, it lacks high-level abstractions, forcing developers to handle memory management and error checking manually, which can be error-prone compared to modern language clients.

Dependency on External Libraries

Full feature support, such as SSL or OAuth, depends on optional system libraries, complicating deployment in stripped-down or containerized environments without those dependencies.

Frequently Asked Questions

What is librdkafka?

Target Audience

Value Proposition

Use Cases

Best For

Building high-performance Kafka producers and consumers in C or C++ applications.
Developing language bindings or SDKs for Apache Kafka in other programming languages (e.g., Python, Go, .NET).
Implementing transactional messaging with Exactly-Once-Semantics (EOS) in Kafka clients.
Creating cross-platform Kafka applications that need to run on Linux, Windows, macOS, and various Unix-like systems.
Integrating Kafka with systems requiring robust security features like SSL and multiple SASL mechanisms (Kerberos, OAuth, etc.).
Optimizing for maximum throughput and low latency in data streaming pipelines, such as log aggregation or real-time event processing.

Not Ideal For

Developers building simple Kafka integrations in high-level languages like Python or Java who prefer native clients without C dependencies
Projects requiring rapid prototyping or minimal setup where ease of use outweighs raw performance
Teams with limited C/C++ expertise or those prioritizing memory-safe languages for client development
Environments where containerized deployments benefit from lightweight, pure-language clients to reduce image size and complexity

Pros & Cons

Pros

Exceptional Performance

Benchmarks cited in the README show it can handle over 1 million messages per second for producers and 3 million for consumers, making it ideal for high-throughput systems.

Full EOS Support

Provides complete Exactly-Once-Semantics for transactional message delivery, ensuring data integrity in critical applications as highlighted in the features.

Cross-Platform Portability

Runs on a wide range of platforms including Linux, macOS, Windows, Solaris, and FreeBSD, as listed, enabling deployment in diverse environments.

API and ABI Stability

Guaranteed stability for C and C++ APIs with ABI safety for C, offering reliability for long-term projects without breaking changes.

Comprehensive Security Features

Supports SSL and multiple SASL mechanisms like Kerberos, OAuth, and SCRAM, enabling secure communication in enterprise setups as documented in the wiki.

Cons

Complex Build Process

Building from source requires managing dependencies like zlib, OpenSSL, and SASL libraries, and Windows builds need separate instructions from README.win32, adding setup overhead.

Low-Level C Interface

As a C library, it lacks high-level abstractions, forcing developers to handle memory management and error checking manually, which can be error-prone compared to modern language clients.

Dependency on External Libraries

Full feature support, such as SSL or OAuth, depends on optional system libraries, complicating deployment in stripped-down or containerized environments without those dependencies.

Frequently Asked Questions

librdkafka

What is librdkafka?

Overview

Use Cases

Best For

Not Ideal For

Pros & Cons

Pros

Cons

Frequently Asked Questions

Related Projects

Found a gem we're missing?

librdkafka

What is librdkafka?

Overview

Use Cases

Best For

Not Ideal For

Pros & Cons

Pros

Cons

Frequently Asked Questions

Related Projects

Found a gem we're missing?