lua-resty-kafka
A Lua Kafka client driver for OpenResty/ngx_lua using the cosocket API for non-blocking communication.
What is lua-resty-kafka?
lua-resty-kafka is a Lua library that provides a client driver for Apache Kafka, specifically designed for use with OpenResty and the ngx_lua module. It enables Lua applications running within Nginx to produce and consume messages from Kafka topics using a non-blocking, event-driven approach. The library solves the problem of integrating Kafka with high-performance web servers without sacrificing concurrency or responsiveness.
Developers building real-time data processing pipelines, log aggregation systems, or microservices with OpenResty who need to interact with Kafka clusters efficiently. It's particularly suited for those already using ngx_lua for other non-blocking operations.
Developers choose lua-resty-kafka because it leverages OpenResty's cosocket API for truly non-blocking network operations, avoiding the performance pitfalls of blocking I/O in Lua. Its tight integration with the Nginx ecosystem makes it a natural choice for existing OpenResty deployments over generic Kafka clients.
Overview
Lua kafka client driver for the Openresty based on the cosocket API
Use Cases
Best For
- Sending application logs from Nginx to Kafka for centralized logging
- Building real-time event-driven microservices with OpenResty
- Implementing non-blocking message producers in Lua within Nginx
- Creating lightweight Kafka consumers for simple data fetching tasks
- Integrating Kafka with existing OpenResty APIs for stream processing
- Developing high-throughput data pipelines that require SSL/SASL authentication
Not Ideal For
- Applications requiring Kafka consumer groups for coordinated message consumption, as it only has a basic consumer without group support.
- Projects not using OpenResty or LuaJIT, since it's tightly coupled to these technologies and requires specific Nginx modules.
- High-throughput consumer scenarios needing batch fetching or advanced Kafka features like compression, which are unsupported.
- Production systems demanding stable, fully-featured libraries, given its experimental status and early development stage.
Pros & Cons
Pros
Uses ngx_lua's cosocket API for 100% non-blocking operations, ensuring high performance in event-driven Nginx environments without blocking workers.
Supports SSL and multiple SASL mechanisms like PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512, enabling secure connections to Kafka clusters with optional dependencies.
Offers an async producer with configurable buffering, retry logic, and error handling, allowing for efficient message batching and improved throughput.
Provides methods to fetch and refresh Kafka cluster metadata, including broker and partition information, aiding in dynamic topic handling.
Cons
The basic consumer lacks support for consumer groups and batch fetching, making it unsuitable for distributed consumption patterns or high-performance fetching needs.
Explicitly does not support compression, which can impact performance and bandwidth usage in high-volume data streaming scenarios.
Still under early development with experimental status, and requires specific ngx_lua or OpenResty versions, potentially leading to instability or compatibility issues.
Frequently Asked Questions
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.