A Lua Kafka client driver for OpenResty/ngx_lua using the cosocket API for non-blocking communication.
lua-resty-kafka is a Lua library that provides a client driver for Apache Kafka, specifically designed for use with OpenResty and the ngx_lua module. It enables Lua applications running within Nginx to produce and consume messages from Kafka topics using a non-blocking, event-driven approach. The library solves the problem of integrating Kafka with high-performance web servers without sacrificing concurrency or responsiveness.
Developers building real-time data processing pipelines, log aggregation systems, or microservices with OpenResty who need to interact with Kafka clusters efficiently. It's particularly suited for those already using ngx_lua for other non-blocking operations.
Developers choose lua-resty-kafka because it leverages OpenResty's cosocket API for truly non-blocking network operations, avoiding the performance pitfalls of blocking I/O in Lua. Its tight integration with the Nginx ecosystem makes it a natural choice for existing OpenResty deployments over generic Kafka clients.
Lua kafka client driver for the Openresty based on the cosocket API
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Uses ngx_lua's cosocket API for 100% non-blocking operations, ensuring high performance in event-driven Nginx environments without blocking workers.
Supports SSL and multiple SASL mechanisms like PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512, enabling secure connections to Kafka clusters with optional dependencies.
Offers an async producer with configurable buffering, retry logic, and error handling, allowing for efficient message batching and improved throughput.
Provides methods to fetch and refresh Kafka cluster metadata, including broker and partition information, aiding in dynamic topic handling.
The basic consumer lacks support for consumer groups and batch fetching, making it unsuitable for distributed consumption patterns or high-performance fetching needs.
Explicitly does not support compression, which can impact performance and bandwidth usage in high-volume data streaming scenarios.
Still under early development with experimental status, and requires specific ngx_lua or OpenResty versions, potentially leading to instability or compatibility issues.