How do I set up xk6-kafka to test Avro schemas with Schema Registry?

Import the SchemaRegistry class, configure it with the registry URL, and use its serialize/deserialize methods. The README provides example scripts for fetching schemas and handling Avro data, including compression support.

xk6-kafka vs Apache JMeter for Kafka load testing: which should I use?

xk6-kafka integrates natively with k6 for script-based testing and offers detailed Kafka-specific metrics, while JMeter has a GUI and broader protocol support. Choose xk6-kafka if you're already using k6 or need rich Kafka metrics; opt for JMeter for GUI-based or multi-protocol testing.

Why does my xk6-kafka consumer hang or timeout?

This often occurs if the topic is empty, doesn't exist, or if maxWait is too short. Create topics in the setup() function and adjust maxWait in the Reader config, as recommended in the FAQ section on timeout errors.

Can xk6-kafka handle messages larger than 1 MB?

Yes, but you must increase the batchBytes configuration in the Writer and adjust Kafka broker settings like message.max.bytes. The FAQ explains how to calculate message size and avoid MessageTooLargeError.

How to install xk6-kafka on Windows without Docker?

Download pre-built binaries from the releases page for Windows amd64 or arm64. For custom builds, install Go and xk6, then use the xk6 build command, but be prepared for potential Go environment issues on Windows.

What metrics does xk6-kafka expose for monitoring?

It exposes over 30 metrics including kafka_reader_error_count, kafka_writer_message_bytes, and kafka_reader_lag, all listed in the emitted metrics table. These integrate with k6 thresholds for automated pass/fail criteria.

Open-Awesome

xk6-kafka

Apache-2.0Gov2.2.0

A k6 extension for load testing Apache Kafka with support for multiple serialization formats, authentication, compression, and Schema Registry.

GitHub

219 stars101 forks0 contributors

What is xk6-kafka?

xk6-kafka is a k6 extension that enables load testing of Apache Kafka. It allows developers to produce and consume messages to simulate high traffic and test the performance of their Kafka-based systems. The extension supports various serialization formats, authentication methods, compression, and integration with Schema Registry.

Target Audience

Developers and performance engineers who need to load test Apache Kafka producers and consumers as part of their system validation and performance tuning.

Value Proposition

It provides a native k6 integration for Kafka load testing with extensive feature support, including multiple serialization formats, authentication, and detailed metrics, making it a versatile alternative to custom scripting or commercial tools.

Overview

k6 extension to load test Apache Kafka with support for various serialization formats, SASL, TLS, compression, Schema Registry client and beyond

Use Cases

Best For

Load testing Kafka producers and consumers under high concurrency
Validating Kafka system performance with various serialization formats like Avro and JSON Schema
Testing Kafka authentication setups with SASL, SSL, or AWS IAM
Simulating real-world message compression scenarios (Gzip, Snappy, Lz4, Zstd)
Debugging and verifying message flow in Kafka topics with consumer groups
Integrating Kafka load testing into existing k6 performance test suites

Not Ideal For

Teams requiring a GUI-based load testing interface, as xk6-kafka relies on k6's CLI and scripting environment
Projects focused on testing non-Kafka message brokers like RabbitMQ or AWS SQS, since it's specifically tailored for Apache Kafka
Environments where quick, no-code load testing is a priority, due to the need to write and maintain JavaScript test scripts
Organizations demanding official vendor support and SLAs, as the project is maintained personally and not by the k6 team

Pros & Cons

Pros

Broad Serialization Support

Supports multiple formats including String, JSON, ByteArray, Avro, and JSON Schema, with options to fetch schemas from a Registry or embed them directly, as detailed in the example scripts.

Comprehensive Authentication

Includes SASL PLAIN, SCRAM, SSL, AWS IAM, and TLS with JKS files, covering most enterprise security scenarios outlined in the README.

Rich Metrics Export

Exposes over 30 Kafka-specific metrics like reader/writer error counts and lag, compatible with k6 thresholds for in-depth performance analysis.

Flexible Topic Management

Allows programmatic creation, listing, and deletion of Kafka topics within test scripts, enabling dynamic test environment setup.

Cons

Complex Build Process

Requires installing Go, xk6, and building a custom k6 binary, which adds significant setup overhead compared to standalone load testing tools.

Unofficial Maintenance

Maintained personally and not officially supported by the k6 team, posing potential risks for long-term updates, bug fixes, and enterprise support.

Message Size Constraints

Default Kafka message size limits can cause errors; users must manually adjust configurations like batchBytes, as troubleshooting large messages requires extra steps per the FAQ.

Frequently Asked Questions

Related Projects

xk6-faker

Random fake data generator for k6.

Stars117

Forks22

Last commit2 days ago

xk6-mqtt

k6 mqtt extension

Stars58

Forks42

Last commit1 month ago

xk6-playwright

k6 extension that adds support for browser automation and end-to-end web testing using playwright-go

Stars54

Forks14

Last commit2 years ago

xk6-prometheus

Prometheus HTTP exporter for k6