How to set up a Kafka consumer group with Sarama?

Use the ConsumerGroup type from the sarama package, configure brokers and topics, and handle rebalancing events. Refer to the examples directory for complete implementations, and ensure proper error handling for production use.

Sarama vs confluent-kafka-go: which is better?

Sarama is a pure Go implementation with full protocol support and mocks, ideal for control and testing. Confluent-kafka-go wraps the C library, offering better performance in some cases but with C dependencies. Choose Sarama for Go-native simplicity or confluent-kafka-go for performance-critical scenarios.

Does Sarama support Kafka transactions?

Yes, Sarama supports Kafka transactions for atomic message production. Check the API documentation for the Producer with transactional capabilities, and ensure your Kafka cluster is configured for it, as noted in the compatibility guarantee.

Best practices for error handling in Sarama?

Implement retry logic for network errors, monitor consumer group rebalancing, and use the mock framework for testing error scenarios. The wiki and examples provide guidance, but robust handling often requires custom code due to low-level exposure.

How to mock Kafka producers in unit tests with Sarama?

Use the mocks subpackage, such as MockAsyncProducer, to simulate Kafka interactions without a live cluster. The examples directory shows how to set up mocks for verifying message production and handling errors in isolation.

Sarama compatibility with Kafka 3.0?

Yes, Sarama supports Kafka 3.0 under its '2 releases + 2 months' policy. Always check the changelog for specific version updates, as older releases might still work but are not guaranteed.

Open-Awesome

sarama

MITGov1.47.0

A Go client library for Apache Kafka with comprehensive API support and testing utilities.

GitHub

12.5k stars1.8k forks0 contributors

What is sarama?

Sarama is a Go client library for Apache Kafka that provides a complete implementation of the Kafka protocol. It enables Go applications to produce and consume messages, manage topics and partitions, and interact with Kafka clusters programmatically. The library solves the need for a reliable, production-grade Kafka integration in Go-based systems.

Target Audience

Go developers building distributed systems, microservices, or data pipelines that require Kafka for event streaming, message queuing, or real-time data processing.

Value Proposition

Developers choose Sarama because it's the most comprehensive and widely-used Kafka client for Go, offering full protocol support, semantic versioning for API stability, and excellent testing utilities including mocks for isolated unit testing.

Overview

Sarama is a Go library for Apache Kafka.

Use Cases

Best For

Building Go microservices that publish events to Kafka topics
Creating Kafka consumers in Go for real-time data processing
Developing administrative tools to manage Kafka clusters programmatically
Unit testing Kafka interactions with mock objects
Implementing event-driven architectures in Go applications
Migrating Java-based Kafka clients to Go while maintaining protocol compatibility

Not Ideal For

Projects built in languages other than Go, as Sarama is exclusively a Go library
Teams using fully managed Kafka services (e.g., Confluent Cloud) that offer SDKs with higher-level abstractions and cloud integrations
Applications needing only basic message queuing without Kafka's distributed streaming complexity
Rapid prototyping where setting up and maintaining a Kafka cluster is impractical

Pros & Cons

Pros

Full Protocol Support

Implements all Kafka API versions for producing, consuming, and administrative operations, ensuring comprehensive compatibility with Kafka clusters.

Integrated Mock Testing

Includes a comprehensive mock framework in the mocks subpackage, enabling unit testing of Kafka interactions without a live cluster.

Strong Compatibility Guarantee

Follows a '2 releases + 2 months' policy for Kafka and Go version support, providing predictable upgrade paths and backward compatibility.

Useful Command Line Tools

The tools directory contains utilities for testing, diagnostics, and instrumentation, enhancing development and operational workflows.

Cons

Steep Learning Curve

Requires deep understanding of Kafka's protocol and architecture, which can be challenging for developers new to distributed systems or event streaming.

Fragmented Documentation

Documentation is spread across pkg.go.dev, the wiki, and example directories, making it harder to find consolidated guidance for common tasks.

Manual Error Handling

Exposes low-level errors from the Kafka protocol directly, necessitating custom implementation for robust error recovery, retry logic, and monitoring.

Frequently Asked Questions

Related Projects

Asynq

Simple, reliable, and efficient distributed task queue in Go

Stars13,187

Forks925

Last commit10 days ago

Centrifugo

Scalable real-time messaging server in a language-agnostic way. Self-hosted alternative to Pubnub, Pusher, Ably, socket.io, Phoenix.PubSub, SignalR. Set up once and forever.

Building event-driven applications the easy way in Go.

Stars9,670

Forks495

Last commit4 days ago

gorush

A push notification server written in Go (Golang).

Stars8,719

Forks886

Last commit9 days ago

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

Submit a project Star on GitHub