How to handle Protobuf messages in Ziggurat?

Use the provided middleware like `protobuf->hash` to deserialize Protobuf messages into Clojure maps. Customize by passing the Java class and stream ID, as shown in the usage examples.

Ziggurat vs using plain Kafka clients in Clojure?

Ziggurat abstracts boilerplate like retry logic and HTTP server setup, while plain clients offer more control. Choose Ziggurat for faster development in stateless apps; use plain clients for highly custom or stateful needs.

Can Ziggurat process messages from multiple topics concurrently?

Yes, it supports multiple streams with distinct handler functions, allowing concurrent consumption from different Kafka topics based on configuration in the routes map.

How to configure retry counts in Ziggurat?

Retry counts are set via RabbitMQ configuration in the config file. Adjust settings to control retry limits before messages move to the dead letter queue, as detailed in the configuration docs.

What are the deployment considerations for Ziggurat apps?

Deploy as standard Clojure applications with dependencies on Kafka and RabbitMQ. Use the graceful shutdown features for clean resource management, and ensure production environments mirror the local setup services.

Open-Awesome

Ziggurat

Apache-2.0Clojurev4.13.2

A Clojure framework for building stateless stream processing applications on Kafka with built-in retry mechanisms.

GitHub

406 stars70 forks0 contributors

What is Ziggurat?

Ziggurat is a Clojure framework for building stateless stream processing applications on Kafka. It simplifies reading and processing messages from Kafka by abstracting common infrastructure concerns like message consumption, retry mechanisms via RabbitMQ, and HTTP server setup. Developers can focus on implementing business logic while Ziggurat handles the underlying stream processing complexities.

Target Audience

Clojure developers building real-time data processing applications, event-driven microservices, or ETL pipelines that consume from Kafka. It's particularly useful for teams needing a structured framework to reduce boilerplate in stream processing.

Value Proposition

Ziggurat provides a production-ready, opinionated framework that abstracts away the intricacies of Kafka and RabbitMQ integration, offering built-in retry logic, multiple stream support, and graceful shutdown capabilities. Its middleware system and configuration-driven approach make it flexible yet easy to adopt for Clojure-based stream processing.

Overview

A stream processing framework to build stateless applications on Kafka

Use Cases

Best For

Building real-time data processing pipelines in Clojure
Creating event-driven microservices that consume Kafka topics
Implementing retry mechanisms for failed Kafka message processing
Developing stateless applications that require graceful shutdown and startup
Prototyping stream processing applications with minimal boilerplate
Handling multiple Kafka streams with separate processing logic

Not Ideal For

Projects using programming languages other than Clojure
Applications requiring stateful stream processing or complex event sourcing
Teams that prefer Kafka-native retry mechanisms over RabbitMQ

Pros & Cons

Pros

Abstracted Kafka Consumption

Handles message reading from Kafka topics and partitions, reducing the need for low-level consumer code and simplifying stream setup.

Built-in Retry with RabbitMQ

Provides configurable retry counts for failed messages via RabbitMQ, automating error recovery without custom implementation.

Multiple Stream Support

Enables reading from multiple Kafka streams with separate handler functions, allowing complex data flows from different topics.

HTTP Server Integration

Includes an embedded HTTP server for adding REST endpoints, useful for health checks or admin interfaces without extra setup.

Cons

Technology Coupling

Mandates the use of both Kafka and RabbitMQ, limiting flexibility for projects using alternative messaging systems like Apache Pulsar or Redis.

Setup Complexity

Local development requires running Kafka, ZooKeeper, and RabbitMQ via Docker, which can be resource-heavy and slow to start.

Feature Deprecation

Features like Sentry integration have been deprecated, indicating potential breaking changes or reduced long-term support in updates.

Frequently Asked Questions

Related Projects

Metabase

The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:

Stars49,127

Forks6,797

Last commit9 hours ago

Logseq

A privacy-first, open-source platform for knowledge management and collaboration. Download link: http://github.com/logseq/logseq/releases. roadmap: https://logseq.io/p/NX4mc_ggEV

Stars44,819

Forks2,802

Last commit10 hours ago

mal

mal - Make a Lisp

Stars10,728

Forks2,705

Last commit10 months ago

Jepsen

A framework for distributed systems verification, with fault injection

Stars7,489

Forks754

Last commit4 days ago

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

Submit a project Star on GitHub