How do I set up Exq with Phoenix and Ecto for database jobs?

Add Exq to your supervision tree with start_on_application: false to control startup, and lower concurrency for database queues to avoid connection pool exhaustion, as recommended in the Using with Phoenix and Ecto section.

Exq vs Oban: which is better for Elixir background jobs?

Exq is ideal for Sidekiq compatibility and Redis-based durability, while Oban is PostgreSQL-native and may suit teams already using Postgres. Choose Exq for interoperability with Ruby systems or if Redis is preferred.

How to handle failed jobs and retries in Exq?

Configure scheduler_enable: true and max_retries in settings, and use exponential backoff; failed jobs beyond retries move to a dead queue, viewable via the Web UI or Sidekiq interface.

Can I use the Sidekiq UI to monitor Exq jobs?

Yes, Exq is compatible with the Sidekiq stats format, so you can point the Sidekiq UI to the same Redis namespace to view job statuses and processing rates, as noted in the README.

How to prevent duplicate jobs in Exq?

Use the unique_for and unique_until options when enqueuing, and ensure Exq.Middleware.Unique is in your middleware chain; note that this is best-effort and not foolproof.

What are the performance costs of using Redis with Exq?

Redis adds network latency and serialization overhead, which can impact job throughput; for high-performance, in-memory needs, consider OTP alternatives like GenStage or Task.

How to scale Exq workers across multiple nodes?

Configure multiple Erlang nodes to process from the same Redis pool, and enable heartbeat for node recovery in auto-scaling setups, as described in the Node Recovery section.

Open-Awesome

exq

NOASSERTIONElixirv0.23.0

A Redis-backed job processing library for Elixir with Resque/Sidekiq compatibility.

GitHub

1.5k stars182 forks0 contributors

What is exq?

Exq is a background job processing library for Elixir that enables reliable, asynchronous task execution using Redis for storage. It solves the need for durable job queues with retries, concurrency control, and cross-node processing, especially in applications where jobs must persist through restarts. Its compatibility with Resque and Sidekiq allows easy integration with existing Ruby-based systems.

Target Audience

Elixir developers building applications that require robust background job processing, such as web apps with async tasks, data pipelines, or integrations with existing Sidekiq/Resque setups.

Value Proposition

Developers choose Exq for its proven Resque/Sidekiq compatibility, ensuring seamless migration from Ruby ecosystems, combined with Elixir's concurrency model and reliability features like atomic enqueuing and unique job handling.

Overview

Job processing library for Elixir - compatible with Resque / Sidekiq

Use Cases

Best For

Adding background job processing to Elixir Phoenix applications
Migrating from Sidekiq/Resque to an Elixir-based stack
Handling durable jobs that require retries with exponential backoff
Running distributed job processing across multiple Erlang nodes
Integrating with existing Ruby/Rails projects using Resque-compatible queues
Ensuring job persistence and reliability in microservices architectures

Not Ideal For

Applications that only need in-memory, ephemeral job queues without persistence requirements
Real-time systems where sub-millisecond job processing latency is critical and Redis overhead is unacceptable
Serverless or edge computing environments where deploying and managing Redis is impractical
Simple async tasks that can be adequately handled by Elixir's built-in OTP primitives like Task or GenStage

Pros & Cons

Pros

Sidekiq/Resque Interoperability

Uses the same job format as Resque/Sidekiq, enabling seamless integration with existing Ruby/Rails projects and the Sidekiq UI, as detailed in the compatibility section of the README.

Durable Job Persistence

Jobs are stored in Redis with backup queues using RPOPLPUSH, ensuring they survive worker restarts and node failures, providing reliability for long-running tasks.

Configurable Retry Logic

Supports automatic job retries with exponential backoff and a configurable max retries setting, making it robust for handling transient failures in distributed systems.

Unique Job Handling

Prevents duplicate job enqueuing with lock-based strategies for idempotency and debouncing, though it's noted as best-effort in the Unique Jobs section.

Atomic Batch Enqueuing

Allows enqueuing multiple jobs atomically with enqueue_all, ensuring consistency in batch operations, as described in the Enqueuing Many Jobs Atomically section.

Cons

Redis Dependency and Overhead

Requires a dedicated Redis instance, adding infrastructure complexity, serialization latency, and operational costs that may be overkill for simple use cases.

Configuration Complexity

Multiple settings for queues, concurrency, modes, middleware, and recovery options can be overwhelming, with misconfiguration risks highlighted in the README's setup guides.

Best-Effort Uniqueness Guarantee

The unique jobs feature is implemented via middleware and explicitly described as 'best effort' in the README, not a strict guarantee against duplicate execution.

Default Heartbeat Disabled

Node recovery via heartbeat is not enabled by default, requiring manual configuration for auto-scaling environments, which could lead to orphaned jobs if overlooked.

Open Source Alternative To

exq is an open-source alternative to the following products:

Resque

Resque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later.

Sidekiq

Sidekiq is a background job processing framework for Ruby applications, using Redis to queue and execute jobs asynchronously.

Frequently Asked Questions

Related Projects

oban

💎 Robust job processing in Elixir, backed by modern PostgreSQL, SQLite3, and MySQL

Stars3,945

Forks367

Last commit20 hours ago

broadway

Concurrent and multi-stage data ingestion and data processing with Elixir

Stars2,665

Forks178

Last commit24 days ago

verk

A job processing system that just verks! 🧛‍

Stars729

Forks60

Last commit4 years ago

honeydew

Job Queue for Elixir. Clustered or Local. Straight BEAM. Optional Ecto. 💪🍈

Stars725

Forks59

Last commit2 years ago

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

Submit a project Star on GitHub

exq