How to set up rate limiting with OPQ in Elixir?

Configure the :interval option in milliseconds when initializing the queue, e.g., OPQ.init(interval: 1000) for one job per second. This controls processing speed without external dependencies, as detailed in the README examples.

OPQ vs Oban: which is better for background jobs?

OPQ is ideal for lightweight, in-memory queues with simple needs like rate limiting, while Oban offers persistence, distributed support, and advanced features like retries. Choose OPQ for transient tasks and Oban for production-critical workloads.

Does OPQ save jobs if the server crashes?

No, OPQ is an in-memory queue, so all pending jobs are lost on crashes or restarts. For persistence, you'd need to integrate with external storage like databases or use a different library that supports disk-based queues.

Can I use OPQ in a distributed Elixir cluster?

Not directly; OPQ is designed for single-node in-memory use. For distributed queues, you'd need to implement custom synchronization or consider alternatives like Broadway or external message brokers.

How to handle failed jobs in OPQ?

OPQ provides timeout handling via the :timeout option, but for error recovery, you must add custom logic in your worker module. There's no built-in retry mechanism, so errors require manual intervention or wrapping in try-catch blocks.

What's the performance impact of OPQ with high concurrency?

Performance is generally good due to Erlang's BEAM VM, but high concurrency with many workers can lead to memory bloat since it's in-memory. Monitor resource usage and adjust worker counts based on your system's capacity.

Open-Awesome

opq

Elixirv4.0.4

A simple, in-memory FIFO queue for Elixir with worker pooling and rate limiting.

Visit Website GitHub

269 stars9 forks0 contributors

What is opq?

OPQ is an Elixir library that provides a simple, in-memory FIFO queue with built-in worker pooling and rate limiting. It solves the problem of managing background tasks and concurrent job processing in Elixir applications efficiently. The library leverages Erlang's queue module and Elixir's GenStage for reliable performance.

Target Audience

Elixir developers who need a lightweight, in-memory queue for background job processing, task management, or rate-limited operations in their applications.

Value Proposition

Developers choose OPQ for its simplicity, performance, and essential features like worker pooling and rate limiting without the overhead of more complex job processing systems. It integrates seamlessly with Elixir's concurrency model and supervision trees.

Overview

Elixir queue! A simple, in-memory queue with worker pooling and rate limiting in Elixir.

Use Cases

Best For

Processing background tasks in Elixir applications
Implementing rate-limited API calls or operations
Managing concurrent job processing with worker pools
Handling lightweight in-memory job queues
Integrating queue functionality into Elixir supervision trees
Building crawlers or scrapers with controlled request rates

Not Ideal For

Applications requiring persistent job storage across restarts
Distributed systems needing queue replication across multiple nodes
Projects needing advanced job features like priority queuing or automatic retries

Pros & Cons

Pros

Fast In-Memory Performance

Leverages Erlang's queue module for efficient FIFO operations, ensuring low-latency processing as emphasized in the README's focus on speed.

Simple Supervision Integration

Easily starts as part of Elixir supervision trees with minimal configuration, demonstrated in the usage examples for seamless application setup.

Flexible Worker Pooling

Supports configurable worker counts and custom worker modules, allowing tailored job processing as shown in the CustomWorker example.

Built-in Rate Limiting

Offers configurable intervals to control processing speed, useful for scenarios like rate-limited API calls, with bypass option when not needed.

Cons

No Persistence

Jobs are stored only in memory, so they are lost on application crashes or restarts, limiting reliability for critical workloads without external backups.

Single-Node Limitation

Designed for in-memory use on a single node, lacking built-in support for distributed queues or cluster synchronization, which restricts scalability.

Basic Error Handling

While timeouts are configurable, there's no inherent retry logic, dead-letter queues, or advanced error recovery mechanisms, requiring manual implementation.

Frequently Asked Questions

Related Projects

oban

💎 Robust job processing in Elixir, backed by modern PostgreSQL, SQLite3, and MySQL

Stars3,964

Forks368

Last commit16 hours ago

broadway

Concurrent and multi-stage data ingestion and data processing with Elixir

Stars2,680

Forks179

Last commit1 month ago

exq

Job processing library for Elixir - compatible with Resque / Sidekiq

Stars1,531

Forks181