How do I scale RQ workers for high traffic?

Use rq worker-pool to run multiple processes on a single machine, or deploy distributed workers across servers connected to the same Redis instance. For production, integrate with process managers like systemd or Kubernetes for resilience and auto-scaling.

RQ or Celery: which should I pick for background jobs?

Choose RQ for simplicity and Redis-only setups; it's lighter and faster to start. Celery is better for complex needs like multiple brokers (RabbitMQ), advanced routing, workflow orchestration, or if you need features like task result backends beyond Redis.

How to schedule a job to run every hour in RQ?

Use the built-in cron support: define the job with cron.register() in a config file using cron syntax (e.g., '0 * * * *' for hourly), then run rq cron to enqueue it automatically at intervals.

What happens to failed jobs in RQ?

Jobs without retries move to a failed queue where they can be inspected or requeued manually. With retry configured, RQ automatically re-queues them up to the max attempts, with optional delays between retries as per the Retry settings.

How to integrate RQ with a Django project?

Use the django-rq package, which provides Django management commands, settings integration, and seamless queue setup. It simplifies worker management and allows enqueuing jobs directly from Django views or models.

Can RQ handle async/await Python functions?

Not natively; RQ workers are designed for synchronous functions. For async tasks, you need to wrap them in sync code or use alternatives like ARQ that are built for asyncio, as RQ's process-based model doesn't support event loops directly.

Open-Awesome

rq

NOASSERTIONPythonv2.10

A simple Python library for queueing jobs and processing them in the background with workers, backed by Redis/Valkey.

Visit Website GitHub

10.7k stars1.5k forks0 contributors

What is rq?

RQ (Redis Queue) is a simple Python library for queueing jobs and processing them in the background with workers. It allows developers to offload time-consuming tasks like sending emails, generating reports, or processing images from their main application flow, improving responsiveness and scalability. It is backed by Redis or Valkey and designed to be easy to integrate into any Python project.

Target Audience

Python developers building web applications or services that need to handle background tasks, from small projects to large-scale enterprise systems requiring reliable job processing.

Value Proposition

Developers choose RQ for its simplicity, low barrier to entry, and Redis/Valkey-backed reliability, offering a lightweight alternative to more complex queueing systems like Celery while still providing essential features like priority queues, job scheduling, retries, and cron support.

Overview

Simple job queues for Python

Use Cases

Best For

Offloading email sending or notification tasks from web request handlers
Processing image uploads or video encoding in the background
Generating and sending periodic reports (daily, weekly, monthly)
Performing database maintenance or cleanup operations on a schedule
Handling long-running API calls without blocking user interactions
Scaling task processing across multiple worker processes in production

Not Ideal For

Applications requiring complex workflow orchestration with job dependencies (e.g., DAGs) or task chaining
Teams in serverless environments wanting a fully managed queue service without self-hosted Redis/Valkey
Projects where real-time, sub-millisecond job processing is critical, as process isolation adds latency
Systems needing built-in advanced monitoring, admin UI, or support for brokers beyond Redis

Pros & Cons

Pros

Dead-Simple API

Enqueuing jobs requires minimal code—just a Redis connection and function call, as shown in the 'Getting started' example with count_words_at_url.

Flexible Scheduling Options

Supports delayed, repeating, and cron-based jobs via enqueue_at/enqueue_in and the rq cron command, eliminating need for separate scheduler services.

Configurable Error Recovery

Automatic retry mechanisms with customizable intervals and max attempts (Retry class) ensure resilience for transient failures without manual intervention.

Worker Performance Choices

Offers Worker/SpawnWorker for safe process isolation and SimpleWorker for high-speed in-process execution, allowing trade-offs based on job trust and latency needs.

Cons

Redis Infrastructure Burden

Requires a dedicated Redis or Valkey instance, adding operational complexity, potential single points of failure, and extra costs for scaling.

Safety vs. Speed Trade-off

Default process-isolated workers (Worker/SpawnWorker) incur fork/spawn overhead; while SimpleWorker is faster, the README warns it's unsafe for untrusted jobs due to crash risks.

Sparse Built-in Tooling

Lacks native web dashboard or advanced monitoring; you must rely on third-party add-ons like rq-dashboard for real-time queue insights and management.

Open Source Alternative To

rq is an open-source alternative to the following products:

Celery

Celery is a distributed task queue system for Python that handles asynchronous job processing, scheduling, and message passing, often used with web frameworks like Django.

Frequently Asked Questions

Related Projects

Celery

Distributed Task Queue (development branch)

a little task queue for python

A fast and reliable background task processing library for Python 3.

Stars5,293

Forks375

Last commit1 day ago

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

Submit a project Star on GitHub

rq

NOASSERTIONPythonv2.10

A simple Python library for queueing jobs and processing them in the background with workers, backed by Redis/Valkey.

Visit Website GitHub

10.7k stars1.5k forks0 contributors

What is rq?

Target Audience

Python developers building web applications or services that need to handle background tasks, from small projects to large-scale enterprise systems requiring reliable job processing.

Value Proposition

Overview

Simple job queues for Python

Use Cases

Best For

Offloading email sending or notification tasks from web request handlers
Processing image uploads or video encoding in the background
Generating and sending periodic reports (daily, weekly, monthly)
Performing database maintenance or cleanup operations on a schedule
Handling long-running API calls without blocking user interactions
Scaling task processing across multiple worker processes in production

Not Ideal For

Applications requiring complex workflow orchestration with job dependencies (e.g., DAGs) or task chaining
Teams in serverless environments wanting a fully managed queue service without self-hosted Redis/Valkey
Projects where real-time, sub-millisecond job processing is critical, as process isolation adds latency
Systems needing built-in advanced monitoring, admin UI, or support for brokers beyond Redis

Pros & Cons

Pros

Dead-Simple API

Enqueuing jobs requires minimal code—just a Redis connection and function call, as shown in the 'Getting started' example with count_words_at_url.

Flexible Scheduling Options

Supports delayed, repeating, and cron-based jobs via enqueue_at/enqueue_in and the rq cron command, eliminating need for separate scheduler services.

Configurable Error Recovery

Automatic retry mechanisms with customizable intervals and max attempts (Retry class) ensure resilience for transient failures without manual intervention.

Worker Performance Choices

Offers Worker/SpawnWorker for safe process isolation and SimpleWorker for high-speed in-process execution, allowing trade-offs based on job trust and latency needs.

Cons

Redis Infrastructure Burden

Requires a dedicated Redis or Valkey instance, adding operational complexity, potential single points of failure, and extra costs for scaling.

Safety vs. Speed Trade-off

Default process-isolated workers (Worker/SpawnWorker) incur fork/spawn overhead; while SimpleWorker is faster, the README warns it's unsafe for untrusted jobs due to crash risks.

Sparse Built-in Tooling

Lacks native web dashboard or advanced monitoring; you must rely on third-party add-ons like rq-dashboard for real-time queue insights and management.

Open Source Alternative To

rq is an open-source alternative to the following products:

Celery

Celery is a distributed task queue system for Python that handles asynchronous job processing, scheduling, and message passing, often used with web frameworks like Django.