SidekiqIteration
A Sidekiq extension that makes long-running jobs interruptible and resumable with checkpointing.
What is SidekiqIteration?
Sidekiq Iteration is a Ruby gem that extends Sidekiq to make long-running background jobs interruptible and resumable. It solves the problem of job progress loss during deploys, worker restarts, or infrastructure failures by checkpointing iteration state after each unit of work.
Ruby developers using Sidekiq for background job processing, especially those dealing with large datasets, long-running tasks, or operating in cloud environments with frequent deploys.
Developers choose Sidekiq Iteration because it provides a robust, built-in pattern for resilient job processing without the complexity of manually splitting jobs or managing state. It reduces Redis overhead, simplifies monitoring, and ensures no data is processed twice.
Overview
Make your long-running sidekiq jobs interruptible and resumable.
Use Cases
Best For
- Processing millions of Active Record records in a single job without duplicates
- Handling CSV imports or large file processing with resume capability
- Running jobs in cloud environments with unpredictable instance restarts
- Implementing nested iterations over related models (e.g., shops, products, variants)
- Adding lifecycle hooks (start, resume, shutdown) to Sidekiq jobs
- Ensuring job progress survives frequent application deploys
Not Ideal For
- Projects with short-lived jobs that complete in seconds without interruption risks
- Teams using job processors other than Sidekiq or lacking Redis infrastructure
- Jobs where each iteration consistently exceeds 30 seconds, as the gem recommends keeping iterations brief for safe checkpoints
- Scenarios requiring minimal code changes without adopting the build_enumerator/each_iteration pattern
Pros & Cons
Pros
Automatically checkpoints iteration state to Redis after each unit of work, allowing jobs to be safely stopped during deploys or restarts and resumed without data loss, as emphasized in the background section.
Provides built-in enumerators for Active Record records, batches, relations, arrays, CSV files, and nested structures, enabling versatile data processing without custom boilerplate.
Offers callbacks like on_start, on_resume, on_shutdown, and on_complete for custom logic, giving developers fine-grained control over job stages without monkey-patching.
Integrates with existing Sidekiq jobs by including a module and defining two methods, minimizing setup effort while leveraging Sidekiq's ecosystem.
Cons
The README explicitly recommends each_iteration should take less than 30 seconds to ensure safe interruptions, which can force refactoring of complex operations into smaller units or nested jobs.
Requires jobs to implement build_enumerator and each_iteration instead of a simple perform method, introducing a learning curve and potential boilerplate for developers accustomed to standard Sidekiq jobs.
Frequent checkpointing to Redis adds network latency and storage usage, which might impact performance in high-throughput environments or where Redis is a bottleneck.
Frequently Asked Questions
Related Projects
ResqueResque is a Redis-backed Ruby library for creating background jobs, placing them on multiple queues, and processing them later.
Delayed JobDatabase based asynchronous priority queue system -- Extracted from Shopify
good_jobMultithreaded, Postgres-based, Active Job backend for Ruby on Rails.
Sucker PunchSucker Punch is a Ruby asynchronous processing library using concurrent-ruby, heavily influenced by Sidekiq and girl_friday.
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.