A Sidekiq extension that makes long-running jobs interruptible and resumable with checkpointing.
Sidekiq Iteration is a Ruby gem that extends Sidekiq to make long-running background jobs interruptible and resumable. It solves the problem of job progress loss during deploys, worker restarts, or infrastructure failures by checkpointing iteration state after each unit of work.
Ruby developers using Sidekiq for background job processing, especially those dealing with large datasets, long-running tasks, or operating in cloud environments with frequent deploys.
Developers choose Sidekiq Iteration because it provides a robust, built-in pattern for resilient job processing without the complexity of manually splitting jobs or managing state. It reduces Redis overhead, simplifies monitoring, and ensures no data is processed twice.
Make your long-running sidekiq jobs interruptible and resumable.
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
Automatically checkpoints iteration state to Redis after each unit of work, allowing jobs to be safely stopped during deploys or restarts and resumed without data loss, as emphasized in the background section.
Provides built-in enumerators for Active Record records, batches, relations, arrays, CSV files, and nested structures, enabling versatile data processing without custom boilerplate.
Offers callbacks like on_start, on_resume, on_shutdown, and on_complete for custom logic, giving developers fine-grained control over job stages without monkey-patching.
Integrates with existing Sidekiq jobs by including a module and defining two methods, minimizing setup effort while leveraging Sidekiq's ecosystem.
The README explicitly recommends each_iteration should take less than 30 seconds to ensure safe interruptions, which can force refactoring of complex operations into smaller units or nested jobs.
Requires jobs to implement build_enumerator and each_iteration instead of a simple perform method, introducing a learning curve and potential boilerplate for developers accustomed to standard Sidekiq jobs.
Frequent checkpointing to Redis adds network latency and storage usage, which might impact performance in high-throughput environments or where Redis is a bottleneck.