pluck_in_batches
A Ruby gem providing ActiveRecord batch methods for plucking columns, reducing SQL queries and memory usage by up to 50%.
What is pluck_in_batches?
PluckInBatches is a Ruby gem that adds `pluck_each` and `pluck_in_batches` methods to ActiveRecord for efficiently batch-processing selected database columns. It solves performance issues when iterating over large datasets by reducing SQL queries and memory usage by up to 50% compared to manual combinations of `in_batches` and `pluck`.
Ruby on Rails developers working with large ActiveRecord datasets who need to optimize memory usage and query performance during batch operations.
Developers choose PluckInBatches because it provides a clean, ActiveRecord-native API for efficient column plucking in batches, delivering measurable performance improvements without complex workarounds.
Overview
A faster alternative to the custom use of in_batches with pluck
Use Cases
Best For
- Processing large user email lists in background jobs
- Batch exporting specific columns from million-record tables
- Reducing memory consumption in data migration scripts
- Optimizing report generation that requires only subset of columns
- Iterating over filtered ActiveRecord relations with selected fields
- Replacing manual `in_batches` + `pluck` patterns with cleaner code
Not Ideal For
- Projects that require full ActiveRecord objects with callbacks and associations during iteration
- Applications needing to perform batch updates or deletions alongside data reading
- Environments not using ActiveRecord or stuck on versions older than ActiveRecord 6
- Scenarios with complex multi-table joins where plucking might not handle all relational data efficiently
Pros & Cons
Pros
Reduces SQL queries by half and cuts memory allocation by up to 50% compared to manual in_batches with pluck, as benchmarked in the README, leading to faster batch processing.
Supports custom batch sizes, start/finish values, cursor columns, and ordering options, allowing fine-tuned control over iteration based on specific needs.
Enables plucking of multiple columns in a single batch iteration, and integrates with Arel for custom SQL expressions, facilitating advanced queries without full record loading.
Extends ActiveRecord with familiar methods like pluck_each and pluck_in_batches, providing a clean, expressive interface that aligns with Rails conventions.
Cons
Requires ActiveRecord 6+ and Ruby 2.7+, excluding legacy applications or projects using older Rails versions, as noted in the requirements section.
Focused solely on reading data via pluck; lacks built-in support for batch updates or deletions, requiring additional code for modification operations.
May raise errors if an order clause is present without configuring error_on_ignore, adding overhead for relations with custom sorting, as mentioned in the options.
Frequently Asked Questions
Related Projects
bootsnapBoot large Ruby/Rails apps faster
yajl-rubyA streaming JSON parsing and encoding library for Ruby (C bindings to yajl)
fast_blankfast_blank is a Ruby C extension that delivers a high-performance alternative to Active Support's `String#blank?` method. It offers significant speed improvements while maintaining compatibility with Ruby's standard behavior for determining if a string is empty or contains only whitespace. ## Key Features - **Performance Optimized** — Provides 1.2–20x faster execution compared to Active Support's implementation, depending on string length. - **Zero String Allocation** — Operates without allocating additional strings, reducing garbage collection overhead. - **Full Compatibility** — Includes `blank_as?` method for 100% compatibility with Active Support's Unicode-aware `blank?` behavior. - **MRI & Rubinius Support** — Compatible with MRI Ruby 1.9.3 through 2.2 and Rubinius 2.x. ## Philosophy fast_blank prioritizes raw performance and minimal memory overhead while maintaining behavioral parity with Ruby's native string handling, making it an ideal drop-in replacement for performance-critical applications.
fast_countQuickly get a count estimation for large tables (>99% of accuracy for PostgreSQL).
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.