How to process large CSV files without running out of memory in Ruby?

Use SmarterCSV's chunked processing with the chunk_size option or the Enumerator API via Reader#each, which streams rows in batches to minimize memory usage, ideal for database imports or parallel jobs.

SmarterCSV vs Ruby CSV: which should I use for a Rails app?

Choose SmarterCSV for production data imports due to its speed, robustness, and Rails-ready hashes; Ruby CSV is fine for simple, controlled files but risks silent data corruption and requires more post-processing code.

How to skip or quarantine bad rows in a CSV import with SmarterCSV?

Set the on_bad_row: :collect option when creating a Reader, which accumulates errors in reader.errors[:bad_rows] without halting, allowing you to report or retry problematic lines separately.

Does SmarterCSV work with JRuby or other Ruby implementations?

Yes, but it falls back to a pure Ruby implementation without the C extension, so performance is slower; it's fully functional but misses the acceleration benefits on MRI Ruby.

How to convert CSV strings to hashes with automatic number parsing in SmarterCSV?

Use SmarterCSV.parse(csv_string), which by default converts numeric values, strips whitespace, and returns symbol-keyed hashes, similar to CSV.parse with headers but more robust and faster.

What are the real-world performance gains of SmarterCSV compared to other libraries?

Benchmarks show it's 7–129× faster than CSV.table for equivalent hash output, and often outperforms ZSV when including full pipeline work like hash construction and conversions, not just tokenization.

smarter_csv — High-Performance Ruby CSV

What is smarter_csv?

SmarterCSV is a high-performance CSV ingestion and generation library for Ruby that solves the silent data corruption issues of Ruby's built-in CSV library. It provides fast, robust parsing with intelligent defaults, returning Rails-ready hashes for seamless integration with ActiveRecord, Sidekiq, and data pipelines.

Target Audience

Ruby developers and teams working with CSV data imports, especially those using Rails, Sidekiq, or building data pipelines that handle messy, real-world user-uploaded files.

Value Proposition

Developers choose SmarterCSV for its combination of raw speed (C-accelerated), robustness against common CSV pitfalls, and developer-friendly output that eliminates post-processing boilerplate, significantly reducing import time and code complexity.

Fastest end-to-end CSV ingestion for Ruby (with C acceleration). SmarterCSV auto-detects formats, applies smart defaults, and returns Rails-ready hashes for seamless use with ActiveRecord, Sidekiq, parallel jobs, and S3 pipelines — even for messy user-uploaded real-world data.

Use Cases

Best For

Importing large CSV files into Rails applications with ActiveRecord
Building resilient data pipelines that handle messy, user-uploaded CSV
Processing CSV in parallel with Sidekiq or other background job systems
Reducing CSV import times from hours to minutes with C acceleration
Quarantining bad rows instead of crashing entire CSV imports
Streaming or chunking massive CSV files to minimize memory usage

Not Ideal For

Projects on JRuby or TruffleRuby where maximum parsing speed is critical, as the C acceleration is unavailable
Simple, one-off CSV parsing scripts where Ruby's built-in CSV library suffices and adding a gem dependency is unnecessary
Applications requiring raw, unmodified CSV data without automatic transformations like whitespace stripping or numeric conversion

Pros & Cons

Pros

Blazing Fast Performance

With C acceleration, it's up to 129× faster than CSV.table for end-to-end ingestion, including hash construction and conversions, as shown in benchmarks with real-world files.

Robust Error Handling

Handles 8 out of 10 common CSV failure modes by default, such as duplicate headers and BOMs, preventing silent data corruption that plagues Ruby's built-in CSV.

Developer-Friendly Output

Returns Rails-ready hashes with symbol keys out of the box, eliminating post-processing boilerplate for direct use with ActiveRecord, Sidekiq, and JSON workflows.

Advanced Processing Features

Supports chunked and streaming APIs for memory-efficient large file handling, plus bad row quarantine to collect errors without crashing entire imports.

Cons

C Extension Dependency

Requires native compilation on MRI Ruby, which can fail on systems without build tools; while it falls back to Ruby, performance drops significantly.

Configuration Overhead

With extensive transformation pipelines and over 30 options, it can be overwhelming for simple tasks, demanding careful tuning to avoid unintended data changes.

Limited Performance on Alternative Rubies

On JRuby and TruffleRuby, the library falls back to pure Ruby, missing the C acceleration and resulting in slower speeds compared to MRI.

Frequently Asked Questions

What is smarter_csv?

Target Audience

Ruby developers and teams working with CSV data imports, especially those using Rails, Sidekiq, or building data pipelines that handle messy, real-world user-uploaded files.

Value Proposition

Use Cases

Best For

Importing large CSV files into Rails applications with ActiveRecord
Building resilient data pipelines that handle messy, user-uploaded CSV
Processing CSV in parallel with Sidekiq or other background job systems
Reducing CSV import times from hours to minutes with C acceleration
Quarantining bad rows instead of crashing entire CSV imports
Streaming or chunking massive CSV files to minimize memory usage

Not Ideal For

Projects on JRuby or TruffleRuby where maximum parsing speed is critical, as the C acceleration is unavailable
Simple, one-off CSV parsing scripts where Ruby's built-in CSV library suffices and adding a gem dependency is unnecessary
Applications requiring raw, unmodified CSV data without automatic transformations like whitespace stripping or numeric conversion

Pros & Cons

Pros

Blazing Fast Performance

With C acceleration, it's up to 129× faster than CSV.table for end-to-end ingestion, including hash construction and conversions, as shown in benchmarks with real-world files.

Robust Error Handling

Handles 8 out of 10 common CSV failure modes by default, such as duplicate headers and BOMs, preventing silent data corruption that plagues Ruby's built-in CSV.

Developer-Friendly Output

Returns Rails-ready hashes with symbol keys out of the box, eliminating post-processing boilerplate for direct use with ActiveRecord, Sidekiq, and JSON workflows.

Advanced Processing Features

Supports chunked and streaming APIs for memory-efficient large file handling, plus bad row quarantine to collect errors without crashing entire imports.

Cons

C Extension Dependency

Requires native compilation on MRI Ruby, which can fail on systems without build tools; while it falls back to Ruby, performance drops significantly.

Configuration Overhead

With extensive transformation pipelines and over 30 options, it can be overwhelming for simple tasks, demanding careful tuning to avoid unintended data changes.

Limited Performance on Alternative Rubies

On JRuby and TruffleRuby, the library falls back to pure Ruby, missing the C acceleration and resulting in slower speeds compared to MRI.

Frequently Asked Questions

smarter_csv

What is smarter_csv?

Overview

Use Cases

Best For

Not Ideal For

Pros & Cons

Pros

Cons

Frequently Asked Questions

Related Projects

Found a gem we're missing?

smarter_csv

What is smarter_csv?

Overview

Use Cases

Best For

Not Ideal For

Pros & Cons

Pros

Cons

Frequently Asked Questions

Related Projects

Found a gem we're missing?