How does Sep compare to CsvHelper for parsing CSV files?

Sep significantly outperforms CsvHelper in speed, often by multiples due to SIMD and zero-allocation design, with benchmarks showing up to 35x faster parallel parsing. However, CsvHelper offers more features like automatic type mapping and extensive configuration, making it better for complex scenarios where performance isn't critical.

How to handle CSV files with comments in Sep?

Sep doesn't directly support comments. You need to manually check each row's span for comment markers (e.g., '#') and skip them within your enumeration loop, as described in the limitations section, which adds extra code for preprocessing.

Is Sep compatible with .NET Framework or older .NET Core versions?

No, Sep requires .NET 7 or higher due to its reliance on modern features like Span and ref structs. For older frameworks, consider libraries like CsvHelper or Sylvan that support broader .NET versions.

Can I use Sep with LINQ for data querying?

No, because SepReader.Row is a ref struct, it cannot be used directly with LINQ. You must enumerate rows and parse them into other types first, or use Sep's Enumerate methods with custom delegates, as explained in the API section.

What are the performance trade-offs when enabling unescaping in Sep?

Enabling unescaping adds a slight overhead due to extra state tracking during parsing, but Sep remains fast. Benchmarks show a minor slowdown in column access scope, but it's still competitive, though in-place modification can complicate debugging.

How to write objects like records to CSV using Sep?

You need to map object properties to columns manually using SepWriter, as Sep doesn't provide automatic object serialization. The example catalogue shows how to write records by setting columns individually, which requires more boilerplate code compared to libraries with built-in mapping.

Does Sep support async reading and writing for large files?

Yes, Sep supports efficient async reading with ValueTask and IAsyncEnumerable for .NET 9+/C# 13+, allowing non-blocking IO. However, async usage has limitations with ref structs and requires careful handling, as detailed in the async support section.

Open-Awesome

Sep

MITC#v0.17.0

A modern, minimal, and high-performance .NET library for reading and writing CSV/TSV files with zero allocations and SIMD-accelerated parsing.

Visit Website GitHub

1.5k stars54 forks0 contributors

What is Sep?

Sep is a high-performance .NET library for reading and writing separated values like CSV and TSV files. It solves the need for fast, memory-efficient data parsing in modern .NET applications, especially in machine learning and data-intensive scenarios, by leveraging SIMD vectorization and zero-allocation techniques.

Target Audience

.NET developers working with large CSV/TSV datasets, particularly in machine learning, data science, or high-throughput data processing applications where performance and memory efficiency are critical.

Value Proposition

Developers choose Sep for its unmatched speed (up to 35x faster than CsvHelper), zero-allocation design, and modern API. It outperforms other .NET CSV parsers with SIMD acceleration, parallel parsing, and seamless integration with latest .NET features.

Overview

World's Fastest .NET CSV Parser. Modern, minimal, fast, zero allocation, reading and writing of separated values (csv, tsv etc.). Cross-platform, trimmable and AOT/NativeAOT compatible with blazing fast SIMD vectorized parsing.

Use Cases

Best For

Parsing large CSV files for machine learning feature extraction
High-throughput data processing pipelines requiring minimal memory overhead
Applications needing AOT/NativeAOT compatibility for small deployment sizes
Real-time data ingestion with strict performance requirements
Multi-threaded CSV parsing for parallel data analysis
Scenarios where zero allocations after warmup is critical for GC pressure

Not Ideal For

Applications requiring extensive CSV validation, automatic type mapping, or complex data transformations beyond basic parsing
Teams stuck on .NET Framework or older .NET Core versions, as Sep requires .NET 7+
Projects that heavily depend on LINQ for immediate data manipulation post-parsing, due to ref struct limitations
Scenarios where drop-in simplicity and extensive community support are more critical than raw performance

Pros & Cons

Pros

SIMD-Accelerated Speed

Uses architecture-specific SIMD vectorization (AVX2, AVX-512, NEON) and csFastFloat to achieve parsing speeds up to 21 GB/s on modern hardware, as demonstrated in benchmarks against CsvHelper and Sylvan.

Zero Allocation Design

Intelligent memory management with ArrayPool<T> and pooled string handling eliminates allocations after warmup, even for array-based operations, reducing GC pressure significantly in data-intensive scenarios.

Parallel Parsing Support

ParallelEnumerate provides multi-threaded parsing with up to 35x speedup over alternatives for large datasets, leveraging .NET's thread pool efficiently for machine learning use cases.

Modern .NET Integration

Built with .NET 7+/C# 11+ features like Span<T>, ref struct, and ISpanParsable, ensuring optimal performance and seamless integration with latest .NET advancements for cross-platform and AOT compatibility.

AOT/NativeAOT Compatibility

Fully trimmable and NativeAOT compatible, producing small executables (a few MBs) ideal for deployment in constrained environments, as highlighted in the README's philosophy.

Cons

Ref Struct Limitations

API uses ref structs for rows and columns, preventing LINQ compatibility and making it impossible to store rows in arrays or use common .NET collection patterns without extra parsing into other types, as admitted in the 'Why SepReader Was Not IEnumerable' section.

Minimal Feature Set

Lacks built-in support for comments, automatic escaping/unescaping by default, and advanced CSV features like schema validation, requiring manual implementation for edge cases as noted in the limitations section.

Complex Configuration for Advanced Use

Enabling features like string pooling or unescaping requires explicit options configuration, and in-place modification for unescaping/trimming can lead to confusing state if not handled correctly, adding overhead for developers.

Separator and API Constraints

Separator char is validated and limited (not any char can be used), and the opinionated API with short names (e.g., Sep, Col) may be less intuitive for those accustomed to more descriptive libraries, potentially increasing onboarding time.

Frequently Asked Questions

Related Projects

Polly

Polly is a .NET resilience and transient-fault-handling library that allows developers to express policies such as Retry, Circuit Breaker, Timeout, Bulkhead Isolation, and Fallback in a fluent and thread-safe manner. From version 6.0.1, Polly targets .NET Standard 1.1 and 2.0+.

Stars14,236

Forks1,290

Last commit2 days ago

Humanizer

Humanizer meets all your .NET needs for manipulating and displaying strings, enums, dates, times, timespans, numbers and quantities

Stars9,865

Forks1,067

Last commit4 days ago

Fluent Validation

A popular .NET validation library for building strongly-typed validation rules.

Stars9,752

Forks1,276

Last commit27 days ago

CsvHelper

Library to help reading and writing CSV files

Stars5,211

Forks1,104

Last commit1 year ago

Community-curated · Updated weekly · 100% open source

Found a gem we're missing?

Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.

Submit a project Star on GitHub

Sep