omniparser
A native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, and custom formats.
What is omniparser?
Omniparser is a native Golang ETL parser that ingests input data of various formats (CSV, fixed-length text, XML, EDI/X12/EDIFACT, JSON, and custom formats) in a streaming fashion and transforms it into desired JSON output based on a schema written in JSON. It addresses the lack of robust, streaming-capable ETL libraries in the Go ecosystem, offering a flexible solution for data processing pipelines.
Go developers building data processing pipelines that require streaming ingestion of multiple file formats without loading entire inputs into memory. It is particularly suited for engineers working with EDI, hierarchical data, or custom formats who need a schema-driven transformation tool.
Developers choose Omniparser because it provides a performant, extensible, and streaming-efficient ETL library specifically for Go, filling a gap where alternatives in other languages are limited, heavyweight, or lack streaming support. Its unique selling points include a JSON-based schema system, support for complex formats like EDI/X12/EDIFACT, and extensibility through custom functions and plugins.
Overview
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
Use Cases
Best For
- Streaming ETL pipelines in Go that process large CSV, fixed-length, or XML files without loading entire inputs into memory.
- Transforming EDI/X12/EDIFACT data into JSON using a declarative schema, avoiding heavyweight code-generation approaches.
- Building custom data parsers for proprietary or non-standard file formats via its extensible plugin architecture.
- Implementing complex data transformations with JavaScript (ES6) logic embedded within schema definitions for advanced manipulation.
- Creating command-line tools or HTTP services for one-off or long-running data transformation tasks using the provided CLI.
- Filtering and extracting data from hierarchical formats like XML or JSON using XPath-based queries within schemas.
Not Ideal For
- Teams needing a GUI or low-code interface for ETL transformations without writing JSON schemas.
- Projects in non-Go environments that require a language-agnostic ETL solution.
- Simple data munging tasks where a quick Python script is more efficient than schema development.
- Real-time systems requiring sub-millisecond processing latency, as streaming may introduce overhead.
Pros & Cons
Pros
Processes large CSV, XML, and EDI files without loading entire inputs into memory, addressing a key limitation in other libraries that lack streaming support.
Supports complex formats like EDI/X12/EDIFACT, which alternatives such as Jolt or JSONata cannot handle, making it unique in the Go ecosystem.
Allows adding custom functions, schema handlers, and new file formats via plugins, enabling adaptation to proprietary data sources as shown in the custom file format examples.
Includes a `javascript` custom function with full ES6 support, providing advanced data manipulation within schemas, as detailed in the custom functions documentation.
Cons
The online playground is non-functional, removing a valuable tool for testing schemas without local setup, forcing users to rely on CLI or code.
Writing schemas requires understanding XPath, custom functions, and nested JSON structures, leading to a steep learning curve, especially for hierarchical data.
Major updates have introduced incompatible schema changes, such as the shift to 'omni.2.1' with renamed fields, which can disrupt existing pipelines.
Frequently Asked Questions
Related Projects
BenthosFancy stream processing made operationally mundane
CloudQueryData pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.