Spark XML
A library for parsing and querying XML data with Apache Spark SQL and DataFrames.
What is Spark XML?
spark-xml is an open-source library that provides XML data source functionality for Apache Spark, allowing users to read and write XML files as Spark DataFrames. It solves the problem of processing large, format-free XML files in a distributed manner, which isn't fully supported by Spark's built-in data sources. The library offers extensive configuration options for handling XML complexities like attributes, namespaces, and schema inference.
Data engineers and data scientists working with XML data in Apache Spark pipelines, particularly those needing to process large XML datasets in distributed computing environments.
Developers choose spark-xml because it provides production-ready XML processing for Spark with robust error handling, schema flexibility, and multi-language API support. Its planned integration into Apache Spark 4.0 ensures long-term compatibility and community support.
Overview
XML data source for Spark SQL and DataFrames
Use Cases
Best For
- Processing large XML files in distributed Spark clusters
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
