Spark-BigQuery
A Spark library for reading from and writing to Google BigQuery using DataFrames and SQL.
What is Spark-BigQuery?
spark-bigquery is an open-source library that enables Apache Spark to read data from and write data to Google BigQuery directly. It provides Spark SQL and DataFrame APIs for interacting with BigQuery tables, allowing users to run queries and perform distributed data processing without manual data transfers. The library handles GCP authentication, data type mappings, and optimizations for BigQuery's storage system.
Data engineers and data scientists working in GCP environments who use Spark for large-scale data processing and need to integrate with BigQuery datasets. It's particularly useful for teams running Spark on Google Cloud Dataproc clusters.
Developers choose spark-bigquery for its native Spark integration, which simplifies ETL pipelines by eliminating the need for intermediate storage when moving data between Spark and BigQuery. It offers a programmatic alternative to manual BigQuery exports/imports, though note the project is in maintenance mode with best-effort support.
Overview
Google BigQuery support for Spark, SQL, and DataFrames
Use Cases
Best For
- Running Spark analytics on datasets stored in Google BigQuery
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
