spark-testing-base
Base classes for writing Apache Spark tests in Scala and Python, simplifying test setup and teardown.
What is spark-testing-base?
spark-testing-base is a library that provides base classes and utilities for writing tests for Apache Spark applications. It simplifies the process of setting up and tearing down local Spark sessions, reducing boilerplate code and allowing developers to write focused, efficient tests. The library supports both Scala and Python, integrating with common build tools and testing frameworks.
Data engineers and developers who write and maintain Apache Spark applications in Scala or Python and need to create reliable unit and integration tests.
Developers choose spark-testing-base because it eliminates repetitive Spark session management code, provides a consistent testing foundation inspired by Apache Spark's internal test utilities, and supports both Scala and Python ecosystems with easy dependency management.
Overview
Base classes to use when writing tests with Spark
Use Cases
Best For
- Writing unit tests for Spark DataFrame and Dataset transformations
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.