Deep Spark
A thin integration layer connecting Apache Spark with various NoSQL datastores and JDBC databases.
What is Deep Spark?
Deep (Deep-Spark) is an open-source integration layer that connects Apache Spark with various NoSQL datastores and JDBC databases. It enables Spark to read from and write to systems like Cassandra, MongoDB, Elasticsearch, and Aerospike, providing a unified API for data processing across heterogeneous sources. The project simplifies big data workflows by abstracting connector complexities and allowing developers to work with Spark RDDs directly mapped to database entities.
Data engineers and developers working with Apache Spark who need to integrate multiple data stores (NoSQL and SQL) into their Spark processing pipelines. It is particularly useful for teams managing polyglot persistence environments.
Deep offers a single, consistent API for multiple datastores, reducing the need for custom connectors. Its dual interface (ORM and cell-based) provides flexibility for both structured and schema-less data, and it optimizes data fetching to leverage Spark's distributed processing capabilities efficiently.
Overview
Connecting Apache Spark with different data stores [DEPRECATED]
Use Cases
Best For
- Processing data from Cassandra column families in Spark applications
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
