Apache Spark

Deeplearning4jdeeplearning4j/deeplearning4j

A comprehensive JVM-based deep learning ecosystem for building, training, and deploying models with support for model import and distributed training.

Delta Lakedelta-io/delta

An open-source storage framework that enables building a Lakehouse architecture with ACID transactions and scalable metadata handling.

hyperopthyperopt/hyperopt

A Python library for distributed asynchronous hyperparameter optimization over complex search spaces.

Apache Hudiapache/hudi

An open data lakehouse platform for incremental data processing with upserts, deletes, and time-travel queries.

Microsoft ML for Apache SparkAzure/mmlspark

An open-source library for building massively scalable machine learning pipelines on Apache Spark.

spark-nlpJohnSnowLabs/spark-nlp

A state-of-the-art Natural Language Processing library built on Apache Spark, offering 100,000+ pretrained models and pipelines in 200+ languages.

Hunting ELK (HELK)Cyb3rWard0g/HELK

An open-source threat hunting platform with advanced analytics capabilities built on ELK stack, Apache Spark, and Jupyter notebooks.

TensorFlowOnSparkyahoo/TensorFlowOnSpark

Enables distributed TensorFlow training and inferencing on Apache Spark and Hadoop clusters with minimal code changes.

deequawslabs/deequ

A library built on Apache Spark for defining unit tests to measure data quality in large datasets.

Koalasdatabricks/koalas

Koalas provides the pandas DataFrame API on Apache Spark, enabling data scientists to work with big data using familiar pandas syntax.

Apache Kyuubiapache/kyuubi

A distributed, multi-tenant gateway providing serverless SQL on data warehouses and lakehouses.

Apache Sedonaapache/incubator-sedona

A cluster computing framework for processing large-scale geospatial data within Apache Spark, Flink, and other big data systems.

.NET for Apache Sparkdotnet/spark

.NET for Apache Spark provides high-performance .NET APIs for Apache Spark, enabling C# and F# developers to work with structured and streaming data.

DataStax Spark Cassandra Connectordatastax/spark-cassandra-connector

A connector that enables Apache Spark to read from and write to Apache Cassandra databases for distributed data processing.

Gaffergchq/Gaffer

A graph database framework for storing and querying large-scale graphs with rich properties and in-database aggregation.

Oryx 2OryxProject/oryx

A lambda architecture framework on Apache Spark and Kafka for building and deploying real-time large-scale machine learning applications.

Elephasmaxpumperla/elephas

Elephas is a Keras extension for distributed deep learning on Apache Spark, enabling data-parallel training at scale.

spark-testing-baseholdenk/spark-testing-base

Base classes for writing Apache Spark tests in Scala and Python, simplifying test setup and teardown.