Apache Spark

Kotlin for Apache SparkKotlin/kotlin-spark-api

Kotlin bindings and extensions for Apache Spark, enabling idiomatic Kotlin development with data classes, lambdas, and null safety.

spark-fast-testsmrpowers-io/spark-fast-tests

A fast Apache Spark testing helper library with beautifully formatted error messages for Scala applications.

sparkletweag/sparkle

A library for writing Apache Spark applications in Haskell, enabling resilient analytics that scale to thousands of nodes.

sparkllinggorillalabs/sparkling

A fast, fully-featured, and developer-friendly Clojure API for Apache Spark.

SparklingPandassparklingpandas/sparklingpandas

A Python library that provides a Pandas-like API on top of Apache Spark DataFrames for distributed data analysis.

neo4j-spark-connectorneo4j-contrib/neo4j-spark-connector

A bi-directional connector enabling Apache Spark to read from and write to Neo4j graph databases using Spark DataSource APIs.

Hivemallapache/incubator-hivemall

A scalable machine learning library that runs on Apache Hive, Spark, and Pig for distributed ML directly in SQL.

Spark-MongoDBStratio/Spark-MongoDB

A Spark library for reading and writing data between Spark SQL and MongoDB collections.

Genizero-one-group/geni

An idiomatic Clojure dataframe library that runs on Apache Spark, providing a seamless interface for data processing and machine learning.

isolation-forestlinkedin/isolation-forest

A distributed Spark/Scala implementation of Isolation Forest and Extended Isolation Forest algorithms for scalable unsupervised outlier detection.

hgraphdbrayokota/hgraphdb

A client layer that enables HBase to function as a TinkerPop 3-compliant graph database.

ruby-sparkondra-m/ruby-spark

A Ruby wrapper for Apache Spark, enabling large-scale data processing with Ruby's expressive syntax.

DynaMLtranscendent-ai-labs/DynaML

A Scala and JVM machine learning toolbox for research, education, and industry with an interactive REPL and end-to-end pipelines.

Deep SparkStratio/deep-spark

A thin integration layer connecting Apache Spark with various NoSQL datastores and JDBC databases.

LiFTlinkedin/LiFT

A Scala/Spark library for measuring fairness and mitigating bias in large-scale machine learning workflows.

CrossdataStratio/Crossdata

A distributed framework extending Apache Spark with unified SQL access to multiple datastores, optimized connectors, and streaming support.

ArchiveSparkhelgeho/ArchiveSpark

An Apache Spark framework for efficient data processing, extraction, and derivation from web archives and archival collections.

Archives Unleashed Toolkitarchivesunleashed/aut

An open-source toolkit for analyzing web archives at scale using Apache Spark.

Spark-BigQueryspotify/spark-bigquery

A Spark library for reading from and writing to Google BigQuery using DataFrames and SQL.

yuritapaypal/yurita

An open-source framework for developing large-scale anomaly detection models using Apache Spark.

JPMML-Sparkjpmml/jpmml-spark

A PMML evaluator library for Apache Spark that provides ML-compatible transformers for deploying predictive models.

Scylla-Migratorscylladb/scylla-migrator

A Spark application for migrating data to ScyllaDB from CQL-compatible databases or DynamoDB via Alternator.

RS-FISHPreibischLab/RS-FISH

An open-source tool for precise, interactive, fast, and scalable spot detection in 2D/3D microscopy images for FISH-based spatial genomics.

timely-security-analyticsawslabs/timely-security-analytics

Demo code for analyzing AWS CloudTrail and S3 logs using Apache Spark to detect security anomalies and enable SQL queries.

Archives Unleashed Notebooksarchivesunleashed/notebooks

Example notebooks for analyzing web archives using the Archives Unleashed Toolkit.

sample KafkaSparkCassandrainstaclustr/sample-KafkaSparkCassandra

A Scala sample application demonstrating Spark Streaming integration with Kafka and Cassandra for data processing.

Sparklinginternetarchive/Sparkling

A Scala/Spark library for efficient processing, extraction, and derivation of web archive data (CDX/WARC).

Tweet Archvies Unleashed Toolkitarchivesunleashed/twut

An open-source toolkit for analyzing line-oriented JSON Twitter archives using Apache Spark.

Spin up PySpark and PySparkling on AWSkcrandall/EMR_Spark_Automation

Automates AWS EMR cluster deployment and Spark job submission with integrated logging and PySparkling support for H2O.

sample Spark Job Server Cassandrainstaclustr/sample-SparkJobserverCassandra

A sample Spark job demonstrating how to use Spark Jobserver to run Apache Spark analytics with Cassandra.

spark-resources-metrics-plugindutrevis/spark-resources-metrics-plugin

An Apache Spark plugin that collects system resource metrics (CPU, memory) not provided by Spark's native metrics system.

sample Spark Cassandra with SSLinstaclustr/sample-SparkCassandrawithSSL

A sample Spark job demonstrating analytics on Cassandra with SSL encryption for secure data processing.

Import CSV files with sparkmarkthebault/importCSVSparkCassandra

Import CSV files from AWS S3 into Cassandra using Apache Spark with a simple configuration-based approach.

TD-Sparkliuyq-617/TD-Spark

A Java project that reads from and writes to TDengine using Apache Spark for data processing.