docker-spark
A Docker image for Apache Spark on YARN, built on Hadoop and CentOS for easy deployment.
What is docker-spark?
SequenceIQ's docker-spark is a Docker image that packages Apache Spark with Hadoop and YARN for simplified deployment. It provides a pre-configured environment to run Spark applications in containerized setups, eliminating manual installation and configuration hassles. This is particularly useful for developers and data engineers who need a reproducible Spark stack for testing or development.
Data engineers, developers, and DevOps professionals who want to quickly deploy and test Apache Spark in Docker containers, especially for YARN-based cluster environments.
It offers a ready-to-use, Dockerized Spark setup that reduces deployment time and ensures consistency across environments, building on a stable Hadoop base for reliable big data processing.
Overview
This project provides a Dockerized version of Apache Spark, pre-configured to run on YARN with Hadoop 2.6.0. It simplifies setting up a Spark environment by packaging everything into a container, making it ideal for development, testing, and reproducible deployments.
Key Features
- Pre-configured Spark on YARN — Includes Apache Spark 1.6.0 and Hadoop 2.6.0, ready to run in YARN-client or YARN-cluster modes.
- Docker-based Deployment — Offers Docker images for easy pulling, building, and running, reducing setup complexity.
- CentOS Base — Built on a stable CentOS foundation for compatibility and reliability.
- Remote Submission Support — Allows submitting Spark jobs from outside the container using environment variables like YARN_CONF_DIR.
- Integrated Testing Examples — Includes sample commands for testing Spark functionality, such as estimating Pi.
Philosophy
The project focuses on providing a streamlined, containerized Spark environment that leverages Docker for consistency and ease of use, building on existing Hadoop Docker images to ensure compatibility.
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
