sparkmagic
Jupyter magics and kernels for interactively working with remote Spark clusters via Livy, Lighter, or Ilum.
What is sparkmagic?
Sparkmagic is a Jupyter extension that provides magics and kernels for interactively working with remote Apache Spark clusters. It allows users to run Spark code in Scala, Python, or R directly from Jupyter notebooks by connecting to REST servers like Livy, enabling seamless data analysis and processing on distributed clusters without local Spark setup.
Data scientists, data engineers, and analysts who use Jupyter notebooks for big data processing and need to interact with remote Spark clusters for scalable computations.
Developers choose Sparkmagic for its ability to integrate remote Spark clusters into Jupyter workflows effortlessly, offering multi-language support, automatic visualization, and flexible authentication, all without requiring Spark installations on the local machine.
Overview
Jupyter magics and kernels for working with remote Spark clusters
Use Cases
Best For
- Running interactive Spark queries from Jupyter notebooks on remote clusters
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.