Koalas
Koalas provides the pandas DataFrame API on Apache Spark, enabling data scientists to work with big data using familiar pandas syntax.
What is Koalas?
Koalas is an open-source library that provides the pandas DataFrame API on top of Apache Spark, allowing data scientists to use familiar pandas syntax for big data processing. It solves the problem of transitioning from single-node pandas workflows to distributed Spark environments without requiring users to learn a new API. The project is now deprecated as its functionality has been integrated into PySpark starting with Apache Spark 3.2.
Data scientists and Python developers who are familiar with pandas and need to scale their data processing to large, distributed datasets using Apache Spark.
Developers choose Koalas because it dramatically reduces the learning curve for using Spark, enables code reuse across pandas and Spark environments, and increases productivity by leveraging existing pandas knowledge for big data tasks.
Overview
Koalas: pandas API on Apache Spark
Use Cases
Best For
- Migrating pandas workflows to distributed Spark clusters
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.