vaex
A high-performance Python DataFrame library for lazy out-of-core processing and visualization of billion-row datasets at interactive speeds.
What is vaex?
Vaex is a high-performance Python library for lazy Out-of-Core DataFrames that enables visualization and exploration of massive tabular datasets. It calculates statistics on N-dimensional grids for over a billion rows per second using memory mapping and zero-copy policies to avoid memory waste. The library supports interactive exploration through histograms, density plots, and 3D volume rendering.
Data scientists, researchers, and engineers working with large tabular datasets (billions of rows) who need efficient data processing and visualization on a single machine without moving to distributed clusters.
Vaex offers unparalleled performance for big data tasks on a laptop by leveraging memory mapping, lazy computations, and efficient algorithms, avoiding the overhead of copying data and enabling interactive exploration of datasets that would otherwise require cluster computing.
Overview
Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀
Use Cases
Best For
- Analyzing and visualizing datasets with billions of rows on a single machine
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
