datatable
A high-performance Python package for fast, multi-threaded manipulation of large tabular datasets, inspired by R's data.table.
What is datatable?
Datatable is a Python package for manipulating two-dimensional tabular data structures, similar to pandas or R's data.table. It is designed for high-speed processing of large datasets (up to 100GB) on a single machine, with a focus on performance, multi-threading, and efficient memory usage. The library aims to support modern machine-learning applications that require processing large volumes of data to generate features for model accuracy.
Data scientists, machine learning engineers, and developers working with large tabular datasets who need fast data manipulation and feature engineering capabilities. It is particularly useful for those dealing with big data on single-node systems.
Developers choose Datatable for its superior speed and efficiency in handling big data operations compared to alternatives like pandas. Its native C implementation, multi-threaded processing, and memory-mapped datasets allow for faster data manipulation and reduced memory overhead, making it ideal for performance-critical applications.
Overview
A Python package for manipulating 2-dimensional tabular data structures
Use Cases
Best For
- Processing large datasets (up to 100GB) on a single machine
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
