Dataset
A Python library for building lazy data processing and machine learning workflows that handle datasets larger than memory.
What is Dataset?
BatchFlow is a Python library that provides tools for building and executing data processing and machine learning workflows on large datasets. It solves the problem of working with datasets that do not fit into memory by enabling lazy, batch-based operations and pipeline definitions.
Data scientists, machine learning engineers, and researchers who need to process large-scale datasets or build complex, reproducible ML pipelines efficiently.
Developers choose BatchFlow for its ability to handle out-of-memory datasets seamlessly, its flexible pipeline system with lazy evaluation, and its built-in support for proven neural network architectures and parallel training.
Overview
BatchFlow helps you conveniently work with random or sequential batches of your data and define data processing and machine learning workflows even for datasets that do not fit into memory.
Use Cases
Best For
- Processing datasets larger than available RAM
- Building reproducible machine learning pipelines
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
