Model Parallelism

Colossal-AI - An Integrated Large-scale Model Training System with Efficient Parallelization TechniquesPython

A unified deep learning system for efficient large-scale model training and inference with advanced parallelism strategies.

DSSTNE - Amazon's library for building Deep Learning modelsC++

An open-source library for training and deploying deep learning recommendation models with sparse data at scale using multi-GPU support.

Mesh TensorFlowPython

A language for distributed deep learning that simplifies model parallelism by specifying tensor computations across hardware meshes.