Gpu Optimization

vllmPython

A high-throughput, memory-efficient inference and serving engine for large language models (LLMs).

Colossal-AI - An Integrated Large-scale Model Training System with Efficient Parallelization TechniquesPython

A unified deep learning system for efficient large-scale model training and inference with advanced parallelism strategies.

triton-inference-serverPython

An open-source inference serving platform for deploying AI models from multiple frameworks across cloud, data center, and edge devices.

BigDLPython

An LLM acceleration library for Intel XPU (GPU, NPU, CPU) to speed up local inference and finetuning of popular models.

BentoMLPython

A Python library for building production-ready model inference APIs, job queues, and multi-model serving systems for AI applications.