Inference

vllmPython

A high-throughput, memory-efficient inference and serving engine for large language models (LLMs).

whisper.cppC++

High-performance C/C++ port of OpenAI's Whisper for efficient, cross-platform speech recognition.

Bindings for many languagesC++

A high-performance C/C++ port of OpenAI's Whisper model for efficient, cross-platform speech recognition.

Colossal-AI - An Integrated Large-scale Model Training System with Efficient Parallelization TechniquesPython

A unified deep learning system for efficient large-scale model training and inference with advanced parallelism strategies.

MediaPipeC++

Cross-platform framework for building customizable on-device machine learning pipelines for live and streaming media.

sglangPython

A high-performance serving framework for large language models and multimodal models, delivering low-latency and high-throughput inference.

ncnnC++

A high-performance neural network inference framework optimized for mobile platforms, enabling efficient AI deployment on edge devices.

faster-whisperPython

A fast, memory-efficient reimplementation of OpenAI's Whisper speech-to-text model using CTranslate2.

ONNXPython

An open standard format for representing machine learning models to enable interoperability between frameworks.

ts-patternTypeScript

An exhaustive pattern matching library for TypeScript with smart type inference and expressive API.

ggmlC++

A low-level tensor library for machine learning with integer quantization, automatic differentiation, and zero runtime allocations.

TensorRTC++

NVIDIA's SDK for high-performance deep learning inference optimization and deployment on NVIDIA GPUs.

triton-inference-serverPython

An open-source inference serving platform for deploying AI models from multiple frameworks across cloud, data center, and edge devices.

RunAnywhereC++

SDKs for adding private, on-device AI features like LLM chat, speech-to-text, and text-to-speech to mobile and web apps.

simple-statisticsJavaScript

A lightweight, dependency-free JavaScript library for descriptive, regression, and inference statistics.

nnablaPython

A deep learning framework for research, development, and production with flexible Python API and C++ core.