Quantization

llama.cppC++

A C/C++ library for efficient, cross-platform LLM inference with extensive hardware support and quantization.

vllmPython

A high-throughput, memory-efficient inference and serving engine for large language models (LLMs).

sglangPython

A high-performance serving framework for large language models and multimodal models, delivering low-latency and high-throughput inference.

faster-whisperPython

A fast, memory-efficient reimplementation of OpenAI's Whisper speech-to-text model using CTranslate2.

candle-wasm-examplesRust

A minimalist, high-performance machine learning framework for Rust with a focus on serverless inference and GPU support.

ggmlC++

A low-level tensor library for machine learning with integer quantization, automatic differentiation, and zero runtime allocations.

BigDLPython

An LLM acceleration library for Intel XPU (GPU, NPU, CPU) to speed up local inference and finetuning of popular models.

mistral.rsRust

A fast, flexible, and hardware-aware LLM inference engine with zero-config support for any Hugging Face model.

mozjpegC

A JPEG encoder library that improves compression efficiency for higher quality and smaller file sizes.