Llm Inference

llama.cppC++

A C/C++ library for efficient, cross-platform LLM inference with extensive hardware support and quantization.

vllmPython

A high-throughput, memory-efficient inference and serving engine for large language models (LLMs).

gpt4allC++

Run large language models (LLMs) privately on everyday desktops and laptops without requiring API calls or GPUs.

bitnet.cppPython

Official inference framework for 1-bit LLMs, enabling fast and lossless CPU/GPU inference with significant speed and energy efficiency gains.

BentoMLPython

A Python library for building production-ready model inference APIs, job queues, and multi-model serving systems for AI applications.

mistral.rsRust

A fast, flexible, and hardware-aware LLM inference engine with zero-config support for any Hugging Face model.

planoRust

An AI-native proxy and data plane for agentic applications, providing built-in orchestration, safety, observability, and smart LLM routing.

SmileJava

A fast and comprehensive machine learning framework for Java, Scala, and Kotlin with state-of-the-art algorithms and data visualization.

shimmyRust

A lightweight, single-binary Rust inference server providing 100% OpenAI-API compatible endpoints for local GGUF models.

ruvectorRust

A self-learning vector database with graph intelligence, local AI, and PostgreSQL integration, built for real-time adaptation.