Openai Api Compatible

privateGPTPython

A production-ready API for building private, context-aware AI applications that query documents using local LLMs with no data leaving your environment.

FastChatPython

An open platform for training, serving, and evaluating large language model based chatbots.

sglangPython

A high-performance serving framework for large language models and multimodal models, delivering low-latency and high-throughput inference.

shimmyRust

A lightweight, single-binary Rust inference server providing 100% OpenAI-API compatible endpoints for local GGUF models.

TurboPilotC++

An open-source, locally-runnable code completion engine using large language models that works on CPU.