llama_cpp

gpt4allnomic-ai/gpt4all

Run large language models (LLMs) privately on everyday desktops and laptops without requiring API calls or GPUs.

LocalAImudler/LocalAI

An open-source AI engine that runs LLMs, vision, voice, and image/video models on any hardware with drop-in OpenAI API compatibility.

bitnet.cppmicrosoft/BitNet

Official inference framework for 1-bit LLMs, enabling fast and lossless CPU/GPU inference with significant speed and energy efficiency gains.

RunAnywhereRunanywhereAI/runanywhere-sdks

SDKs for adding private, on-device AI features like LLM chat, speech-to-text, and text-to-speech to mobile and web apps.

spark-nlpJohnSnowLabs/spark-nlp

A state-of-the-art Natural Language Processing library built on Apache Spark, offering 100,000+ pretrained models and pipelines in 200+ languages.

shimmyMichael-A-Kuykendall/shimmy

A lightweight, single-binary Rust inference server providing 100% OpenAI-API compatible endpoints for local GGUF models.

koharumayocream/koharu

A local-first, ML-powered desktop application for translating manga, built in Rust with automated text detection, OCR, inpainting, and LLM translation.

LLamaSharpSciSharp/LLamaSharp

A C#/.NET library for efficient local inference of LLaMA and other large language models, based on llama.cpp.