llama.cpp
A C/C++ library for efficient, cross-platform LLM inference with extensive hardware support and quantization.
What is llama.cpp?
llama.cpp is an open-source library for running large language model (LLM) inference locally using C/C++. It provides efficient, dependency-free execution of models like LLaMA, Mistral, and Gemma across diverse hardware, from consumer CPUs to enterprise GPUs. The project solves the need for performant, portable LLM inference without cloud dependencies.
Developers and researchers who need to deploy LLMs on local hardware, edge devices, or specialized infrastructure (e.g., Apple Silicon, embedded systems). It's also used by AI application builders creating offline-capable chat tools, coding assistants, or multimodal systems.
Developers choose llama.cpp for its unmatched performance per watt, extensive hardware support, and avoidance of Python/PyTorch overhead. Its quantization support enables running billion-parameter models on consumer hardware, while the permissively licensed codebase allows integration into commercial products.
Overview
LLM inference in C/C++
Use Cases
Best For
- Running LLMs locally on Apple Silicon Macs with Metal acceleration
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
