shimmy
A lightweight, single-binary Rust inference server providing 100% OpenAI-API compatible endpoints for local GGUF models.
What is shimmy?
Shimmy is a lightweight, single-binary Rust inference server that provides 100% OpenAI-API compatible endpoints for local GGUF models. It solves the problem of vendor lock-in and privacy concerns by enabling developers to run language models locally with existing OpenAI SDKs and tools, offering a drop-in replacement that requires no code changes.
Developers and teams building AI applications who want privacy, cost control, and reliability by running language models locally instead of using cloud APIs. It's particularly valuable for those using tools like VSCode Copilot, Cursor, or Continue.dev with local models.
Developers choose Shimmy for its zero-dependency deployment, automatic GPU detection, and perfect OpenAI API compatibility that works with existing tools. Its unique selling point is being a single binary that's 142x smaller than alternatives like Ollama while offering advanced features like Mixture of Experts support for large models.
Overview
⚡ Python-free Rust inference server — OpenAI-API compatible. GGUF + SafeTensors, hot model swap, auto-discovery, single binary. FREE now, FREE forever.
Use Cases
Best For
- Running local language models with existing OpenAI SDKs without code changes
Related Projects
Found a gem we're missing?
Open-Awesome is built by the community, for the community. Submit a project, suggest an awesome list, or help improve the catalog on GitHub.
