# The Privacy-First Alternative to Ollama
### π Local AI Without the Lock-in π
[](https://opensource.org/licenses/MIT)
[](https://github.com/Michael-A-Kuykendall/shimmy/security)
[](https://crates.io/crates/shimmy)
[](https://crates.io/crates/shimmy)
[](https://rustup.rs/)
[](https://github.com/Michael-A-Kuykendall/shimmy/stargazers)
[](https://github.com/sponsors/Michael-A-Kuykendall)
**Shimmy will be free forever.** No asterisks. No "free for now." No pivot to paid.
### π Support Shimmy's Growth
π **If Shimmy helps you, consider [sponsoring](https://github.com/sponsors/Michael-A-Kuykendall) β 100% of support goes to keeping it free forever.**
- **$5/month**: Coffee tier β - Eternal gratitude + sponsor badge
- **$25/month**: Bug prioritizer π - Priority support + name in [SPONSORS.md](SPONSORS.md)
- **$100/month**: Corporate backer π’ - Logo placement + monthly office hours
- **$500/month**: Infrastructure partner π - Direct support + roadmap input
[**π― Become a Sponsor**](https://github.com/sponsors/Michael-A-Kuykendall) | See our amazing [sponsors](SPONSORS.md) π
---
## Drop-in OpenAI API Replacement for Local LLMs
Shimmy is a **5.1MB single-binary** that provides **100% OpenAI-compatible endpoints** for GGUF models. Point your existing AI tools to Shimmy and they just work β locally, privately, and free.
## π€ What are you building with Shimmy?
**New developer tools and specifications included!** Whether you're forking Shimmy for your application or integrating it as a service, we now provide:
- **π§ Integration Templates**: Copy-paste guidance for embedding Shimmy in your projects
- **π Development Specifications**: GitHub Spec-Kit methodology for planning Shimmy-based features
- **π‘οΈ Architectural Guarantees**: Constitutional principles ensuring Shimmy stays reliable and lightweight
- **π Complete Documentation**: Everything you need to build on Shimmy's foundation
**Building something cool with Shimmy?** These tools help you do it systematically and reliably.
### π **GitHub Spec-Kit Integration**
Shimmy now includes [GitHub's brand-new Spec-Kit methodology](https://github.com/github/spec-kit) β specification-driven development that just launched in September 2025! Get professional-grade development workflows:
- **ποΈ Systematic Development**: `/specify` β `/plan` β `/tasks` β implement
- **π€ AI-Native Workflow**: Works with Claude Code, GitHub Copilot, and other AI assistants
- **π Professional Templates**: Complete specification and planning frameworks
- **π‘οΈ Constitutional Protection**: Built-in governance and architectural validation
[**π Complete Developer Guide β**](DEVELOPERS.md) β’ [**π οΈ Learn GitHub Spec-Kit β**](https://github.com/github/spec-kit)
### Try it in 30 seconds
```bash
# 1) Install + run
cargo install shimmy --features huggingface
shimmy serve &
# 2) See models and pick one
shimmy list
# 3) Smoke test the OpenAI API
curl -s http://127.0.0.1:11435/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model":"REPLACE_WITH_MODEL_FROM_list",
"messages":[{"role":"user","content":"Say hi in 5 words."}],
"max_tokens":32
}' | jq -r '.choices[0].message.content'
```
## π Works with Your Existing Tools
**No code changes needed** - just change the API endpoint:
- **VSCode Extensions**: Point to `http://localhost:11435`
- **Cursor Editor**: Built-in OpenAI compatibility
- **Continue.dev**: Drop-in model provider
- **Any OpenAI client**: Python, Node.js, curl, etc.
### Use with OpenAI SDKs
- Node.js (openai v4)
```ts
import OpenAI from "openai";
const openai = new OpenAI({
baseURL: "http://127.0.0.1:11435/v1",
apiKey: "sk-local", // placeholder, Shimmy ignores it
});
const resp = await openai.chat.completions.create({
model: "REPLACE_WITH_MODEL",
messages: [{ role: "user", content: "Say hi in 5 words." }],
max_tokens: 32,
});
console.log(resp.choices[0].message?.content);
```
- Python (openai>=1.0.0)
```python
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:11435/v1", api_key="sk-local")
resp = client.chat.completions.create(
model="REPLACE_WITH_MODEL",
messages=[{"role": "user", "content": "Say hi in 5 words."}],
max_tokens=32,
)
print(resp.choices[0].message.content)
```
## β‘ Zero Configuration Required
- **Auto-discovers models** from Hugging Face cache, Ollama, local dirs
- **Auto-allocates ports** to avoid conflicts
- **Auto-detects LoRA adapters** for specialized models
- **Just works** - no config files, no setup wizards
## π― Perfect for Local Development
- **Privacy**: Your code never leaves your machine
- **Cost**: No API keys, no per-token billing
- **Speed**: Local inference, sub-second responses
- **Reliability**: No rate limits, no downtime
## Quick Start (30 seconds)
### Installation
#### **πͺ Windows**
```bash
# RECOMMENDED: Use pre-built binary (no build dependencies required)
curl -L https://github.com/Michael-A-Kuykendall/shimmy/releases/latest/download/shimmy.exe -o shimmy.exe
# OR: Install from source (requires LLVM/Clang)
# First install build dependencies:
winget install LLVM.LLVM
# Then install shimmy:
cargo install shimmy --features huggingface
```
> **β οΈ Windows Notes**:
> - **Pre-built binary recommended** to avoid build dependency issues
> - If Windows Defender flags the binary, add an exclusion or use `cargo install`
> - For `cargo install`: Install [LLVM](https://releases.llvm.org/download.html) first to resolve `libclang.dll` errors
#### **π macOS / π§ Linux**
```bash
# Install from crates.io
cargo install shimmy --features huggingface
```
### GPU Acceleration
Shimmy supports multiple GPU backends for accelerated inference:
#### **π₯οΈ Available Backends**
| Backend | Hardware | Installation |
|---------|----------|--------------|
| **CUDA** | NVIDIA GPUs | `cargo install shimmy --features llama-cuda` |
| **Vulkan** | Cross-platform GPUs | `cargo install shimmy --features llama-vulkan` |
| **OpenCL** | AMD/Intel/Others | `cargo install shimmy --features llama-opencl` |
| **MLX** | Apple Silicon | `cargo install shimmy --features mlx` |
| **All GPUs** | Everything | `cargo install shimmy --features gpu` |
#### **π Check GPU Support**
```bash
# Show detected GPU backends
shimmy gpu-info
```
#### **β‘ Usage Notes**
- GPU backends are **automatically detected** at runtime
- Falls back to CPU if GPU is unavailable
- Multiple backends can be compiled in, best one selected automatically
- Use `--gpu-backend