shimmy
Shimmy has a single binary file developed for GGUF models and is a OpenAI API compatible inference server. It is written entirely in Rust and does not need a Python or C++ tool chain. WebGPU (WGSL) runs on different graphics processors such as NVIDIA, AMD, Intel, and Apple Silicon using computational shaders.
Michael-A-Kuykendall · Local AI / Self-Hosted LLM
Common commercial products in this category: ChatGPT, Claude, Gemini
- You redirect existing AI tools to this server in a local and secret way.
- You run models in the GGUF format without a Python or C++ dependency.
- You increase context length with SHIMMY_MAX_CTX parameter and YaRN RoPE scaling.
- ↑ PositiveHacker News
Show HN: Shimmy – 5MB privacy-first, local alternative to Ollama (680MB)
“Hey HN! I built this because I was tired of waiting 10 seconds for Ollama's 680MB binary to start just to run a 4GB model locally. Quick demo - working VSCode + local AI in 30 seconds: curl -L ./shimmy serve # Point VSCode/Cursor to localhost:11435 The technical achievement: Got…”
- • NeutralHacker News
Show HN: Shimmy – 5MB privacy-first, local alternative to Ollama (680MB)
“Shimmy is designed to be "invisible infrastructure" - the simplest possible way to get local inference working with your existing AI tools. llama-server gives you more control, llama-swap gives you multi-model management. Key differences: - Architecture: llama-swap = proxy +…”