llama.cpp
llama.cpp is a pure C/C++-based project that allows for LLM and VLM extraction with minimal dependence on a wide range of hardware. Apple Silicon offers optimizations for x86 and RISC-V architectures; it also supports integer quantization methods ranging from 1.5-bit to 8-bit. it is based on the ggml library and includes a large number of backend support such as CUDA, Metal, Vulcan.
llama.cpp
100ggml-org · Local AI / Self-Hosted LLM
Common commercial products in this category: ChatGPT, Claude, Gemini
- Download the model directly from Hugging Face and run it from the command line
- Using llama serve tool, you launch a OpenAI compatible API server with built-in web interface
- You run large models that exceed VRAM capacity with CPU and GPU hybrid inference support
- • High number of open issues (2,591)
- • NeutralHacker News
Llama.cpp 30B runs with only 6GB of RAM now
“It's just the same llama.cpp repo everyone else is using. You just git clone it to your android phone in termux and then run make and you're done. Assuming you have the model file downloaded (you can use wget to download it) these are the instructions to install and run: pkg…”
- • NeutralHacker News
Llama.cpp 30B runs with only 6GB of RAM now
“You can change the number of threads llama.cpp uses with the -t argument. By default it only uses 4. For example, if your CPU has 16 physical cores then you can run ./main -m model.bin -t 16 16 cores would be about 4x faster than the default 4 cores. Eventually you hit memory…”
Verified against README · Oct 26, 2026 · source
My note
Alternative pages
More in: Local AI / Self-Hosted LLM
- Ollama★ 185,398 · today
- Open WebUI★ 157,312 · today
- vLLM★ 95,942 · today
- Unsloth★ 78,833 · today
- GPT4All★ 77,346 · 1 yr ago
- AnythingLLM★ 67,771 · today