PowerInfer
PowerInfer is an inference engine that uses activation locality to run a high-speed large language model (LLM) on the consumer class GPU and CPU. Developed with C++, this tool includes frames optimized for smartphones and local devices, and includes AMD hardware with ROCm support. The project works in harmony with sparse (sparse) models, enabling efficient token production in local distributions.
Tiiny-AI · Local AI / Self-Hosted LLM
Common commercial products in this category: ChatGPT, Claude, Gemini
- You can run major language models locally on consumer-grade GPU and CPU hardware.
- You can take advantage of the optimized inference framework for smartphones and mobile devices.
- On AMD devices, you can perform model inference operations with ROCm support.
- • No published release
- ↑ PositiveHacker News
PowerInfer: Fast Large Language Model Serving with a Consumer-Grade GPU [pdf]
“From their GitHub[0]: > Evaluation shows that PowerInfer attains an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs (including OPT-175B) on a single NVIDIA RTX 4090 GPU, only 18% lower than that achieved by a top-tier…”
- • NeutralHacker News
PowerInfer: Fast Large Language Model Serving with a Consumer-Grade GPU [pdf]
“This is basically fork of llama.cpp. I created a PR to see the diff and added my comments on it: One thing that caught my interest is this line from their readme: > PowerInfer exploits such an insight to design a GPU-CPU hybrid inference engine: hot-activated neurons are…”