nano-vllm
nano-vllm is a lightweight implementation of vLLM written from scratch with Python code of about 1,200 lines. It offers offline inference speeds comparable to vLLM with optimizations such as prefix caching, tensor parallelism, Torch compilation and CUDA graphics.
NA
GeeeekExplorer · Local AI / Self-Hosted LLM
STARS16.2K
FORKS2.8K
COMMITS67
CONTRIBUTORS12
VERSION—
UPDATED6 mo ago
PythonvLLMInferenceCUDA GraphMIT
Common commercial products in this category: ChatGPT, Claude, Gemini
What can it do?
- You download model weights on Hugging Face and run offline inference.
- You produce text using a vLLM-like API interface.
- You test optimization features such as prefix cache and tensor parallelism.
Heads up
- • No published release