airllm
AirLLM is a library that allows large language models to run GPU with low VRAM capacity. Quantization allows for layer-based processing of models without resorting to distillation or pruning methods. It also supports training large models with limited memory resources.
airllm
88lyogavin · Local AI / Self-Hosted LLM
Common commercial products in this category: ChatGPT, Claude, Gemini
- You can run large language models with a low VRAM capacity GPU 70B parameter.
- You can train large-scale models on limited hardware resources.
- With the block-based compression method, you can increase the rate of inference by up to three times.
- ↓ NegativeGitHub Issue
[Partially solved] What's the problem with these AI toys? Not one that works at fist try
“jesus.... I tired a lot of these project for staying ahead and updated and they are a mass of problems and bugs.... Let's get to this one..... Trying to run on windows....... followed all instruction (also bitsandbytes install) First trouble: [ and the suggestion of doing…”
- ↓ NegativeGitHub Issue
More clever batching of layers
“Hello this is an awesome project, I replicated it on Modal Labs on a small T4 GPU. The problem I see now is that by loading one layer per time, you are not maximizing the GPU VRAM usage, for instance in this case it used only 1.6 GB of VRAM, I guess it is the size of one layer.…”
Verified against README · Oct 26, 2026 · source