LightLLM
LightLLM is a language model (LLM) inference and presentation infrastructure known for its lightweight design, easy scalability and high-speed performance based on Python. FasterTransformer combines the strengths of open-source applications such as TGI, vLLM and FlashAttention. Data offers Prefix KV Cache transfer support between parallel sequencers and provides token-level KV Cache management.
ModelTC · Local AI / Self-Hosted LLM
Common commercial products in this category: ChatGPT, Claude, Gemini
- You build rapid inference and presentation infrastructure for large language models.
- You transfer Prefix KV Cache between data parallel sequencers.
- At the Token level, you develop research projects with KV Cache management.
- • NeutralGitHub Issue
Llama2 and llama-30b does not work
“- Only able to run llama-7b model , llama2-13b , llama2-70b and llama-30 Failing - Using A100 GPU , followed the instruction about `triton` dependencies. ```python Task exception was never retrieved future: exception='cef3986efedb4d1a966c56365091ddde' ========= Remote Traceback…”