lmdeploy

LMDeploy is a toolkit developed for compression, distribution and presentation of large language models. It supports different model architectures and hardware platforms through the TurboMind engine and PyTorch infrastructure. It provides quantization and inference optimizations at various sensitivity levels.