SGLang
SGLang is an open-source inference framework developed for large language, vision-language and diffusion models, optimized for agent workloads, large-scale service, and RL rollout. RadixAttention targets low latency and high yield with prefix cache and multiple GPU parallels; Hugging Face and OpenAI API compatible. NVIDIA supports a wide range of hardware such as AMD, Google TPU (PyTorch and JAX roads), Intel, Apple Silicon, and Huawei Ascend.
SGLang
90sgl-project · AI / LLM / RAG
Common commercial products in this category: ChatGPT, Claude, Gemini
- You serve large language models via OpenAI compatible API with high efficiency.
- With RadixAttention prefix cache, you gain speed in multi-round chat and agent workloads.
- With SGLang Diffusion, you accelerate video and image creation processes.
- • High number of open issues (6,299)
- ↓ NegativeHacker News
Implement Flash Attention Back End in SGLang – Basics and KV Cache
“Here is a list of inference engines i've tried: - SGLang - vLLM - TGI (Huggingface's) - llama.cpp - infinity (great for embedding / reranking models not for LLMs) My personal feeling is SGLang / vLLM have issues that make me not want to use it. Sure it's fast, but there are…”
- ↓ NegativeGitHub Issue
Instruction for Running DeepSeek with Large-scale PD and EP
“## Using main branch ~~NOTE: The feature is already on main, but the performance still needs some improvements on main branch.~~ will be good after a few already opened PRs - PR 6680, 6727, 6728 ~~NOTE: I will try other config like 4 node for P and 9 node for D later.~~ updated…”
Verified against README · Oct 26, 2026 · source
My note
Alternative pages
More in: AI / LLM / RAG
- rea★ 412,812 · today
- ECC★ 293,487 · yesterday
- hermes-agent★ 258,923 · today
- ponytail★ 181,048 · 2 days ago
- graphify★ 139,557 · today
- MoneyPrinterTurbo★ 136,645 · today