Back: Catalog

vllm-rlt

A vLLM-inspired inference engine for looped transformers, with loop-level scheduling and execution across composable stages.

VL

vllm-rlt

New70

InferMatrixLab · AI / LLM / RAG

STARS213
FORKS34
COMMITS0
CONTRIBUTORS0
VERSIONv0.1.0
UPDATEDyesterday
inference-enginellmtransformerApache-2.0

Common commercial products in this category: ChatGPT, Claude, Gemini

What can it do?
  • Three native models: Ouro-1.4B with fixed-depth or learned early-exit
  • Measured serving performance: on the B300 ShareGPT C8 workload below,
  • Loop-aware runtime: continuous batching, depth-aware paged KV, and opt-in
User reviews
  • • NeutralGitHub Issue

    [RFC] Module boundaries and incremental refactoring of the RLT inference runtime

    “## Summary This RFC proposes a module-by-module refactoring of vllm-lt, grounded in detailed code walkthroughs. The goal is to establish clear responsibilities, state ownership, and interfaces while preserving recurrent looped transformer (RLT) semantics: loop-level batching,…”

  • ↑ PositiveGitHub Issue

    [RFC] Speculative Decoding for rlt

    “## 1. Motivation Native RLT decoding executes all recurrent loops before producing each token. At low request concurrency, the small token batch can leave GPU capacity underused. This proposal uses shallow loops to draft tokens, reuses their hidden states and KV, and batches the…”

GitHub

My note

Alternative pages

ChatGPT alternativesClaude alternativesGemini alternatives

More in: AI / LLM / RAG