Deep Dive - SGLang

An LLM request can be a single prompt, or it can be a program: retrieve evidence, ask for several candidates, select one, call a tool, then generate a structured answer. A server that treats every step as an unrelated text completion repeats prefixes and loses the structure the application already knows.

SGLang co-designs two layers for that workload. Its frontend lets Python code describe generation, choices, roles, and control flow. Its SGLang Runtime (SRT) turns those requests into continuously batched GPU work, with a radix-tree key-value (KV) cache, grammar-aware decoding, speculative execution, and distributed model execution. The frontend is optional: an OpenAI-compatible request can enter SRT directly. The runtime is the common path either way.

This chapter follows one request from a frontend expression or HTTP payload through tokenization, prefix matching, scheduling, model execution, and streamed output. Keep three signals in view: time to first token (TTFT), inter-token latency (ITL), and tokens per second. SGLang's optimizations change those signals by changing how much state is reused and how much work the scheduler can overlap.

Before reading code, name the two SGLang layers and one thing each owns.

Answer

The frontend owns a program-shaped request: text, roles, generation sites, choices, and control flow. SRT owns execution: tokenization handoff, KV-cache allocation, scheduling, model runners, and response streaming.

What SGLang is for

The original SGLang paper describes a frontend language plus a runtime for efficient execution of structured language-model programs.^{[1]Reference 1SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104} The current repository describes SGLang as a serving framework for language and multimodal models, with an OpenAI-compatible API, broad model support, and hardware backends.^{[2]Reference 2SGLang Source Repositoryhttps://github.com/sgl-project/sglang} Those descriptions point to a useful distinction:

The program surface can make repeated context visible. A system prompt, demonstrations, and retrieved documents become a shared prefix instead of three independently serialized strings. The runtime still needs an exact token match, so application-level similarity is not enough. A changed space, chat-template token, adapter identifier, or cache salt can split the prefix.

SGLang also serves requests that never use the frontend. This matters when comparing engines. A benchmark that sends plain OpenAI requests measures SRT's serving path, not the convenience or scheduling hints supplied by the frontend. A program benchmark adds branch parallelism, variable binding, and prefix reuse to the workload.

Frontend: a small intermediate representation for model calls

The public sgl module exposes a few operations with clear semantics. sgl.gen asks for model tokens, sgl.select chooses among explicit strings, and role helpers add chat-template boundaries. A decorated function becomes an SglFunction; the interpreter walks its expressions and sends generated spans to a backend. The source code for these pieces lives under python/sglang/lang/ in the local clone.

One generation site has a finite choice set. Another is open-ended but has a stop condition. Its result contains named values such as label and explanation, so application code can use one generation to shape the next prompt. A frontend program can also run several branches in parallel or bind a cached prefix for a batch.

The frontend doesn't make the model deterministic. Sampling parameters still apply, and a select choice can be scored with the configured method. It also doesn't guarantee that a JSON-looking prompt returns valid JSON. Use json_schema or a regex constraint when output syntax is part of the contract.

SGLang's interpreter keeps a stream executor and a program state. The stream executor submits text and generation operations to the backend; the state holds named variables and assembled text. This split lets one program use a local engine, a remote runtime endpoint, or a test backend without changing its control flow. It also means that a bug can live above SRT: an unexpected branch, a missing variable, or a chat-template mismatch may never reach the scheduler.

Why can a frontend program be faster than concatenating its final prompt and calling an endpoint once?

Answer

The runtime sees generation boundaries, repeated prefixes, and independent branches before serialization. It can cache a full prefix, schedule branches together, and avoid recomputing context that those branches share. A flattened prompt hides those relationships.

Runtime path: from API to model runner

For an online request, the path is roughly:

The process split can vary with server mode and parallelism, but responsibilities stay recognizable. Entrypoints parse requests and return protocol responses. The tokenizer manager owns tokenization and detokenization boundaries. The scheduler chooses which request tokens run next. A radix cache supplies reusable KV locations. A model runner loads weights, builds a forward batch, selects attention backends, and returns logits or sampled tokens. Grammar and sampling code constrain or select the next token before the stream is updated.^{[3]Reference 3SGLang Documentationhttps://docs.sglang.io/}

Debugging starts at the handoff between stages. High TTFT with low GPU utilization points to tokenization, queue wait, prefix lookup, or admission before kernel work. Rising ITL after a model update points to forward-batch shapes, attention backend selection, collective wait, and grammar or speculative work. Malformed output calls for separating model logits from grammar masking and detokenization.

One step, one token budget

SRT maintains active requests and schedules a batch repeatedly. Decodes usually need one new token per request. Prefills need many prompt tokens. Chunked prefill divides a long prompt so it doesn't monopolize a step while decodes wait. The exact flags and defaults change by release, so inspect Scheduler.init_chunked_prefill and the server arguments for the clone's version.

Consider a teaching trace with budget six:

The numbers are an invariant exercise, not a performance claim. The scheduler spends a bounded token budget while keeping decode work eligible. A larger prefill slice may improve TTFT for new prompts but can delay existing decodes. A smaller slice can protect ITL but stretch prompt processing.

SGLang also has an overlap path. CPU scheduling, token movement, and GPU execution can be pipelined when the backend and flags permit it. Overlap hides some host work, but it adds queues and synchronization. A race in request state, a delayed grammar update, or a stale KV location can become a correctness bug rather than a simple latency regression.

What does chunked prefill protect, and what can it cost?

Answer

It protects decode ITL by bounding prompt work in one step. It can increase a new request's TTFT because its prompt is spread across multiple steps, and an oversized chunk can still delay decodes.

RadixAttention: prefix reuse as a tree

RadixAttention is SGLang's name for using a radix tree to find reusable KV-cache prefixes. It isn't an attention kernel. The tree stores token-key paths and references to KV locations; an attention backend later reads those locations. The distinction matters when reading performance reports or debugging a hardware-specific kernel.

Suppose two requests share tokens A B C D and then diverge:

The tree contains one shared path for the first four tokens, then two branches. A request's prefix match returns the longest cached path and the corresponding token-to-KV indices. The scheduler can skip prefill for that matched region and allocate new KV slots only for the suffix.

The local RadixKey stores token IDs plus an optional extra key. Current source uses extra keys for namespaces such as LoRA adapters, cache salt, version, or retrieval context, and the cache's match operation checks compatible keys before comparing tokens. This keeps cache namespaces disjoint as cache partitioning, not an authorization boundary.

The match is exact and page-aware. If a cache policy or backend requires aligned units, a partial final unit isn't reusable as a complete prefix. The practical result is simple: stable system prompts and demonstrations help; per-request timestamps and random IDs near the front of a prompt destroy hits.

Insert, lock, evict

A prefix cache needs ownership rules alongside its lookup map. While a request is active, its matched KV locations are protected. When a request finishes or releases a branch, nodes can become evictable. The eviction policy can be LRU or another configured strategy, but it must never reclaim state still referenced by a live request.

The source path is visible in python/sglang/srt/mem_cache/radix_cache.py:

1. match_prefix finds the longest compatible path and returns KV indices.
2. insert adds a new token path after forward computation.
3. Lock and unlock operations protect prefixes while requests use them.
4. evict removes unprotected branches according to the selected policy.

This sequence exposes a common failure. If a request's KV reference is released too early, another request can overwrite memory while the first is still decoding. If references never drop, free KV capacity shrinks until admission stalls. Instrument active references, evictable tokens, match lengths, and allocator failures together.

The cache is local to an engine or replica unless a higher-level routing layer coordinates it. A load balancer that sends identical prompts to different replicas may see low hit rates even when each replica has a healthy tree. Cache-aware routing can improve locality, but adding replicas still changes hit probability and warm-up behavior.

Why doesn't an embedding similarity qualify as a RadixAttention hit?

Answer

The cache reuses exact token-key paths and compatible extra keys. Similar meaning doesn't imply identical hidden states, so semantic similarity needs retrieval or application logic, not KV reuse.

Structured generation: constrain the next token

Many applications need valid JSON, a tool-call schema, a regular-language field, or one label from a known set. Asking a model to "please emit JSON" is a prompt preference, not a syntax guarantee. SGLang can carry a regex, JSON schema, or choice constraint through sgl.gen and its server APIs.^{[4]Reference 4Structured Outputshttps://docs.sglang.io/advanced_features/structured_outputs.html}

At each decoding step, a grammar engine tracks an automaton state. It computes the set of tokens that keep the partial output valid, and the sampler masks other logits. The model still chooses among allowed tokens. The grammar doesn't add facts to the model or repair a semantically wrong field.

SGLang supports multiple grammar backends in the repository. The selected backend, tokenizer, vocabulary, and schema shape affect CPU work and compatibility. A grammar with many alternatives can take more time to update than a small enum. Measure grammar preparation, per-step mask time, and end-to-end latency rather than assuming all constrained requests cost the same.

The original SGLang work also introduced compressed finite-state-machine techniques for faster structured decoding.^{[1]Reference 1SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104} The enduring design is the boundary: grammar state stays on the control side, while logits remain on the model side. That makes constraints inspectable and lets the runtime reuse model KV state even when grammar states differ.

Failure handling needs an explicit policy. If no token satisfies the grammar, the request should return an error or a clearly marked incomplete result. Retrying with a looser grammar can change semantics. Don't silently strip braces or run a best-effort JSON parser and call the result valid.

Speculative decoding: verify a draft in parallel

Autoregressive decoding normally produces one token per target-model forward step. Speculative decoding asks a cheaper draft path for several candidates, then lets the target model verify them in one pass. Accepted tokens advance the sequence faster; rejected tokens fall back to a target-sampled token.

SRT exposes several speculative algorithms, including EAGLE, EAGLE3, n-gram lookup, standalone draft models, and DFlash-family methods in the cloned version. The enum and worker registry live under python/sglang/srt/speculative/. The exact flags and model requirements are version-specific, so check SpeculativeAlgorithm and its validator before enabling a mode.^{[5]Reference 5Speculative Decodinghttps://docs.sglang.io/advanced_features/speculative_decoding.html}

An abstract step looks like this:

EAGLE-2 and EAGLE-3 research explores draft generation from target-model representations and tree-shaped candidates.^{[6]Reference 6EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treeshttps://arxiv.org/abs/2406.16858[7]Reference 7EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testhttps://arxiv.org/abs/2503.01840} DFlash-family work targets faster diffusion-style or parallel draft paths in newer SGLang releases.^{[8]Reference 8DFlash: Block Diffusion for Flash Speculative Decodinghttps://arxiv.org/abs/2602.06036} The implementation must still preserve token ordering, position IDs, grammar state, and cache ownership.

Speculation helps when acceptance is high and draft work is cheaper than target work. It can hurt when the model is uncertain, prompts are short, grammar masks reject candidates, or draft KV consumes capacity needed by ordinary requests. Track acceptance length, draft and verify latency, rejected tokens, extra memory, and ITL by model and workload.

What evidence would convince you speculative decoding is hurting a service?

Answer

Low accepted-token length together with draft overhead, rising memory pressure, or unchanged target-model time. Compare against the same traffic with speculation disabled and split draft, verify, and commit timings.

Parallelism and prefill-decode disaggregation

SGLang maps model work onto hardware through several parallelism axes. Names vary by model, but the trade-offs are stable:

The runtime has to align these choices with attention backends, quantization, CUDA graphs, and model-specific layers. A TP setting that fits weights can still reduce ITL if collectives dominate. A DP replica can raise throughput while lowering prefix-cache locality. Read the parallelism docs and benchmark on the target topology before copying a flag set.^{[9]Reference 9SGLang Server Arguments and Parallelismhttps://docs.sglang.io/advanced_features/server_arguments.html}

Prefill and decode have different resource shapes. Prefill consumes many prompt tokens and tends to be compute-heavy. Decode consumes one or a few tokens per request and repeatedly reads KV memory, making it more latency and memory sensitive. Prefill-decode (P/D) disaggregation places them on separate worker pools and transfers KV state between them.^{[10]Reference 10Prefill-Decode Disaggregationhttps://docs.sglang.io/advanced_features/pd_disaggregation.html}

The P/D path can protect decode latency from a long prompt and let each pool scale for its own traffic. It adds transfer bytes, synchronization, routing, and a cache locality decision. A prompt that is prefetched on one node but decoded on another needs a reliable connector and enough bandwidth. If transfer time exceeds the compute saved, colocated execution wins.

P/D disaggregation is not a free "scale out" switch. Test queueing at both pools, transfer latency, failure recovery, and cache hit rate. A decode worker that loses its prefill connection needs a clear retry or cancellation path so stale KV state isn't treated as valid.

Applications and workload shape

SGLang's main advantage appears when request structure matches runtime structure. Choose the smallest example that exposes that match:

The paper's evaluations span agent control, logical reasoning, few-shot tasks, JSON decoding, retrieval-augmented generation, and multi-turn chat.^{[1]Reference 1SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104} Treat those workloads as evidence for mechanisms, not a current throughput guarantee. A release, GPU, model, tokenizer, and request mix can change the winner.

Strengths, weaknesses, and failure boundaries

Most incidents fit one of four boundaries:

1. Prompt boundary: tokenization or chat-template output differs from what the application expects.
2. Cache boundary: prefix keys, references, or allocator capacity don't match request lifetime.
3. Execution boundary: a scheduler batch, attention backend, or collective stalls or returns wrong shapes.
4. Output boundary: grammar, sampler, detokenizer, or stream assembly changes the result.

Separate those boundaries in telemetry. A single end-to-end latency number can't tell whether a prefix hit helped or whether a grammar mask consumed the saved time.

A production failure drill

Suppose a service reports low GPU utilization, high TTFT, and almost no radix hits.

1. Log tokenized prompt lengths and chat-template versions. A frontend and OpenAI caller may be producing different prefix bytes.
2. Check router placement. Identical requests on different replicas won't share a local tree unless routing is cache-aware.
3. Inspect scheduler queue wait, tokenizer time, and max_prefill_tokens or chunked-prefill settings.
4. Check KV allocator free capacity and evictable tokens. A full cache can trigger admission stalls even when compute is idle.
5. Compare grammar and speculation flags. CPU grammar work or rejected drafts can make GPU gaps look like a cache issue.
6. Run a fixed prompt replay with speculation and grammar disabled. This gives a clean baseline for model-runner and backend timing.

Now invert the symptoms: high GPU utilization, good radix hits, but ITL spikes when a long prompt arrives. That points toward prefill scheduling, chunk size, collective wait, or P/D routing. Increasing cache size won't fix a token-budget policy that lets prefill consume every step.

Which observation separates "cache is cold" from "cache is full"?

Answer

A low match length shows reuse is absent, while allocator free capacity, evictable tokens, and admission failures show whether memory is full. You need both sets of signals before changing cache policy.

Project identity

The SGLang paper came from researchers and engineers around UC Berkeley and the LMSYS community, including Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Ying Sheng, Ion Stoica, Joseph Gonzalez, and collaborators.^{[1]Reference 1SGLang: Efficient Execution of Structured Language Model Programshttps://arxiv.org/abs/2312.07104} LMSYS is a nonprofit organization that incubates open-source systems and research projects; SGLang is one of its flagship projects.^{[11]Reference 11About LMSYS Orghttps://www.lmsys.org/about/} The current repository is a multi-contributor project rather than a single-vendor SDK, so support and performance depend on model, hardware, and release paths.

Read the history as a sequence of co-design decisions:

The code keeps these ideas modular enough to inspect. A paper explains why a mechanism helps. A release guide explains which flags and models expose it. The local source shows ownership, state transitions, and failure handling. Read all three before making a production claim.

Code-reading route in the local clone

This walkthrough uses the official SGLang repository at commit f8e62a9224815cc9c6fc56b940eb7fde791a8870.^{[2]Reference 2SGLang Source Repositoryhttps://github.com/sgl-project/sglang}

Read one request through these files:

1. python/sglang/lang/api.py: public function, gen, select, role, regex, and JSON-schema operations.
2. python/sglang/lang/ir.py: expression nodes and the program representation passed to a backend.
3. python/sglang/lang/interpreter.py: program state, stream executor, batch execution, and optional prefix tracing.
4. python/sglang/srt/entrypoints/http_server.py: online request boundary and server routes.
5. python/sglang/srt/managers/tokenizer_manager.py: tokenization, request state, and output handoff.
6. python/sglang/srt/managers/scheduler.py: token pools, chunked prefill, overlap setup, admission, and scheduling loop.
7. python/sglang/srt/mem_cache/radix_cache.py: match_prefix, insert, reference protection, and eviction.
8. python/sglang/srt/model_executor/model_runner.py: model loading, forward-batch setup, attention backends, and parallel state.
9. python/sglang/srt/constrained/grammar_manager.py: grammar backend selection and constrained token state.
10. python/sglang/srt/speculative/spec_info.py: speculative algorithm registry and draft/verify capability checks.
11. python/sglang/srt/disaggregation/prefill.py and decode.py: P/D role boundaries and KV transfer lifecycle.

Keep these invariants beside the source:

• A frontend generation name identifies a value in program state; it isn't a cache key by itself.
• A radix match is exact over token IDs and compatible extra keys; semantic similarity doesn't qualify.
• A scheduler may execute only tokens backed by valid KV locations and request ownership.
• A grammar filter can remove logits but can't make an invalid model state valid.
• Speculative verification can commit only accepted tokens in order and must update grammar and KV state together.
• P/D transfer must complete before decode consumes the transferred KV range.

Rerun and measure

Start with a tiny local engine or mock backend. Verify the frontend program returns named values. Then send the same tokenized prompt twice and compare matched-prefix length and prefill work. Add a third request with one changed system-prompt token to prove that reuse is exact.

For serving tests, record:

Change one mechanism at a time. Compare four fixed configurations: plain decoding, radix cache on, grammar on, and speculation on. For P/D, compare colocated and disaggregated modes with the same prompt mix. Keep quality and correctness checks beside latency, because a faster malformed JSON response is a regression.

Key takeaways

• SGLang combines a program frontend with SRT, but plain API requests can use SRT without the frontend.
• Frontend expressions expose generation sites, branches, and constraints that a flattened prompt hides.
• RadixAttention is a prefix-cache data structure and scheduling aid, not an attention kernel.
• A token budget, chunked prefill, and overlap let decodes and prefills share repeated steps, with TTFT and ITL trade-offs.
• Grammar constraints mask invalid next tokens; they don't guarantee semantic correctness.
• Speculative decoding trades draft work and memory for accepted target tokens. Acceptance length is the key health signal.
• TP, PP, DP, EP, and P/D disaggregation solve different fit and latency problems, each with communication costs.
• Production debugging needs token, cache, scheduler, runner, grammar, and stream signals, not one throughput number.

Explain the runtime path without looking at the diagram. Then answer why a stable system prefix helps two requests, why a grammar can increase CPU time, and why a P/D split can lose to colocated execution on a bandwidth-limited network.