Personalize this lesson
Adapt explanations and teaching visuals to your background and preferred voice.
Suppose a serving alert says GPU utilization is 35%, time to first token has doubled, and two classify_and_explain requests that used to share work now prefill separately. One request picked up a new chat-template token; another landed on a different replica. The model weights didn't change. Either change can lose a prefix hit, but neither alone explains low utilization. You need to locate both the lost reuse and the time spent waiting.
The previous chapter's Hugging Face Transformers lesson established a model-definition boundary: a revision, config, weights, and tokenizer travel together. SGLang consumes that contract, then spends its effort on request programs, key-value (KV) reuse, and GPU scheduling.
Trace the requests through five handoffs: frontend or HTTP payload, tokenization, prefix lookup and scheduling, model execution, then constrained sampling and streaming. Each handoff owns different state, so one latency symptom can have several causes.
SGLang pairs a Python frontend for program-shaped calls with SGLang Runtime (SRT), which batches GPU work, reuses exact KV prefixes, and can constrain or speculate during decoding. The frontend is optional; an OpenAI-compatible request can enter SRT directly.[1][2]
Keep two classify_and_explain requests in view as we follow the path. Measure time to first token (TTFT), the wait until first output; inter-token latency (ITL), the interval between output tokens; and tokens per second, an aggregate throughput signal. Together they show whether a change saved input work, protected decode, or shifted cost to a host-side stage.
Before reading code, name the two SGLang layers and one thing each owns.
Answer
The frontend owns a program-shaped request: text, roles, generation sites, choices, and control flow. SRT owns execution: tokenization handoff, KV-cache allocation, scheduling, model runners, and response streaming.
Frontend program or HTTP request
Start at the caller boundary. A program and a JSON request can carry the same visible question, but they expose different opportunities to the runtime. The original SGLang paper describes a frontend language plus a runtime for structured language-model programs.[1]
SGLang serves language and multimodal models through several hardware backends. Support depends on the model, attention backend, device, and release; a supported device doesn't imply that every grammar or speculative mode works there.[2][3]
The entry path determines whether the application exposes a multi-call program or sends individual generation requests:
| Entry path | Reader writes | SGLang must optimize | Good fit |
|---|---|---|---|
| Frontend program | @sgl.function, sgl.gen, sgl.select, fork / join, roles | Shared prefixes, parallel program branches, output dependencies | Agents, few-shot programs, extraction, multi-turn workflows |
| OpenAI-compatible API | JSON chat or completion request | Batching, KV reuse, sampling, streaming | Drop-in model endpoint |
| Offline engine | Python calls against sgl.Engine | Batch admission and reusable cache state | Evaluation, replay, test fixtures |
| RL rollout client | Requests from a trainer or environment | Throughput, weight refresh, deterministic boundaries | Post-training and verifiable rewards |
The frontend can organize a system prompt, demonstrations, and retrieved documents into reusable prompt state. The runtime still compares exact token IDs, not application-level similarity. A space matters if it changes tokenization; a changed template token can shorten the hit. An adapter identifier or cache salt can partition the cache even when tokens match.
Now hold that program aside and use the API row. SGLang also serves requests that never use the frontend. A benchmark that sends plain OpenAI requests measures SRT's serving path, not convenience or scheduling hints supplied by the frontend.
A program benchmark adds branch parallelism, variable binding, and prefix reuse to the workload. Compare those results only after naming which path you measured; they answer different questions.
The paper's evaluations on agent control, logical reasoning, few-shot tasks, JSON decoding, retrieval-augmented generation, and multi-turn chat reported up to 6.4× higher throughput than the compared systems on those 2024 workloads.[1] Treat that as evidence for the mechanisms, not a current throughput guarantee. A release, GPU, model, tokenizer, and request mix can change the winner.
Frontend: a small intermediate representation for model calls
Read the frontend as an intermediate representation for model calls. sgl.gen asks for model tokens, sgl.select chooses among explicit strings, and role helpers add chat-template boundaries. Before looking at code, predict which operation exposes a finite choice and which one can continue until a stop condition.
The paper also names fork and join for parallel copies of prompt state, so a program can run several candidates without flattening them into one prompt.[1] A decorated function becomes an SglFunction; the interpreter walks its expressions and sends generated spans to a backend.
The source for these pieces lives under python/sglang/lang/. The code-reading route below pins one commit; the web documentation describes a moving release surface.
The snippet needs sglang and a configured backend. It defines a frontend program, not a server launch. The interpreter submits generation operations as the Python function runs; the GPU scheduler doesn't receive the whole Python control-flow graph. This integration example hasn't been executed against a model.
1import sglang as sgl
2
3@sgl.function
4def classify_and_explain(s, question):
5 s += sgl.system("Return concise, evidence-backed answers.")
6 with s.user():
7 s += question
8 with s.assistant():
9 s += "Label: "
10 s += sgl.select("label", ["bug", "feature", "question"])
11 s += "\nExplanation: "
12 s += sgl.gen("explanation", max_tokens=64, stop="\n")Call this twice. R0 classifies "Login button throws 500" and is already decoding the explanation. R1 classifies "Add dark mode to settings" and is still prefilling a longer ticket body.
Both calls share the system prompt and the user-role opening. Their ticket tokens then diverge. The later Label: and Explanation: strings are repeated text, but they aren't reusable prefixes across these two tickets: their KV depends on the different preceding context. Within one request, the explanation can reuse the context already computed for its label.
One generation site has a finite choice set. Another is open-ended but has a stop condition. The result contains named values such as label and explanation, so application code can use one generation to shape the next prompt.
The frontend doesn't guarantee deterministic sampling. select scores a finite choice set using its configured method, while gen samples under its generation parameters. A JSON-looking prompt still isn't a JSON contract. Use an explicit constraint through an API that supports it.
SGLang's interpreter keeps a stream executor and a program state. The executor submits text and generation operations to the backend; the state holds named variables and assembled text. This split lets one program use a local engine, a remote runtime endpoint, or a test backend without changing its control flow.
That boundary also localizes bugs. An unexpected branch, missing variable, or chat-template mismatch can fail above SRT before the scheduler sees any model work.
Does using the frontend automatically create more prefix hits than sending the same tokenized requests through HTTP?
Answer
No. SRT can reuse exact prefixes from either entry path. The frontend helps express generation dependencies, named values, and independent branches. It can expose opportunities to reuse state, but identical tokens and placement can produce the same cache hits through HTTP. A single flattened prompt isn't an equivalent replacement for a program with dependent model calls.
Runtime path: from API to model runner
A frontend expression and a plain HTTP payload look different at the edge. Inside SRT they share owners. Entrypoints parse requests; the tokenizer manager prepares model inputs and tracks request/output state. A separate detokenizer manager converts generated IDs to text in the usual multiprocess serving path.
The scheduler plus radix cache decide which tokens run next and which KV locations already exist. The model runner loads weights, builds a forward batch, selects attention backends, and returns logits or sampled tokens.[2]
A delay before the first model batch suggests input, cache, or admission work. A delay between generated tokens can include scheduling, the runner, communication, grammar, speculation, and output delivery. TTFT includes the first forward pass and sampling too; these metrics don't divide neatly into CPU and GPU stages.
The process split can vary with server mode and parallelism, but those responsibilities stay recognizable. Grammar and sampling code constrain or select the next token before the stream is updated. Follow the data, not a fixed process layout.
Debugging starts at a handoff. High TTFT with low GPU utilization points to tokenization, queue wait, prefix lookup, or admission before kernel work. Rising ITL after a model update points to forward-batch shapes, attention backend selection, collective wait, and grammar or speculative work. Malformed output calls for separating model logits from grammar masking and detokenization.
One step, one token budget
SRT maintains active requests and schedules a batch repeatedly. Decodes usually need one new token per request; prefills need many prompt tokens. Chunked prefill divides a long prompt so it doesn't monopolize a step while decodes wait. Exact flags and defaults change by release, so inspect Scheduler.init_chunked_prefill and the server arguments for the pinned commit.
Use a simplified mixed-batch policy for the two classify requests. Shared prefix length is 4. R0 is decoding (1 token); R1 has 9 unmatched prompt tokens. With budget 6 and one decode token reserved, five R1 prompt tokens fit:
| Step | Ready work | Tokens admitted | State after step |
|---|---|---|---|
| 0 | R0 decode needs 1; R1 prefill needs 9 | R0=1, R1=5 | R0 advances; R1 has 4 prompt tokens left |
| 1 | R0 decode needs 1; R1 prefill needs 4 | R0=1, R1=4 | R1 finishes prefill |
| 2 | R0 and R1 decode | R0=1, R1=1 | Both requests continue generation |
This is an accounting exercise, not SGLang's default scheduling trace. At the pinned commit, enable_mixed_chunk defaults to false; when enabled, PrefillAdder subtracts mixed decode tokens from its input and chunk budgets. Admission also checks KV capacity, page alignment, and reservations for future output. Chunked prefill alone doesn't promise that both modes run in the same batch.[4]
At step 1, the final prefill can already sample R1's first output token. Step 2 feeds that output token back through the model. A larger prefill slice may improve TTFT for a new prompt but delay existing decodes; a smaller slice can protect ITL while stretching prompt processing.
SGLang also has an overlap path. CPU scheduling, token movement, and GPU execution can be pipelined when the backend and flags permit it. Overlap hides some host work, but it adds queues and synchronization.
Use the ownership question to test an overlap change: if a request state, grammar update, or KV location arrives late, which consumer reads it first? A race can become a correctness bug, not only a latency regression.
What does chunked prefill protect, and what can it cost?
Answer
It protects decode ITL by bounding prompt work in one step. It can increase a new request's TTFT because its prompt is spread across multiple steps, and an oversized chunk can still delay decodes.
RadixAttention: prefix reuse as a tree
RadixAttention names SGLang's use of a radix tree, a compressed prefix tree whose edges can hold several tokens, to find reusable KV-cache prefixes. Start with the ownership boundary: the tree stores token-key paths and references to KV locations; an attention backend later reads those locations.
RadixAttention is not an attention kernel. A cache hit and a faster hardware kernel are separate claims, so keep them separate in a performance report.
The two classify requests share tokens A B C D and then diverge. Predict the lookup before reading the table: which path is reusable, and how many new tokens compete for this step's budget?
| Request | Token sequence | Reused path | New work this step |
|---|---|---|---|
R0 | A B C D plus generated suffix | A B C D | 1 decode token |
R1 | A B C D plus 9 unmatched prompt tokens | A B C D | 9 prefill tokens, 5 admitted |
The tree contains one shared path for the first four tokens. A request's prefix match returns the longest cached path and the corresponding token-to-KV indices. The scheduler can skip prefill for that matched region and allocate new KV slots only for the suffix.
In the pinned source, RadixKey stores token IDs plus an optional extra_key. Request construction combines the supplied extra key with a LoRA identifier when present. The tree keeps different namespaces disjoint; direct RadixKey.match calls reject incompatible keys. Cache salts can separate tenants' reuse, but authentication and authorization still belong at the request boundary.[4]
Model-equivalent context and request-local output state are different contracts. A tool schema rendered into the prompt changes tokens and can change KV. A grammar applied only to output logits needn't change an otherwise identical prompt's KV; each request still needs its own grammar state. Model or adapter updates must invalidate or partition stale KV even if the prompt text stays identical.
The match is exact and page-aware. If a cache policy or backend requires aligned units, a partial final unit isn't reusable as a complete prefix. Stable system prompts and demonstrations therefore help, while per-request timestamps and random IDs near the front of a prompt destroy hits.
You can check the same rules in a few lines of Python. Before running it, predict two results: matching extra keys should return four shared tokens, while a different key should return zero.
The helper below models tree lookup, not the internal RadixKey.match method: it returns zero for a different namespace rather than raising. It then truncates the match to a page multiple. These letters stand for token IDs, and the example assumes ordinary full attention, not EAGLE bigram keys or hybrid-state caches.
1def match_len(cached, request, extra_a, extra_b, page_size=1):
2 if not isinstance(page_size, int) or page_size < 1:
3 raise ValueError("page_size must be a positive integer")
4 if extra_a != extra_b:
5 return 0
6 n = 0
7 for a, b in zip(cached, request):
8 if a != b:
9 break
10 n += 1
11 return (n // page_size) * page_size
12
13shared = ["A", "B", "C", "D"]
14r1 = shared + [f"P{i}" for i in range(1, 10)]
15hit = match_len(shared, r1, extra_a="base", extra_b="base")
16prefill_left = len(r1) - hit
17budget = 6
18r0_decode = 1
19r1_now = min(prefill_left, budget - r0_decode)
20
21assert hit == 4
22assert prefill_left == 9
23assert r1_now == 5
24assert match_len(shared, r1, extra_a="base", extra_b="lora-7") == 0
25assert match_len(shared + ["P1"], r1, extra_a="base", extra_b="base", page_size=4) == 4
26assert match_len(["A", "B", "X", "D"], shared, "base", "base") == 2
27assert match_len(["X", "B", "C", "D"], shared, "base", "base") == 0
28print(f"hit={hit} decode={r0_decode} prefill_now={r1_now} remaining={prefill_left - r1_now}")1hit=4 decode=1 prefill_now=5 remaining=4A LoRA extra key turns a four-token hit into zero even though the token IDs match. A page size of 4 still keeps the shared prefix, but a five-token shared run would reuse only the first full page.
1 + 5 = 6 admitted tokens, with four prompt tokens still waiting. Physical pages and per-layer KV layouts are omitted.When the waiting queue is long, request order changes hit rate. The paper's cache-aware scheduler prefers requests with longer matched prefixes instead of pure first-come, first-served order.[1] That can raise reuse and also starve unrelated traffic, so fairness is a real trade-off.
Current docs also describe hierarchical KV caching (HiCache) for host and storage tiers when GPU memory isn't the only cache.[2] The same exact-match rule still applies; a colder tier changes where the bytes live, not what counts as a match.
Insert, lock, evict
A prefix cache needs ownership rules alongside its lookup map. While a request is active, its matched KV locations are protected. When a request finishes or releases a branch, nodes can become evictable. The eviction policy can be LRU or another configured strategy, but it must never reclaim state still referenced by a live request.
The source path is visible in python/sglang/srt/mem_cache/radix_cache.py:
match_prefixfinds the longest compatible path and returns KV indices.insertadds a new token path after forward computation.- Lock and unlock operations protect prefixes while requests use them.
evictremoves unprotected branches according to the selected policy.
This sequence exposes a common failure. If a request's KV reference is released too early, another request can overwrite memory while the first is still decoding. If references never drop, free KV capacity shrinks until admission stalls. Instrument active references, evictable tokens, match lengths, and allocator failures together.
Try releasing the shared prefix while R1 is still using it. This CPU state model has one leaf segment and two owners; the real tree propagates locks through ancestors and coordinates an allocator. It checks the lifetime rule, not SGLang's concurrent implementation:
1class PrefixSegment:
2 def __init__(self, locations):
3 self.locations = tuple(locations)
4 self.owners = set()
5 self.evicted = False
6
7 def acquire(self, request):
8 if self.evicted or request in self.owners:
9 raise ValueError("invalid acquisition")
10 self.owners.add(request)
11
12 def release(self, request):
13 if request not in self.owners:
14 raise ValueError("request does not own this segment")
15 self.owners.remove(request)
16
17 def evict(self):
18 if self.owners:
19 return False
20 self.evicted = True
21 return True
22
23prefix = PrefixSegment([12, 13, 14, 15])
24prefix.acquire("R0")
25prefix.acquire("R1")
26prefix.release("R0")
27assert not prefix.evict()
28print("R0 finished; R1 still owns four KV locations: eviction blocked")
29prefix.release("R1")
30assert prefix.evict()
31try:
32 prefix.acquire("R2")
33except ValueError:
34 print("Both finished: evicted segment cannot be reacquired")
35else:
36 raise AssertionError("stale locations reused")1R0 finished; R1 still owns four KV locations: eviction blocked
2Both finished: evicted segment cannot be reacquiredThe cache is local to an engine or replica unless a higher-level routing layer coordinates it. A load balancer that sends identical prompts to different replicas may see low hit rates even when each replica has a healthy tree. Cache-aware routing can improve locality, but adding replicas still changes hit probability and warm-up behavior.
Why doesn't an embedding similarity qualify as a RadixAttention hit?
Answer
The cache reuses exact token-key paths and compatible extra keys. Similar meaning doesn't imply identical hidden states, so semantic similarity needs retrieval or application logic, not KV reuse.
Structured generation: constrain the next token
Many applications need valid JSON, a tool-call schema, a regular-language field, or one label from a known set. Asking a model to "please emit JSON" is a prompt preference, not a syntax guarantee.
SGLang's server APIs support regex, JSON schema, and EBNF constraints. Those three parameters are mutually exclusive, not mandatory for every request.[5] At the pinned commit, frontend sgl.gen exposes regex and json_schema, but not an ebnf argument. Don't infer a frontend signature from a server API example.[4]
At each decoding step, a grammar engine tracks an automaton state. It computes the set of tokens that keep the partial output valid, and the sampler masks other logits. The model still chooses among allowed tokens. Predict the boundary: grammar changes which tokens are legal, not which facts the model knows. It doesn't repair a semantically wrong field.
The model supplies the entity value; the schema constrains its serialized shape. This unexecuted integration fragment requires the same configured backend as the first frontend example:
1import sglang as sgl
2
3@sgl.function
4def extract(s, passage):
5 s += "Extract one entity from this passage:\n" + passage
6 s += sgl.gen(
7 "record",
8 json_schema='{"type":"object","properties":{"name":{"type":"string"}},"required":["name"],"additionalProperties":false}',
9 max_tokens=64,
10 )The pinned commit ships three grammar backends: XGrammar (the documented default), Outlines, and llguidance. The selected backend, tokenizer, vocabulary, and schema shape affect CPU work and compatibility.
A grammar with many alternatives can take more time to update than a small enum. Measure grammar preparation, per-step mask time, and end-to-end latency; constrained requests don't all cost the same.
The original SGLang work also introduced compressed finite-state-machine techniques for faster structured decoding.[1] The enduring design is a boundary: grammar state stays on the control side, while logits remain on the model side. Constraints stay inspectable, and the runtime can reuse model KV state even when grammar states differ.
Failure handling needs an explicit policy. If no token satisfies the grammar, the request should return an error or a clearly marked incomplete result. A token limit, cancellation, or transport failure can also truncate an otherwise valid prefix. Check the finish reason and validate the complete object before acting on it. Retrying with a looser grammar changes the contract; silently repairing braces doesn't establish validity.
Speculative decoding: verify a draft in parallel
Autoregressive decoding normally produces one token per target-model forward step. Speculative decoding asks a cheaper draft path for several candidates, then lets the target model verify them in one pass. Accepted tokens advance the sequence faster; rejected tokens fall back to a target-sampled token. The trade is simple to predict: saved target steps help only when draft work and verification cost less than repeated target decoding.
Current docs expose several algorithms: EAGLE-2 and EAGLE-3 (Extrapolation Algorithm for Greater Language-model Efficiency), multi-token prediction (MTP) heads that some models already train, DFlash-family drafts, a standalone smaller draft LLM, and an n-gram lookup that needs no extra model.[6] The enum and worker registry live under python/sglang/srt/speculative/ in the pinned commit, including spec_info.py and spec_registry.py.
Flags and model requirements are version-specific. Check those files before enabling a mode, then test its interaction with grammar, chunked prefill, and overlap.
For a greedy linear example, suppose an eight-token draft first disagrees with the target at position six. Commit the five matching tokens and the target's replacement at position six, not the rejected suffix. Stochastic verification instead uses its acceptance and correction rules; it isn't generally a comparison with the target's argmax. Tree drafts add branch selection to this bookkeeping.
| Phase | Work | State |
|---|---|---|
| Draft | Propose y1 ... yk from a cheap path | Draft tokens and optional draft KV |
| Verify | Target scores candidates with one batched forward | Target logits and acceptance tests |
| Commit | Keep accepted prefix and target replacement/bonus token | Retain valid KV; rejected suffix is discarded |
EAGLE-2 and EAGLE-3 research explores draft generation from target-model representations and tree-shaped candidates.[7][8] DFlash-family work targets faster block-diffusion draft paths in newer SGLang releases.[9] MTP reuses extra heads on the target model, so the draft path may not be a separate checkpoint.
Whatever supplies the draft, implementation state still has to agree: token ordering, position IDs, grammar state, and cache ownership move with the accepted prefix.
The docs' own Llama 3.1 8B Instruct snapshot on MT-bench with one H100 reports 158.34 tokens/s without speculation, 244.10 with EAGLE-2, and 373.25 with EAGLE-3.[6] Those numbers are a dated workload, not a service SLO. Treat them as a reproducible comparison only when model, hardware, request mix, and runtime settings match.
Speculation helps when acceptance is high and draft work is cheaper than the target steps it avoids. Short outputs leave little work to amortize startup; high-concurrency target batches may already use the hardware efficiently. Low draft acceptance, grammar interactions, and extra draft KV can erase the benefit. Prompt length alone doesn't determine whether speculation helps.
DFlash and n-gram modes also disable some overlap and mixed chunked-prefill paths, so a flag that looks free can change the scheduler. Track acceptance length, draft and verify latency, rejected tokens, extra memory, and ITL by model and workload.
What evidence would convince you speculative decoding is hurting a service?
Answer
Low accepted-token length together with draft overhead, rising memory pressure, or unchanged target-model time. Compare against the same traffic with speculation disabled and split draft, verify, and commit timings.
Parallelism and prefill-decode disaggregation
SGLang maps model work onto hardware through several parallelism axes. Before choosing one, predict the bottleneck: fitting weights, doing per-request compute, moving tokens between experts, or keeping decode local. Names vary by model, but the trade-offs are stable:
| Axis | What is split | Helps with | Costs |
|---|---|---|---|
| Tensor parallelism (TP) | Matrix dimensions across devices | Fit and per-request compute | Per-layer collectives |
| Pipeline parallelism (PP) | Layer ranges across stages | Fit depth across nodes | Pipeline bubbles and stage sync |
| Data parallelism (DP) | Full model replicas or request groups | Independent throughput | Weight memory and routing |
| Expert parallelism (EP) | Mixture-of-Experts experts | Expert capacity and load balance | Token routing and all-to-all |
| Context or sequence paths | Sequence positions or attention work | Long contexts and bandwidth | Extra communication and layout constraints |
These choices have to align with attention backends, quantization, CUDA graphs, and model-specific layers. A TP setting that fits weights can still increase ITL if collective overhead dominates. A DP replica can raise throughput while lowering prefix-cache locality. SGLang's DP-attention path also differs from independent full-model replicas: attention can be data-parallel while other model parts remain sharded. Read the parallelism docs and benchmark the target topology before copying flags.[10]
Prefill and decode have different resource shapes. Prefill consumes many prompt tokens and tends to be compute-heavy. Decode consumes one or a few tokens per request and repeatedly reads KV memory, making it more latency and memory sensitive. Prefill-decode (P/D) disaggregation places them on separate worker pools and transfers KV state between them.[11]
The P/D path can protect decode latency from a long prompt and let each pool scale for its own traffic. It adds transfer bytes, synchronization, routing, and a cache locality decision. Current docs use transfer engines such as Mooncake and NIXL, then a router (SGLang Model Gateway) to place prefill and decode instances.[11]
Disaggregation doesn't remove prompt computation. It moves that work away from decode and permits different resource allocation. Compare the reduction in interference and queueing against added transfer, synchronization, and routing costs. A lower transfer bound is KV bytes divided by achieved payload bandwidth, not advertised link speed.
For a hypothetical full-attention model with 32 layers, 8 KV heads, head dimension 128, 4,096 tokens, and two bytes per scalar, count both K and V. The check below assumes no sharding, quantization, compression, or prefix reuse. It measures no network:
1layers, kv_heads, head_dim, tokens, bytes_per_scalar = 32, 8, 128, 4096, 2
2kv_bytes = 2 * layers * kv_heads * head_dim * tokens * bytes_per_scalar
3payload_bytes_per_second = 25_000_000_000 # hypothetical achieved 25 GB/s
4lower_bound_ms = 1000 * kv_bytes / payload_bytes_per_second
5assert kv_bytes == 536_870_912
6print(f"KV={kv_bytes / 2**20:.0f} MiB; transfer lower bound={lower_bound_ms:.2f} ms")1KV=512 MiB; transfer lower bound=21.47 msThat is 512 MiB and about 21.47 ms before setup, queueing, or contention. For a 10 ms transfer allowance, this configuration already misses on payload time alone. P/D could still improve another workload's decode tail latency; the bound doesn't predict the whole service.
P/D disaggregation isn't a free scale-out switch. Test queueing at both pools, transfer latency, failure recovery, and cache hit rate. A decode worker that loses its prefill connection needs a clear retry or cancellation path so stale KV state isn't treated as valid.
Applications and workload shape
The runtime pays off when request structure matches runtime structure. Use the classify-and-explain example as a test: is context stable, are branches independent, and does each extra mechanism preserve its output contract? Widen only when the next workload exposes a new boundary:
| Application | Runtime opportunity | What can break |
|---|---|---|
| Agent loop | Reuse system and tool instructions; batch independent branches | Branches diverge early, tool latency dominates |
| Retrieval-augmented generation | Cache stable prompt and template prefix | Retrieved chunks change order or include random IDs |
| Structured extraction | Grammar masks invalid tokens | Schema is large or semantically underspecified |
| Reasoning and self-consistency | Run candidate branches together | More samples multiply KV and verification cost |
| Multi-turn chat | Keep conversation prefix hot | Context trimming or replica routing loses locality |
| Multimodal chat | Share batching and model runner infrastructure | Media preprocessing, transfer, and model support vary |
| RL rollout | High-throughput generation and weight updates | Version skew between trainer and rollout workers |
The paper evaluates several of these program shapes; RL rollout adds a weight-freshness boundary developed in the next chapter.[1] Not every row benefits from every feature.
Strengths and common pitfalls
Now turn mechanisms into decisions. Each strength below has a boundary that should shape a benchmark or a fallback:
| Dimension | Strength | Boundary or weakness |
|---|---|---|
| Programming model | Frontend names generation sites and branches | Extra abstraction can hide a prompt or state bug |
| Prefix reuse | Radix tree finds exact shared token paths | Small prompt changes or replica routing erase hits |
| Scheduling | Continuous batching, chunked prefill, and overlap share a step | Queue policy can trade TTFT, ITL, and fairness |
| Structured output | Grammar constraints enforce syntax during sampling | Grammar preparation and per-step masks add CPU work |
| Speculation | Accepted draft tokens reduce target-model steps | Low acceptance turns draft work into overhead |
| Hardware | Multiple attention backends and parallelism axes | Compatibility matrix changes quickly by model and accelerator |
| Scale | P/D and distributed execution separate bottlenecks | KV transfer, collectives, and routing add failure surfaces |
| Ecosystem | OpenAI API, frontend, model integrations, and RL clients | API compatibility doesn't mean identical sampling or error behavior |
Most incidents fit one of four boundaries. Start with the earliest boundary that can explain the symptom, then follow state forward:
- Prompt boundary: tokenization or chat-template output differs from what the application expects.
- Cache boundary: prefix keys, references, or allocator capacity don't match request lifetime.
- Execution boundary: a scheduler batch, attention backend, or collective stalls or returns wrong shapes.
- Output boundary: grammar, sampler, detokenizer, or stream assembly changes the result.
Separate those boundaries in telemetry. A single end-to-end latency number can't tell whether a prefix hit helped or whether a grammar mask consumed the saved time.
A production failure drill
Suppose a service reports low GPU utilization, high TTFT, and almost no radix hits. Before changing a flag, predict the cheapest evidence: request identity, replica placement, queueing, then KV capacity.
- Compare token IDs and chat-template versions on controlled fixtures. Length alone can't reveal an early mismatch. Avoid logging production prompt contents; use approved redacted traces and tenant-scoped diagnostics.
- Check router placement. Identical requests on different replicas won't share a local tree unless routing is cache-aware.
- Inspect scheduler queue wait, tokenizer time, and
max_prefill_tokensor chunked-prefill settings. - Check KV allocator free capacity and evictable tokens. A full cache can trigger admission stalls even when compute is idle.
- Compare grammar and speculation flags. CPU grammar work or rejected drafts can make GPU gaps look like a cache issue.
- In an isolated replay, compare speculation and grammar disabled against the production configuration. Don't relax live structured-output contracts to collect a faster baseline.
Now invert the symptoms: high GPU utilization, good radix hits, but ITL spikes when a long prompt arrives. That points toward prefill scheduling, chunk size, collective wait, or P/D routing. Increasing cache size won't fix a token-budget policy that lets prefill consume every step.
Which observation separates "cache is cold" from "cache is full"?
Answer
A low match length shows reuse is absent, while allocator free capacity, evictable tokens, and admission failures show whether memory is full. You need both sets of signals before changing cache policy.
Project identity
The SGLang paper came from researchers and engineers around UC Berkeley, Stanford, and the LMSYS community, including Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Ying Sheng, Ion Stoica, Joseph E. Gonzalez, and collaborators.[1] LMSYS is a 501(c)(3) nonprofit that incubates open-source systems and research projects; SGLang lives under the sgl-project GitHub organization.[12]
The current repository is a multi-contributor project rather than a single-vendor SDK. Support and performance therefore depend on model, hardware, and release.
| Field | Current project fact |
|---|---|
| Origin | Researchers around UC Berkeley, Stanford, and LMSYS built SGLang around language-model programming and efficient serving. The founding paper names Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Ying Sheng, and collaborators.[1] |
| Stewardship | LMSYS, a nonprofit research organization, incubates SGLang under the sgl-project GitHub organization.[12] |
| Contributor path | MAINTAINER.md and CODEOWNERS define merge on-call, code-owner, write, CI, and review responsibilities.[13] |
| Source license | Core SGLang source is Apache-2.0.[14] |
| Commercial boundary | Sponsors, vendors, and consultants can support or integrate SGLang without owning its nonprofit-hosted repository. |
| Asset boundary | Model checkpoints, tokenizers, datasets, and third-party kernels can carry licenses separate from SGLang's source license. |
Read the history as a sequence of co-design decisions:
| Research or release thread | Runtime idea it motivates |
|---|---|
| SGLang frontend plus runtime paper | Treat a multi-call LM program as one schedulable workload |
| RadixAttention | Reuse exact token prefixes through a tree of KV locations |
| Compressed grammar decoding | Keep structured output constraints on the decoding path |
| v0.4 scheduler and cache-aware work | Reduce host scheduling overhead and preserve locality as traffic grows[15] |
| EAGLE-2, EAGLE-3, and MTP | Draft and verify multiple candidate tokens, including model-native extra heads[7][8] |
| DFlash-family research | Add newer parallel draft paths to SGLang's speculative registry[9] |
| P/D and distributed runtime docs | Separate prompt compute from decode memory traffic when topology supports it[11] |
The code keeps these ideas modular enough to inspect. A paper explains why a mechanism helps; a release guide explains which flags and models expose it; the pinned source shows ownership, state transitions, and failure handling. Read all three before making a production claim.
Code-reading route in the pinned commit
This walkthrough uses the official SGLang repository at commit f8e62a9224815cc9c6fc56b940eb7fde791a8870.[4] The web documentation was checked on September 2, 2026; a wheel or newer checkout can have different defaults.
Read one request through these files:
python/sglang/lang/api.py: publicfunction,gen,select, role, regex, and JSON-schema operations.python/sglang/lang/ir.py: expression nodes and the program representation passed to a backend.python/sglang/lang/interpreter.py: program state, stream executor, batch execution, and optional prefix tracing.python/sglang/srt/entrypoints/http_server.py: online request boundary and server routes.python/sglang/srt/managers/tokenizer_manager.pyanddetokenizer_manager.py: input preparation, request tracking, and output text conversion.python/sglang/srt/managers/scheduler.pyandschedule_policy.py: token pools, chunked prefill, overlap setup, andPrefillAdderadmission budgets.python/sglang/srt/mem_cache/radix_cache.py:match_prefix, insert, reference protection, and eviction.python/sglang/srt/model_executor/model_runner.py: model loading, forward-batch setup, attention backends, and parallel state.python/sglang/srt/constrained/grammar_manager.py: grammar backend selection and constrained token state.python/sglang/srt/speculative/spec_info.pyandspec_registry.py: speculative algorithm registry and draft/verify capability checks.python/sglang/srt/disaggregation/prefill.pyanddecode.py: P/D role boundaries and KV transfer lifecycle.
Keep these invariants beside the source:
- A frontend generation name identifies a value in program state; it isn't a cache key by itself.
- A radix match is exact over token IDs and compatible extra keys; semantic similarity doesn't qualify.
- A scheduler may execute only tokens backed by valid KV locations and request ownership.
- A grammar filter can remove logits but can't make an invalid model state valid.
- Speculative verification can commit only accepted tokens in order and must update grammar and KV state together.
- P/D transfer must complete before decode consumes the transferred KV range.
Rerun and measure
Start with the three standard-library checks above: exact prefix matching and budget accounting, prefix lifetime, and a KV-transfer bound. They don't launch SRT or load weights. For an integration run on supported hardware, configure a backend and verify that the frontend program returns named values. Send the same tokenized prompt twice, then change one early token and compare matched-prefix tokens and prefill work. A mock backend can check call order and names, but can't demonstrate radix hits or model correctness.
For serving tests, a performance number needs a receipt. Record the accelerator model, count, and topology; SGLang, model, and tokenizer revisions; a realistic request or public trace with prompt and output-length distributions, concurrency or arrival process, and warmup and steady-state window.
Also record precision, quantization, attention backend, flags, and cache state, plus the exact baseline and correctness check. Without that tuple, a speedup is hard to interpret.
Record these runtime signals:
| Layer | Minimum signals |
|---|---|
| Input | token count, template version, media preprocessing time |
| Admission | queue wait, matched-prefix tokens, allocated KV tokens |
| Scheduler | batch token count, prefill chunks, overlap delay, preemptions |
| Model runner | forward time, attention backend, collectives, GPU utilization |
| Grammar or speculation | mask time, accepted draft length, rejection count |
| Output | TTFT, ITL, finish reason, detokenization time |
For serving, report goodput as the offered request rate whose required fraction of valid completions meets explicit TTFT and ITL SLOs, not raw tokens per second alone. Keep p50 and p99 latency, queue wait, cache hit or matched tokens, accepted draft length, memory pressure, warmup or compile time, and error rate beside that number.
Change one mechanism at a time against an explicit baseline; don't assume the server starts with radix caching off. Separate cold-cache from warm-cache trials and keep schemas fixed while comparing speculation. A grammar-off replay diagnoses overhead but isn't a valid replacement for a constrained service. For P/D, compare colocated and disaggregated modes with the same prompt mix and latency targets.
When service metrics and kernel timing disagree, use the tool that answers the layer in question. Nsight Systems can show host scheduling, queue gaps, transfers, and GPU launch timelines; Nsight Compute can explain a selected kernel's occupancy, memory behavior, and instruction mix.[16][17]
Measure one change, inspect the trace, then verify the same workload again. A profiler screenshot without a fixed replay is an observation, not an attribution.
Architectural summary
- The program frontend is optional. Plain API requests can use SRT without it.
- Frontend expressions expose generation sites, branches, and constraints that a flattened prompt hides.
- RadixAttention is a prefix-cache data structure and scheduling aid, not an attention kernel.
- A token budget, chunked prefill, and overlap let decodes and prefills share repeated steps, with TTFT and ITL trade-offs.
- Grammar constraints mask invalid next tokens; they don't guarantee semantic correctness.
- Speculative decoding trades draft work and memory for accepted target tokens. Acceptance length is the key health signal.
- TP, PP, DP, EP, and P/D disaggregation solve different fit and latency problems, each with communication costs.
- Production debugging needs token, cache, scheduler, runner, grammar, and stream signals, not one throughput number.
Explain the runtime path without looking at the diagram. Then answer why a stable system prefix helps two classify requests, why a grammar can increase CPU time, and why a P/D split can lose to colocated execution on a bandwidth-limited network.
Evaluation rubric
- Context identity: Locate the first differing token, distinguish repeated suffixes from shared prefixes, and separate model KV from request-local grammar state.
- Memory lifetime: Follow matched KV locations through acquisition, completion, cancellation, and eviction without freeing live state.
- Scheduling: Reproduce the six-token example, then identify the actual mixed-batch, page, and capacity checks it omits.
- Execution evidence: Pin model and source revisions; separate frontend checks, CPU arithmetic, GPU correctness, and service-level goodput.
- Distributed readiness: Account for KV bytes and ensure decode waits for valid transferred state, with a defined retry or cancellation path.
Follow-up questions
The four-token prefix is shared, but R1 changes only its output grammar. Does it need fresh prompt KV?
Answer
Not solely because its output mask changed. If model state, prompt tokens, positions, and cache namespace remain compatible, prompt KV can be shared while grammar state remains per request. If the schema is rendered into the prompt or policy requires a different namespace, the reuse decision changes.
The preview deployment enables chunked prefill and still delays decodes. What would you inspect before setting the chunk to six tokens?
Answer
Check the installed mixed-chunk setting, actual batch trace, page alignment, KV reservations, running-request limits, and kernel/collective timings. Six is an illustrative accounting budget, not a recommended production flag. Measure ITL and TTFT on the same arrival pattern after each change.