LearnInference & Production ScaleContinuous Batching & Scheduling

🚀HardInference Optimization

Continuous Batching & Scheduling

Understand how LLM schedulers use continuous batching, chunked prefill, and prefill-decode disaggregation to improve throughput without violating TTFT, TPOT, or inter-token latency targets.

32 min read

Learning path

Step 141 of 196 in the full curriculum

FlashAttention & Memory Efficiency Scaling LLM Inference

Personalize this lesson

Adapt explanations and teaching visuals to your background and preferred voice.

At 09:47 during a release freeze, an on-call engineer sees one shared code-assistant GPU receive three requests: a one-line completion (about 10 output tokens), an explanation of a failing test (about 50), and a migration plan (about 200). More work is already queued.

If the server starts those requests together, A disappears after 10 generation steps and B after 50. Their seats still belong to the batch while C runs for another 150 steps. The user-visible choice is simple but consequential: protect streams already producing tokens, or admit work that hasn't produced its first token?

FlashAttention makes one attention kernel cheaper. It doesn't decide who shares the next forward pass.

Static batching treats the trio as one job: fill a batch, run it, and wait for the longest completion before anyone new can start. Continuous batching (iteration-level scheduling) rebuilds the active set after every token step. When A stops, queued request D can take that slot on the next iteration. The model doesn't change. The GPU just keeps more useful work in flight.^{[1]Reference 1Orca: A Distributed Serving System for Transformer-Based Generative Models.https://www.usenix.org/conference/osdi22/presentation/yu}^{[2]Reference 2Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}

Admission still depends on free KV-cache memory, so the scheduler only makes sense after paged KV management.

Static batching leaves request A idle after 10 tokens and B idle after 50 while C runs to 200. Continuous batching reuses those slots: D replaces A and E replaces B while C keeps decoding. — On the left, A and B keep reserved seats until C finishes. On the right, D replaces A and E replaces B at the next decode boundary, so all three slots stay useful.

Why LLM batches don't stay the same shape

Image classifiers usually have a fixed input shape and finish in one forward pass. Autoregressive LLMs don't.

Prefill reads the prompt. The prefill phase can process those tokens in parallel. Decode then emits one new token per step. A one-liner might need ten decode steps. A migration plan might need two hundred.

The engine also doesn't know the output length up front. Generation stops on an end-of-sequence token or a length limit, so the scheduler can't freeze a batch shape that will still make sense 200 steps later.

Those two facts are why a request-level batch wastes capacity on mixed traffic. Once A is done, its seat is empty, but static batching still owns the whole group until C finishes.

Request	Task	Output tokens	Static fate
A	One-line completion	10	Done at step 10, waits 190 more
B	Failing-test explanation	50	Done at step 50, waits 150 more
C	Migration plan	200	Finishes at step 200

If the GPU has three decode seats, static batching spends $3 \times 200 = 600$ seat-steps to do $10 + 50 + 200 = 260$ useful tokens. The other 340 seat-steps are idle. Requests A and B are finished, but their KV memory often stays reserved until C completes, so D and E can't join.

Count each active output token as one useful slot-iteration:

static-versus-continuous-slot-use.py

output_lengths = [10, 50, 200]
batch_slots = len(output_lengths)
static_capacity = max(output_lengths) * batch_slots
useful_work = sum(output_lengths)
idle_work = static_capacity - useful_work

print(f"useful slot-iterations: {useful_work}")
print(f"static idle slot-iterations: {idle_work}")
print(f"static utilization: {useful_work / static_capacity:.1%}")

Output

useful slot-iterations: 260
static idle slot-iterations: 340
static utilization: 43.3%

Slot utilization isn't GPU utilization. 43.3% is batch-seat occupancy: how often a request-shaped seat is busy. Filling every seat still leaves tensor cores idle when decode is limited by memory bandwidth, and HBM can be nearly full while model FLOPs utilization (MFU) stays low.

Regime	Seat occupancy	Typical decode MFU
Static batch, mixed lengths	Low after short finishes	Still often bandwidth-bound
Continuous batch, full seats	High	Still often low MFU on pure decode
Larger token budget / chunked prefill co-batch	High, more prefill tokens per step	Higher arithmetic intensity on that mixed step

Continuous batching is a capacity win because it keeps more useful work in flight. It doesn't promise that SM occupancy or MFU jump to 100%. The open question is how the server actually refills those seats without waiting for C.

Continuous batching: schedule every iteration

Orca named this iteration-level scheduling: after each generation step (or after each prefill chunk in engines that chunk prefills), the server checks who finished, frees their slots, and admits waiting work if memory and token-budget constraints allow.^{[1]Reference 1Orca: A Distributed Serving System for Transformer-Based Generative Models.https://www.usenix.org/conference/osdi22/presentation/yu}

Start with [A, B, C]. At iteration 10, A finishes and D joins. At iteration 50, B finishes and E joins. C keeps decoding the whole time. The GPU stays at three busy seats until the waiting queue is empty.

Step	Active set	What changed
0	A, B, C	Initial batch
10	D, B, C	A done, D joins
50	D, E, C	B done, E joins
200	D, E, (C done)	C leaves; next queued work can take the seat

That loop has one invariant: at every boundary, rebuild the active set before the next forward pass. First retire finished sequences and free their KV. Then admit waiting work that fits the batch and token budgets.

If the projected next token would exceed the KV budget, preempt a victim before running the pass. Otherwise, run one forward pass. The order makes the trade-off visible: reuse freed capacity, protect the next token from an OOM, then do useful work.

Diagram showing Retire + free KV, Admit under budgets, Next token fits KV?, and Preempt a victim. — Retire + free KV, Admit under budgets, Next token fits KV?, and Preempt a victim.

Real engines add prefix-cache hits, cancellations, and speculative drafts on top of this loop. The invariant stays the same: the active set is allowed to change at every iteration boundary.

Seat count is the wrong capacity knob, though. One long prefill can consume more scheduler work than many short decode-only streams. Modern schedulers cap a per-step token budget (vLLM's max_num_batched_tokens) and a KV-cache block budget, not only max_num_seqs.

A prefix-cache hit doesn't speed decode by itself, but it does shrink how many prefill tokens compete for this step's budget. Decode-first policy then spends leftover tokens on a bounded prefill chunk.

Suppose 96 live completions already need a decode token each, and a 1,800-token test-log prefill is waiting. With max_num_batched_tokens = 1024, decode takes 96 slots first. The remaining 928 tokens can run as a prefill chunk. The other 872 prompt tokens wait for later steps.

The same 1,896 tokens of work: a stall step packs 96 decode tokens with an 1,800-token prefill, while a 1,024-token decode-first budget runs 96 decode tokens plus a 928-token prefill chunk and defers 872 prompt tokens. — Both bars contain the same 1,896 tokens of work. Decode-first scheduling keeps the live 96 streams in the step, spends the rest of a 1,024-token budget on a 928-token chunk, and defers 872 prompt tokens instead of stalling behind the whole log.

decode-first-token-budget.py

max_num_batched_tokens = 1024
decode_requests = 96
pending_prefill_tokens = 1800
prefill_budget = max_num_batched_tokens - decode_requests
scheduled_prefill = min(pending_prefill_tokens, prefill_budget)

print(f"decode tokens scheduled first: {decode_requests}")
print(f"prefill tokens scheduled now: {scheduled_prefill}")
print(f"prefill tokens left for later: {pending_prefill_tokens - scheduled_prefill}")

Output

decode tokens scheduled first: 96
prefill tokens scheduled now: 928
prefill tokens left for later: 872

That budget only helps if the memory manager can actually hand D the blocks A just freed. Contiguous KV reservations often can't.

Paged KV is what makes refill possible

During decode, the model stores a KV cache: attention keys and values for every token so far. Under continuous batching, sequences join and leave at unpredictable times. A contiguous max_seq_len reservation per request then fails in two ways:

Internal fragmentation. You pre-allocate for 4,096 tokens even if A generates 10.
External fragmentation. When A leaves, it can punch a hole that D's prompt doesn't fit into, even if the GPU has enough free bytes overall.

The KV cache for a sequence grows with every decode step. For Qwen3.6-27B's published Gated Attention geometry (16 full-attention layers, 4 KV heads, head dimension 256, FP16 cache), each new token adds about 64 KiB of KV state.^{[3]Reference 3Qwen3.6-27Bhttps://huggingface.co/Qwen/Qwen3.6-27B} A 4,096-token test log therefore needs roughly 0.25 GiB. If those same 16 layers stored separate K/V tensors for all 24 query heads instead of GQA, the cache would grow to roughly 1.5 GiB at 4K tokens. Modest over-allocation can strand enough VRAM to block D.

M_{KV} = 2 \times L \times s \times H_{kv} \times d \times b

$L$ is the number of full-attention layers that store K/V, $s$ is sequence length, $H_{kv}$ is KV heads, $d$ is head dimension, and $b$ is bytes per element (2 for FP16). The leading 2 counts key and value tensors. Qwen3.6-27B has 64 layers total, but 48 of them are Gated DeltaNet; only the 16 Gated Attention layers grow this cache.

kv-cache-budget.py

layers, kv_heads, head_dim, bytes_per_value = 16, 4, 256, 2
tokens = 4096
bytes_per_token = 2 * layers * kv_heads * head_dim * bytes_per_value
cache_bytes = bytes_per_token * tokens

print(f"KV state per token: {bytes_per_token / 1024:.0f} KiB")
print(f"KV state at {tokens:,} tokens: {cache_bytes / 1024**3:.2f} GiB")

Output

KV state per token: 64 KiB
KV state at 4,096 tokens: 0.25 GiB

PagedAttention^{[2]Reference 2Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} is the memory manager that makes iteration-level admission practical. It splits KV into fixed-size blocks (often 16 or 32 tokens) and maps logical block indices to physical GPU pages through a per-request block table. When A leaves, its pages return to the free list. D can take any of them, even if they aren't adjacent.

After short request A leaves, contiguous KV reservation leaves a 2-block hole that 3-block request D cannot use, while paged KV places D on any free physical blocks and keeps C and B in place. — After A leaves, a contiguous allocator can be stuck with a hole that D doesn't fit. Paged KV lets D take any free physical blocks while C and B stay put.

The vLLM paper reports near-zero KV-cache waste in its evaluated design versus large reservation waste in the baselines.^{[2]Reference 2Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} That claim is about dropping max_seq_len over-allocation, not about last-block rounding. Fixed blocks still leave slack in the final page of each sequence. On short sequences that slack looks large. On long ones it's at most block_size - 1 tokens per request.

paged-kv-slack.py

block_size = 16
sequence_lengths = [17, 31, 48, 65]
reserved = sum(((length + block_size - 1) // block_size) * block_size for length in sequence_lengths)
used = sum(sequence_lengths)
slack = reserved - used

print(f"active tokens: {used}")
print(f"reserved block slots: {reserved}")
print(f"last-block slack: {slack} tokens ({slack / reserved:.1%})")

Output

active tokens: 161
reserved block slots: 192
last-block slack: 31 tokens (16.1%)

PagedAttention manages where KV lives. FlashAttention reduces HBM traffic inside the attention kernel. They're orthogonal. Engines such as vLLM combine them so you can both admit more sequences and run each step faster.

Memory is now flexible enough to refill seats. The next fight is that prefill and decode want different things from the same GPU.

Prefill and decode fight over the same step

LLM serving has two phases with opposing bottlenecks:

Prefill (prompt processing)

Processes input tokens in parallel.
Often compute-heavy because many prompt tokens hit matrix multiplies.
Runs once per request at the start (or in chunks).
Latency goal: Time To First Token (TTFT).

Decode (token generation)

Emits tokens one at a time.
Often memory-bandwidth limited at practical batch sizes: each step reads model weights and KV to produce few new tokens.
Runs once per output token.
Latency goal: inter-token latency (ITL), also called time between tokens (TBT). TPOT is related but not identical. DistServe defines TPOT as the average time per output token after the first.^{[4]Reference 4DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670}

Phase	Arithmetic intensity	Bottleneck	What you want to batch
Prefill	Often high	Frequently compute	Prompt tokens
Decode	Often lower	Frequently memory bandwidth	Active streams

Mixing a full 4k test-log prefill with those 96 live streams can stall ITL. Every in-flight completion waits for the whole prompt before its next token. Chunked prefill, disaggregated pools, and decode-first priority are three ways to bound that interference.

Who gets the next seat

The scheduler rebuilds the active set every iteration. Policy decides which waiting work wins.

First-come-first-served (FCFS)

Admit in arrival order. Easy to reason about, usually fair, and a long migration plan at the head of the queue can block many one-liners.

Shortest-job-first (SJF)

Prioritize requests expected to need fewer output tokens. If you knew lengths, average completion time would drop because A wouldn't sit behind C. You don't know those lengths, so pure SJF is rare. Production systems approximate it with prompt-length buckets, tenant priorities, or token-budget caps.

Unknown lengths turn scheduling into a fairness problem too. A stream of short requests can keep winning an estimate-based SJF queue while a long request waits forever. Use estimates as hints, age waiting requests, or allocate per-tenant shares; cap consecutive wins when a class is falling behind.

Track p99 and maximum queue wait by request class. If short-job latency improves while long-job wait grows, the scheduler is trading fairness for mean completion time, not finding free capacity.

If output lengths were known, SJF would cut average completion time. They aren't, so this is an oracle bound:

shortest-job-first-bound.py

jobs = [("long", 50), ("quick", 5), ("medium", 20)]

def average_completion(order):
    elapsed = 0
    completion_times = []
    for _, tokens in order:
        elapsed += tokens
        completion_times.append(elapsed)
    return sum(completion_times) / len(completion_times)

fcfs = average_completion(jobs)
oracle_sjf = average_completion(sorted(jobs, key=lambda job: job[1]))
print(f"FCFS average completion: {fcfs:.1f} steps")
print(f"oracle SJF average completion: {oracle_sjf:.1f} steps")
print("production caveat: output length is not known up front")

Output

FCFS average completion: 60.0 steps
oracle SJF average completion: 35.0 steps
production caveat: output length is not known up front

Decode-first and priority

Users notice ITL jitter immediately. That makes decode-first scheduling reasonable: protect streams already producing tokens, then spend leftover budget on new prefills. Per-tenant priorities and SLO-aware admission make that preference explicit.^{[4]Reference 4DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670}

Decode-first still needs a backstop. If decode is never preempted, waiting prefills miss TTFT. Cap decode-only iterations or age a waiting request until it wins a slot.

Preemption when KV is full

When GPU KV blocks run out, the scheduler has to pause work, drop state, or stop admitting. Two recovery designs:

Recompute. Drop the victim's KV and replay its prompt when it's rescheduled.
- Pros: no CPU copy; simple.
- Cons: spends GPU compute; cheapest when the prompt is short.
Swap-out. Move the victim's KV from GPU VRAM to CPU RAM.
- Pros: keeps progress; can beat replay when the prompt is long.
- Cons: burns host-device bandwidth and adds resume latency.

These are design options, not universal defaults. Current vLLM V1 uses RECOMPUTE because recomputation has lower overhead in that architecture, and its V1 guide marks GPU-to-CPU KV swapping as removed.^{[5]Reference 5Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html}^{[6]Reference 6vLLM V1 User Guidehttps://docs.vllm.ai/en/stable/usage/v1_guide/} Other runtimes still offer swap. If you tune a particular engine, read its current scheduler docs and trace preemption events.

The teaching scheduler below implements swap-out so the bandwidth path is visible in the code. schedule_step runs before every forward pass. finish_decode_step runs after that pass emits one token per active request. KV blocks grow as sequences get longer, and a tight budget forces one preemption.

preemption-strategies.py

from dataclasses import dataclass

@dataclass
class Request:
    id: str
    prompt_len: int
    max_new_tokens: int
    priority: int = 0  # Higher value = higher priority
    generated_tokens: int = 0
    allocated_blocks: int = 0
    swapped_to_cpu: bool = False

    @property
    def total_tokens(self) -> int:
        return self.prompt_len + self.generated_tokens

    def is_finished(self) -> bool:
        """Check if request has reached EOS or max_new_tokens."""
        return self.generated_tokens >= self.max_new_tokens

class ContinuousBatchScheduler:
    """
    Simplified iteration-level scheduler.

    Real engines also track per-request token budgets, prefix-cache hits,
    EOS detection, and beam-search groups. This example focuses on the core
    loop: retire finished requests, resume paused work, admit new work, then
    preempt if the next decode step would overflow KV-cache capacity.
    """
    BLOCK_SIZE = 16

    def __init__(self, max_batch_size: int, max_memory_blocks: int):
        self.max_batch_size = max_batch_size
        self.max_memory_blocks = max_memory_blocks
        self.running: list[Request] = []  # Currently generating on GPU
        self.waiting: list[Request] = []  # Queued requests
        self.swapped: list[Request] = []  # Preempted to CPU RAM
        self.used_memory_blocks = 0
        self.preemptions = 0

    def schedule_step(self) -> list[Request]:
        # 1. Retire finished requests and free their KV cache.
        active = []
        for req in self.running:
            if req.is_finished():
                self._free_kv_cache(req)
            else:
                active.append(req)
        self.running = active

        # 2. Resume preempted requests first so they don't starve.
        self._admit(self.swapped, from_cpu=True)

        # 3. Admit fresh waiting requests.
        self._admit(self.waiting, from_cpu=False)

        # 4. Preempt only if the next decode step would exceed the safe budget.
        while self.running and not self._can_run_next_decode_step(self.running):
            victim = min(self.running, key=lambda r: (r.priority, r.generated_tokens))
            self.running.remove(victim)
            self._swap_out_to_cpu(victim)
            victim.swapped_to_cpu = True
            self.swapped.append(victim)
            self.preemptions += 1

        return self.running

    def finish_decode_step(self) -> None:
        # Call this after the model emits one token for each running request.
        for req in self.running:
            req.generated_tokens += 1
            required_blocks = self._blocks_for_tokens(req.total_tokens)
            growth = required_blocks - req.allocated_blocks
            if growth > 0:
                self.used_memory_blocks += growth
                req.allocated_blocks = required_blocks

    def _admit(self, queue: list[Request], from_cpu: bool) -> None:
        queue.sort(key=lambda r: r.priority, reverse=True)

        while (
            queue
            and len(self.running) < self.max_batch_size
            and self._can_allocate_current_state(queue[0])
        ):
            req = queue.pop(0)
            if from_cpu:
                self._swap_in_from_cpu(req)
            else:
                self._allocate_kv_cache(req)
            req.swapped_to_cpu = False
            self.running.append(req)

    def _blocks_for_tokens(self, token_count: int) -> int:
        return (token_count + self.BLOCK_SIZE - 1) // self.BLOCK_SIZE

    def _can_allocate_current_state(self, req: Request) -> bool:
        return self.used_memory_blocks + self._blocks_for_tokens(req.total_tokens) <= self.max_memory_blocks

    def _allocate_kv_cache(self, req: Request) -> None:
        req.allocated_blocks = self._blocks_for_tokens(req.total_tokens)
        self.used_memory_blocks += req.allocated_blocks

    def _free_kv_cache(self, req: Request) -> None:
        self.used_memory_blocks -= req.allocated_blocks
        req.allocated_blocks = 0

    def _swap_out_to_cpu(self, req: Request) -> None:
        self._free_kv_cache(req)

    def _swap_in_from_cpu(self, req: Request) -> None:
        self._allocate_kv_cache(req)

    def _projected_growth_for_next_token(self, req: Request) -> int:
        next_blocks = self._blocks_for_tokens(req.total_tokens + 1)
        return max(0, next_blocks - req.allocated_blocks)

    def _can_run_next_decode_step(self, batch: list[Request]) -> bool:
        growth = sum(self._projected_growth_for_next_token(req) for req in batch)
        return self.used_memory_blocks + growth <= self.max_memory_blocks

scheduler = ContinuousBatchScheduler(max_batch_size=3, max_memory_blocks=16)
scheduler.waiting.extend([
    Request("req_A", prompt_len=8, max_new_tokens=3),
    Request("req_B", prompt_len=8, max_new_tokens=1),
    Request("req_C", prompt_len=8, max_new_tokens=2),
    Request("req_D", prompt_len=8, max_new_tokens=2),
    Request("req_E", prompt_len=8, max_new_tokens=1),
])

timeline: list[list[str]] = []
peak_blocks = 0
for _ in range(6):
    active = scheduler.schedule_step()
    timeline.append([req.id for req in active])
    peak_blocks = max(peak_blocks, scheduler.used_memory_blocks)
    scheduler.finish_decode_step()
    peak_blocks = max(peak_blocks, scheduler.used_memory_blocks)

print(timeline[:3])
print(f"peak blocks: {peak_blocks}/{scheduler.max_memory_blocks}")

assert timeline[0] == ["req_A", "req_B", "req_C"]
assert timeline[1] == ["req_A", "req_C", "req_D"]
assert "req_E" in timeline[2]
assert peak_blocks <= scheduler.max_memory_blocks

# A tight KV-block budget forces an actual preemption before decode.
pressure = ContinuousBatchScheduler(max_batch_size=2, max_memory_blocks=2)
pressure.waiting.extend([
    Request("urgent", prompt_len=16, max_new_tokens=2, priority=1),
    Request("background", prompt_len=16, max_new_tokens=2, priority=0),
])
active_under_pressure = pressure.schedule_step()
print([req.id for req in active_under_pressure], f"preemptions={pressure.preemptions}")
assert [req.id for req in active_under_pressure] == ["urgent"]
assert pressure.preemptions == 1

Output

[['req_A', 'req_B', 'req_C'], ['req_A', 'req_C', 'req_D'], ['req_A', 'req_D', 'req_E']]
peak blocks: 3/16
['urgent'] preemptions=1

schedule_step() rebuilds the active set every iteration: retire, admit under the batch and KV budgets, then preempt if the next token wouldn't fit.

In a real server the control loop alternates schedule_step() and finish_decode_step(): pick the next forward pass, run it once, record the extra token and any new KV block, repeat. Prefix-cache reuse, cancellation, and speculative decoding sit on top of that loop. The tight-budget assertion proves the swap-out branch actually runs.

Recompute is the dual policy, and the one many engines default to:

recompute-preemption-sketch.py

# Contrast with ContinuousBatchScheduler's swap-out path above.
def preempt_by_recompute(self, victim: Request) -> None:
    self._free_blocks(victim.allocated_blocks)
    victim.allocated_blocks = 0
    victim.generated_tokens = 0  # or keep text and only rebuild KV
    victim.swapped_to_cpu = False
    self.running.remove(victim)
    self.waiting.append(victim)  # re-prefill when admitted again
    self.preemptions += 1
# assert: victim returns to waiting with allocated_blocks == 0

Swap-out pays host-device bandwidth to keep KV. Recompute pays prefill FLOPs to rebuild it. Choose from measured prompt length, PCIe bandwidth, and how often preemption fires.

Continuous batching times speculative decoding

Speculative decoding isn't only a per-request speedup. It changes the token-budget math for the whole batch:

Budget inflation. After verify, each active request may emit up to k + 1 tokens (draft span plus a possible bonus). Reserve max_num_batched_tokens as if those multi-token steps can land together, not as one token per request.
ITL jitter from acceptance variance. High acceptance packs more useful tokens per step. Rejects free partial work unevenly and can stall one stream while neighbors look fine, even when mean TPOT looks healthy.
Mixed streams. Speculative and plain decode share one token budget. A burst of long draft verifies can starve ordinary decode seats the same way a fat prefill can.

The accept/reject rule lives in speculative decoding. Keep the iteration loop and token budget from this lesson when you read it.

Serving engines

The same ideas show up across stacks. Some rows are production engines. DistServe is a research system that shaped later pool-splitting designs:

Framework	What it emphasizes	Where it shines
vLLM^{[2]Reference 2Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}	PagedAttention and continuous batching	General-purpose high-throughput serving
TensorRT-LLM^{[7]Reference 7TensorRT-LLM: A High-Performance Inference Framework for LLMs.https://github.com/NVIDIA/TensorRT-LLM}	NVIDIA kernels, in-flight batching, paged KV	Teams standardized on the NVIDIA stack
SGLang^{[8]Reference 8SGLang: Efficient Execution of Structured Language Model Programs.https://arxiv.org/abs/2312.07104}	Continuous batching plus prefix reuse (RadixAttention)	Repeated prefixes or structured generation
DistServe^{[4]Reference 4DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670}	Prefill/decode disaggregation and SLO-aware scheduling	Deployments where TTFT and TPOT beat raw TPS

Feature matrices move quickly. Check current docs before you pick an engine.

Throughput vs goodput

A server can win raw tokens/sec and still feel slow for long prompts or live streams. Judge it by whether your mix of one-liners, test logs, and long plans stays inside TTFT, TPOT, and ITL targets.

Source	What it measured	Why it matters
Orca^{[1]Reference 1Orca: A Distributed Serving System for Transformer-Based Generative Models.https://www.usenix.org/conference/osdi22/presentation/yu}	Iteration-level scheduling against request-level baselines	Per-iteration scheduling can raise throughput and cut latency together
vLLM^{[2]Reference 2Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}	PagedAttention plus continuous batching against prior serving systems	Shows why memory allocation and iteration-level scheduling must be evaluated together
DistServe^{[4]Reference 4DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670}	Phase-disaggregated serving under TTFT/TPOT SLOs	Treats prefill/decode placement as an SLO and goodput decision, not a raw-TPS contest

Raw tokens/sec can look great while a large fraction of requests miss their latency target. DistServe frames goodput as the maximum offered request rate, $\lambda$ , that keeps the required fraction, $\alpha$ , of requests within chosen TTFT and TPOT objectives on each provisioned GPU.^{[4]Reference 4DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670}

\text{Goodput} = \max\left\{\lambda : P(\text{TTFT} \le T_{\text{TTFT}} \text{ and } \text{TPOT} \le T_{\text{TPOT}}) \ge \alpha\right\}

The original DistServe form uses TTFT and TPOT. When smooth streaming is part of your product contract, extend the acceptance predicate with ITL:

\text{Goodput}_{\text{stream}} = \max\left\{\lambda : P\left( \text{TTFT} \le T_{\text{TTFT}} \land \text{TPOT} \le T_{\text{TPOT}} \land \text{ITL} \le T_{\text{ITL}} \right) \ge \alpha\right\}

Define ITL aggregation before measuring: per-stream p99 ITL or a required fraction of tokens under $T_{\text{ITL}}$ are different tests. $T_{\text{TTFT}}$ , $T_{\text{TPOT}}$ , and $T_{\text{ITL}}$ are the chosen limits; $\alpha$ is the attainment fraction, for example 90%. Raise offered load and keep the highest rate whose attainment stays at or above $\alpha$ . Optimize tokens/sec only after those targets exist.

One 10-second window doesn't establish that maximum. It can still report observed SLO-compliant throughput, which is what you use when raw TPS is hiding latency failures:

goodput-under-slo.py

window_seconds = 10
latencies_ms = [
    {"ttft": 120, "tpot": 25},
    {"ttft": 450, "tpot": 22},
    {"ttft": 180, "tpot": 42},
    {"ttft": 190, "tpot": 27},
]
ttft_target, tpot_target = 300, 30
within_slo = [
    item for item in latencies_ms
    if item["ttft"] <= ttft_target and item["tpot"] <= tpot_target
]

print(f"raw throughput: {len(latencies_ms) / window_seconds:.2f} req/s")
print(f"observed SLO-compliant throughput: {len(within_slo) / window_seconds:.2f} req/s")
print(f"SLO-compliant requests: {len(within_slo)}/{len(latencies_ms)}")

Output

raw throughput: 0.40 req/s
observed SLO-compliant throughput: 0.20 req/s
SLO-compliant requests: 2/4

The toy window checks TTFT and TPOT only. A streaming endpoint should add its ITL rule to the same acceptance test, then hold that rule fixed across scheduler configurations.

Turn a mixed-load trace into an operating point

The 10-second window has one useful job: it makes SLO filtering concrete. It doesn't create an arrival process, queue pressure, or output-length mix, so it can't identify an endpoint's capacity. A hand-picked batch can pass while live traffic fails between iterations.

Predict the curve before you run it. As offered arrival rate $\lambda$ rises, queue wait and TTFT should climb first. Once prefills compete with decode, p99 ITL and TPOT should spread; past the operating point, SLO attainment and goodput fall even if raw throughput keeps rising. An open-loop client sends requests according to preassigned timestamps. A closed-loop client that waits for each response before sending the next request hides that queue transition.

MLPerf's Server/Interactive rules make this workload distinction explicit: its load generator sends queries with a Poisson arrival process, tests multiple offered QPS values, and reports the maximum supported rate under scenario latency and quality requirements. It also samples the performance trace from a fixed library with a deterministic seed.^{[9]Reference 9MLPerf Inference Benchmark Rules.https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc} Use that discipline for an API benchmark without claiming MLPerf compliance: replay a public or production trace, or publish a fixed-seed distribution with the realized prompt and output lengths.

Freeze the contract before tuning. Otherwise a scheduler change can quietly change the workload it's being judged on:

Contract item	Record before the sweep
Endpoint and system	Endpoint URL and protocol, streaming mode, model and tokenizer revision, engine commit, GPU model/count/topology, driver/runtime, and precision or quantization
Workload	A replayable public/production trace or fixed-seed distribution; prompt-length and output-length buckets, sampling and stop rules, and the exact request order
Arrival and concurrency	Open-loop arrival timestamps or the named arrival process at each offered $\lambda$ ; client concurrency, max in-flight requests, and observed server concurrency
Warmup and window	Warmup requests or duration until compilation, allocators, and chosen caches are steady; measurement duration, drain time, and the samples excluded from warmup
Baseline	The exact scheduler/engine configuration being compared, with equivalent tuning and the same trace, system, and SLOs
Correctness	Deterministic or fixed-seed decoding where available, output validity, reference outputs or task-quality checks, token-count checks, and error, cancel, and timeout counts
Metrics and operating point	Queue wait, TTFT, per-token ITL, TPOT, raw throughput, p50 and p99 for each latency signal, KV pressure, preemptions, SLO attainment, and goodput; report the highest $\lambda$ that passes every latency and correctness gate plus its neighboring pass/fail points

One synthetic fixture is enough to expose why one uniform batch is weak: 60% short requests with 128-256 prompt tokens and 16-64 output tokens, 30% medium requests with 512-2,048 prompt tokens and 128-512 output tokens, and 10% long requests with 2,048-4,096 prompt tokens and 512-2,048 output tokens. Those proportions are a test fixture, not a claim about production traffic. A public trace or measured workload can replace it, but keep the realized lengths and arrival timestamps so every scheduler sees the same work.

For streaming, p50 alone can hide a stutter. Etalon shows why TTFT, TPOT, and related averages don't fully capture user-facing behavior, and proposes a fluidity-index for that gap.^{[10]Reference 10Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems.https://arxiv.org/abs/2407.07000} You don't have to adopt its metric, but you do need an explicit ITL aggregation and a p99 report if smooth token delivery is an SLO.

Read the curve with the trace. Stable p50 with rising p99 points to queue variance or starvation. Rising TTFT with stable ITL points to admission or prefill wait. Rising p99 ITL with stable TTFT points to prefill interference or an oversized mixed step. Rising raw throughput with flat or falling goodput means the system has passed its SLO-qualified operating point.

With that contract in place, chunked prefill is the usual next knob when colocated prefill still stalls decode.

Chunked prefill keeps streams moving

Naive continuous batching still lets a 4,000-token test log monopolize a step. Sarathi addresses that with chunked-prefills plus decode-maximal batching: split the prompt into equal-sized chunks, then pair one chunk with as many decode requests as the batch can hold. The chunk keeps the GPU's compute units busy. The decodes "piggyback" on the same step, which is the paper's term for that mix.^{[11]Reference 11Sarathi-Serve: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills.https://arxiv.org/abs/2308.16369}

Without chunking, a 4k test-log prefill occupies four scheduling steps while a live completion waits. Chunked prefill splits that prompt so the live stream keeps decoding on every step. — Left, the 4k log owns four steps and the live completion waits. Right, each step mixes a prompt chunk with decode, so tokens keep appearing on screen.

Without chunking, concurrent decode requests wait for the whole prompt. Average TBT can still look fine while the stream hitch is obvious in the UI.

With a 512-token chunk, that 4,000-token log becomes eight pieces (seven 512-token chunks plus a 416-token tail). The scheduler can pair an admitted chunk with decode work that fits the same token budget, then reconsider the mix before the next chunk. The long prompt pays extra scheduling and kernel boundaries. One admitted chunk no longer consumes the entire prompt in one step.

vLLM V1 documents chunked prefill as enabled by default whenever possible: decode requests go first, leftover max_num_batched_tokens budget goes to prefills, and a prefill that doesn't fit is chunked. Larger token budgets tend to help TTFT and throughput. Smaller budgets tend to help ITL.^{[5]Reference 5Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html}

chunked-prefill-plan.py

prompt_tokens = 4000
chunk_size = 512
chunks = []
remaining = prompt_tokens
while remaining:
    current = min(chunk_size, remaining)
    chunks.append(current)
    remaining -= current

print(f"chunk count: {len(chunks)}")
print(f"first/last chunk: {chunks[0]}/{chunks[-1]} tokens")
print(f"decode scheduling boundaries: {len(chunks) - 1}")

Output

chunk count: 8
first/last chunk: 512/416 tokens
decode scheduling boundaries: 7

Chunking still shares one GPU between phases. When prefill and decode want different hardware, the next move is to split the pools.

Disaggregated prefill and decode

Disaggregated serving runs prefill and decode on separate worker pools. Splitwise^{[12]Reference 12Splitwise: Efficient Generative LLM Inference Using Phase Splitting.https://arxiv.org/abs/2311.18677} and DistServe^{[4]Reference 4DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670} study this split directly: prefill often wants dense matrix throughput, decode often wants KV capacity and memory bandwidth.

It isn't the default for every cluster. Short prompts, small fleets, or high prefix-cache hit rates can make "prefill on the decode worker" simpler than a KV hop. Compare goodput under defined TTFT, TPOT, and ITL objectives, including transfer overhead.

Co-located serving runs a long prefill on the same GPU pool so live streams wait, while disaggregated serving keeps prefill and decode on separate pools and hands off about 0.25 GiB of KV state for a 4,096-token Qwen3.6-27B prompt. — Co-located serving makes TTFT and TPOT share SM time. Disaggregation specializes the pools, then pays a KV handoff. For this 4,096-token Qwen3.6-27B prompt that handoff is about 0.25 GiB before protocol overhead.

Prefill workers can be sized for matrix-math throughput. Decode workers can be sized for KV capacity and bandwidth. You scale the two pools independently when traffic skews toward long prompts or long generations.

The cost is the KV transfer. DistServe is a reminder that once you optimize for goodput instead of raw TPS, paying that hop can still win.^{[4]Reference 4DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670}

kv-handoff-cost.py

layers, kv_heads, head_dim, bytes_per_value = 16, 4, 256, 2
prompt_tokens = 4096
link_gib_per_second = 50
kv_bytes = 2 * layers * kv_heads * head_dim * bytes_per_value * prompt_tokens
transfer_ms = kv_bytes / (link_gib_per_second * 1024**3) * 1000

print(f"KV handoff size: {kv_bytes / 1024**3:.2f} GiB")
print(f"idealized transfer time at {link_gib_per_second} GiB/s: {transfer_ms:.1f} ms")
print("production check: add protocol, queueing, and synchronization overhead")

Output

KV handoff size: 0.25 GiB
idealized transfer time at 50 GiB/s: 5.0 ms
production check: add protocol, queueing, and synchronization overhead

Read the failure signature

A scheduler usually contradicts itself in its trace before it fails in a headline throughput number. If finished requests keep their seats, you're still treating the batch as one request. Retire work at the iteration boundary, free its KV, and refill only after budget checks.

If twenty short prompts fit at admission but generation later OOMs, request count hid KV growth. Cap blocks or tokens and leave headroom for long outputs and bursts.

If throughput looks healthy while new prompts wait past their TTFT target, decode preference has become starvation. Add a waiting-time timeout or decode-only cap, then admit or preempt work before the deadline.

A chat stream and a background summarizer can have different targets, but neither target is served by a chart that omits queue wait.

The thread to keep

Static batching leaves holes when lengths differ. Continuous batching closes those holes only when the scheduler can allocate KV flexibly, reserve token headroom, and rebuild the active set at each iteration.

When prefill and decode interfere, chunking bounds each prefill slice. When their hardware needs diverge, disaggregation trades a KV handoff for separate pools.

Judge each step on SLO-qualified goodput under the traffic you actually serve, not raw tokens/sec.

That is the progression: request-level batches, iteration-level refills, bounded prefill, then phase-level separation when the measured curve justifies its transfer cost.

Run a mixed-load trace

Replay the same mixed trace across scheduler settings. Warm up until the endpoint reaches its declared steady state, send requests with the chosen open-loop arrival process, and drain every response before ending the run. Record per-request queue wait, TTFT, ITL, output length, correctness, token-budget use, KV-block pressure, preemptions, and errors.

Plot offered $\lambda$ against p50 and p99 latency, raw throughput, SLO attainment, and goodput. Compare each curve with the policy you think you configured, then keep the trace and pass/fail operating points as the acceptance artifact for the next scheduler change.

Next Step

Continue to Scaling LLM Inference

You can now rebuild a batch at every decode step under KV and token budgets. Scaling inference asks the capacity question under that loop: when weights, KV state, and concurrency compete for HBM, what actually limits tokens per second and cost.

PreviousFlashAttention & Memory Efficiency

Share this article