Deep Dive - vLLM

R0 is halfway through a response when R1 arrives with a nine-token prompt. R0 needs one decode step, producing its next token. R1 needs its prompt processed in prefill. If R1 consumes the whole next step, R0's next token waits. If every request reserves its maximum attention state, short responses leave holes that can't serve another request. One serving decision has to keep both GPU work and growing memory moving.

vLLM is built around that conflict. The public API can look like an OpenAI-compatible endpoint. Under that HTTP surface, the engine still schedules tokens, maps each logical sequence onto physical Key and Value (KV) blocks, chooses a model runner and attention kernel, and streams the result back.

Hold R0 and R1 as the pair. Time to first token (TTFT) is the wait for the first generated token, inter-token latency (ITL) is the gap between later tokens, and the KV cache is the per-request attention state that saves recomputation during decoding.

Two entry paths, one engine loop

vLLM has two public fronts. The LLM Python class offers synchronous offline inference. That doesn't mean its engine and workers all run in the caller's process. The online server accepts requests and streams responses.

The route family includes chat completions, completions, embeddings, scoring, and reranking. A deployment's model and runner still decide which of those tasks it can serve. A generation endpoint isn't automatically an embedding service. Start the server with vllm serve. In v0.30.0, the older python -m vllm.entrypoints.openai.api_server entrypoint is a deprecated compatibility shim. The v0.28.0 file below still contains the server implementation.^{[1]Reference 1vLLM Architecture Overviewhttps://docs.vllm.ai/en/stable/design/arch_overview/[2]Reference 2Online Servinghttps://docs.vllm.ai/en/stable/serving/online_serving/[3]Reference 3vLLM v0.30.0: Legacy API Server Compatibility Shimhttps://github.com/vllm-project/vllm/blob/v0.30.0/vllm/entrypoints/openai/api_server.py}

The reproducible trace uses v0.28.0, released August 26, 2026. v0.31.0, released October 5, is the latest release checked October 10, 2026. Keep the older tag for this reproducible trace; use the installed revision for deployment decisions. The worked examples are CPU-only accounting models, not vLLM executions or performance measurements. They assume a decoder-only, full-attention model with no speculative decoding, cache offload, or hybrid-state cache groups.^{[4]Reference 4vLLM v0.28.0 Releasehttps://github.com/vllm-project/vllm/releases/tag/v0.28.0[5]Reference 5vLLM v0.31.0 Releasehttps://github.com/vllm-project/vllm/releases/tag/v0.31.0}

One consequential v0.30.0 change: scale-out endpoints on ordinary vllm serve are opt-in through --enable-scale-out, replacing VLLM_ENABLE_SCALE_OUT_ENDPOINTS. A copied environment-variable recipe can therefore leave required routes absent. Check registered routes as well as resolved scheduler settings.^{[6]Reference 6vLLM v0.30.0 Releasehttps://github.com/vllm-project/vllm/releases/tag/v0.30.0[2]Reference 2Online Servinghttps://docs.vllm.ai/en/stable/serving/online_serving/}

The same scheduler and KV pool sit under several request shapes:

The last row belongs to a later project: a trainer will keep calling this engine while the sampler's weights change. The holes in the opening come first. Why does reserving each request's maximum KV size leave memory that another request can't use?

The historical idea: page KV memory

Autoregressive generation appends one token at a time. In a full-attention Transformer, each new position attends to the Key and Value vectors of earlier positions. Reserving each request's maximum possible KV size wastes space inside its allocation when it finishes early: internal fragmentation. Allocating differently sized contiguous regions can also leave gaps between allocations that can't satisfy a larger request: external fragmentation. Both reduce the number of requests that fit.

PagedAttention borrowed a familiar operating-system idea: split each sequence's KV state into fixed-size blocks and keep a logical block table. A logical sequence can grow from block 0 to block 1 without owning adjacent physical addresses. A kernel follows the table to load the right physical blocks when it computes attention.^{[7]Reference 7Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}

Use a four-token block for arithmetic, not as a recommended GPU-backend setting. The letters stand for exact token IDs. R0 has already processed A B C D E F G H:

Request code sees one sequence. The GPU sees two physical block IDs. When the sequence grew past four tokens, vLLM could allocate any free physical block instead of searching for four adjacent cache rows. The table is the indirection layer that makes fragmentation manageable.

R1 now arrives with prompt A B C D E F X Y Z. The first four tokens match R0 exactly, so both requests can point at physical slot 12. The suffixes diverge, so they need distinct blocks. R1's ninth token Z sits in a partial third block that isn't full yet. Unreferenced blocks become candidates for least-recently-used (LRU) eviction rather than displacing either request's live prefix.

View larger

The paper reported 2x to 4x higher throughput than FasterTransformer and Orca at comparable latency.^{[7]Reference 7Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} Those results belong to its benchmark setup. A current comparison should match hardware, workload, software versions, and latency targets:

The paper used one-hour traces for most experiments and 15-minute traces for OPT-175B. That detail matters: a short run can miss queue growth and cache churn. The lasting contribution is the memory layout and its scheduling consequences, not an unconditional speedup for every current model, GPU, or vLLM release.^{[7]Reference 7Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}

PagedAttention isn't one current backend

The paper's name remains useful shorthand, but today's vLLM has a backend matrix. V1 model runners select attention implementations for the model, hardware, data type, and feature set. A backend can use paged KV addresses while relying on a fused attention kernel, a FlashAttention-family kernel, a Triton path, or a hardware plugin. Some model families, such as state-space or hybrid models, don't have the same KV semantics as a decoder-only Transformer.

The stable idea is the logical-to-physical contract. To learn which CUDA kernel a current request uses, inspect the V1 model runner and its attention backend registry, then the backend's block-table arguments and shape checks.

V1 request path: separate processes, one engine loop

V0 is fully deprecated. The rest of this walkthrough is V1.^{[8]Reference 8vLLM V1 User Guidehttps://docs.vllm.ai/en/stable/usage/v1_guide/}

V1 separates request handling from scheduling and GPU execution. The API server parses the request, tokenizes text, loads approved media, and streams results. It connects through ZeroMQ (ZMQ) to all engine-core processes, one per data-parallel (DP) rank. Each engine core owns a scheduler and KV-cache manager, then dispatches work to one worker process per GPU.^{[1]Reference 1vLLM Architecture Overviewhttps://docs.vllm.ai/en/stable/design/arch_overview/[8]Reference 8vLLM V1 User Guidehttps://docs.vllm.ai/en/stable/usage/v1_guide/}

With data parallelism, API-server count defaults to the DP size, and routing between API servers and engine cores is many-to-many. A coordinator process is added when DP > 1 so ranks can be balanced.^{[1]Reference 1vLLM Architecture Overviewhttps://docs.vllm.ai/en/stable/design/arch_overview/[8]Reference 8vLLM V1 User Guidehttps://docs.vllm.ai/en/stable/usage/v1_guide/}

Trace R1 across that split: the API server records arrival and tokenizes, an engine core queues the request, the scheduler assigns tokens only alongside enough KV blocks, GPU workers run prefill or decode and sample, and the result travels back for detokenization and streaming. TTFT spans those boundaries, so a slow first token isn't automatically a slow attention kernel.

View larger

For the online multiprocessing topology documented here, let A be API-server count, DP data-parallel size, and N total GPUs. The main serving processes total A + DP + N, plus one coordinator when DP > 1. This isn't a count of every OS process, nor a universal formula for offline, Ray, or externally load-balanced deployments.

This split makes failures easier to place. A slow tokenizer or media download is API-server work. Scheduler delay and block allocation are engine-core work. A kernel launch, CUDA graph, or tensor-parallel collective is worker work. CPU limits matter because every process participates in the hot path. V1's optimization guide calls out engine-core scheduling latency and worker CPU resources as throughput factors.^{[9]Reference 9Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html}

R0 and R1 don't care which process owns them until something stalls. The next question is how the engine core decides how much of each request runs in the next model step.

The V1 scheduler: one token budget

V1 keeps prefill and decode in one accounting system instead of giving them unrelated queues. It tracks how many tokens each request still needs and schedules against a fixed token budget. A decode request may need one new token. A new prompt may need hundreds, but chunked prefill lets the scheduler admit only a slice. The same allocation interface can therefore combine decode, prefill, prefix hits, and speculative work.^{[8]Reference 8vLLM V1 User Guidehttps://docs.vllm.ai/en/stable/usage/v1_guide/}

When V1 can use chunked prefill, it's enabled by default. The optimization guide describes its policy as decode-first, with remaining capacity available for prefill chunks. The code uses unified token accounting across running and waiting requests. KV availability, scheduling policy, and model-specific constraints can prevent a request from running even when token budget remains.^{[9]Reference 9Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html}

Predict the first step before reading the trace. With max_num_batched_tokens = 6, R0 needing one decode token and R1 needing nine prefill tokens, which request should consume the first slot, and how many R1 tokens fit after it? Treat APC as off for a moment, so R1's whole prompt still needs prefill:

The budget counts input positions computed, not output tokens emitted. Step 1 finishes R1's prefill and produces the logits used to sample its first output token. Step 2 feeds that sampled token back through the model to produce the next one. Thus R1 decode = 0 in step 1 doesn't mean R1 emits nothing.

These counts assume sufficient KV capacity and two requests that remain active. The script models decode-first selection followed by one prefill slice; it omits priorities, speculative tokens, asynchronous scheduling, and preemption.

View larger

budget = 6
r1_prefill_left = 9
trace = []

for step in range(3):
    left = budget
    r0_decode = min(1, left)
    left -= r0_decode
    if r1_prefill_left > 0:
        r1_prefill = min(r1_prefill_left, left)
        r1_prefill_left -= r1_prefill
        r1_decode = 0
    else:
        r1_prefill = 0
        r1_decode = min(1, left)
    used = r0_decode + r1_prefill + r1_decode
    row = {
        "step": step,
        "r0_decode": r0_decode,
        "r1_prefill": r1_prefill,
        "r1_decode": r1_decode,
        "used": used,
        "r1_prefill_left": r1_prefill_left,
    }
    trace.append(row)
    print(
        f"step {step}: R0 decode {r0_decode}, "
        f"R1 prefill {r1_prefill}, R1 decode {r1_decode}, "
        f"used {used}/{budget}, R1 prefill left {r1_prefill_left}"
    )

assert [row["r0_decode"] for row in trace] == [1, 1, 1]
assert [row["r1_prefill"] for row in trace] == [5, 4, 0]
assert [row["r1_decode"] for row in trace] == [0, 0, 1]
assert [row["used"] for row in trace] == [6, 5, 2]

step 0: R0 decode 1, R1 prefill 5, R1 decode 0, used 6/6, R1 prefill left 4
step 1: R0 decode 1, R1 prefill 4, R1 decode 0, used 5/6, R1 prefill left 0
step 2: R0 decode 1, R1 prefill 0, R1 decode 1, used 2/6, R1 prefill left 0

Continuous batching and chunked prefill

Continuous batching means a request can join or leave the active set between model steps. The batch isn't a fixed list of sequences built once at the beginning.

Tune max_num_batched_tokens against the service SLO. Smaller caps can protect ITL by limiting prefill work beside live decodes. Larger caps can finish prompts in fewer steps, but longer steps and queueing can erase that TTFT benefit. The optimization guide suggests trying values above 8192 for smaller models on large GPUs; v0.28.0 also raised a default token-budget setting to 16384. Neither number is a workload-independent optimum. Record the resolved configuration rather than assuming a default applies to every entrypoint and device.^{[9]Reference 9Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html[4]Reference 4vLLM v0.28.0 Releasehttps://github.com/vllm-project/vllm/releases/tag/v0.28.0}

Code-completion services with strict ITL may prefer a smaller cap. Batch document summarizers may prefer larger prefill slices. Frequent preemptions call for more KV capacity or fewer active sequences alongside any token-cap change.

In v0.31.0, --max-num-active-seqs separately caps the number of requests admitted to the RUNNING set. For example, with max_num_seqs=256 and max_num_active_seqs=64, spare token budget doesn't admit a 65th running request. This can limit live KV growth while retaining the larger execution capacity. The cap affects running requests; measure queue wait and preemptions to assess its effect on latency.^{[5]Reference 5vLLM v0.31.0 Releasehttps://github.com/vllm-project/vllm/releases/tag/v0.31.0}

Measure useful requests and token throughput

Raw output tokens per second can rise while a streaming service feels worse. Measure goodput as successful, SLO-compliant completions divided by the measurement duration. If you instead use offered rate times a pass fraction, include all offered requests in that fraction. Otherwise errors and unfinished requests disappear from the denominator.

Choose the decode promise explicitly: this example uses each request's p99 ITL, while DistServe uses request-mean time per output token (TPOT). Those aren't interchangeable. Here every request asks for at least two output tokens; a real evaluator also needs a stated policy for requests with no inter-token gaps.^{[10]Reference 10DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670}

Predict before reading the table: which candidate serves more useful requests when the product requires TTFT under 300 ms and p99 ITL under 50 ms?

The counts below are synthetic values for a 100-second window. Count only passing completions inside that window; report arrivals, failures, and outstanding requests separately. A vLLM load test should include warmup and steady-state windows so boundary effects don't dominate.

This CPU check exposes the denominator error. Candidate A's completion-only pass fraction looks like 91.7%, but multiplying it by its arrival rate invents almost four useful requests per second.

window_seconds = 100
counts = {"A": (1000, 600, 550), "B": (800, 780, 752)}
goodput = {}
for name, (offered, completed, passed) in counts.items():
    assert 0 <= passed <= completed <= offered
    goodput[name] = passed / window_seconds
    completion_only = (offered / window_seconds) * (passed / completed)
    print(f"{name}: goodput={goodput[name]:.2f}/s; "
          f"wrong denominator={completion_only:.2f}/s")
assert goodput["B"] > goodput["A"]
assert 0 / window_seconds == 0  # No passing completions means zero goodput.

A: goodput=5.50/s; wrong denominator=9.17/s
B: goodput=7.52/s; wrong denominator=7.71/s

Candidate B wins despite lower offered rate. Measure this curve at each load level, alongside p50 and p99 TTFT, ITL or TPOT, queue wait, completion errors, and KV usage. A larger token budget is a change worth keeping only when it raises goodput under the same workload and preserves output correctness.

When KV blocks run out, V1's default preemption mode is recompute. The engine releases a request's blocks, keeps request state, and schedules its prefix again when capacity returns; surviving cached blocks may still be reusable. V1 recompute differs from V0 CPU-swap preemption. Configured KV offloading is a separate feature. Monitor preemption-counter increases, KV usage, queue wait, TTFT, and ITL together.^{[9]Reference 9Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html[11]Reference 11KV Offloading Usage Guidehttps://docs.vllm.ai/en/latest/features/kv_offloading_usage/}

The scheduler spent two steps on R1's prompt. Did it need those steps? R0 had already computed A B C D.

Block pool, prefix cache, and eviction

The block pool is preallocated when the KV-cache manager starts. Each KVCacheBlock has an immutable physical ID, a hash once full, a reference count, and pointers for a doubly linked free queue. The cache maps block hashes to physical IDs. A request maps its own ID to the block IDs it currently uses.^{[12]Reference 12Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

The prefix hash is chained. Conceptually:

block_hash = H(parent_hash, block_tokens, extra_hashes)
extra_hashes = LoRA IDs + multimodal hashes + optional cache_salt

Including the parent distinguishes identical block tokens after different prefixes. Extra hashes distinguish a multimodal input, adapter, or trust group. This defines reuse identity; it doesn't make finite hashes mathematically collision-free. The documented default is SHA-256. Choosing a faster non-cryptographic hash changes the collision and multi-tenant risk tradeoff.^{[12]Reference 12Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

Serialization also matters. The v0.28.0 path concatenates extra values without source tags: a LoRA named tenant-A with no salt and a request with salt tenant-A and no LoRA can both produce the extra tuple ("tenant-A",). With matching parent and token IDs, the hash input itself is identical; stronger hashing can't repair that ambiguity. v0.31.0 tags these fields by source and includes the LoRA path. Treat adapter and tenant identity as part of the cache contract, rather than relying on text equality or SHA-256 alone.^{[13]Reference 13vLLM v0.28.0 KV Cache Utilitieshttps://github.com/vllm-project/vllm/blob/v0.28.0/vllm/v1/core/kv_cache_utils.py[5]Reference 5vLLM v0.31.0 Releasehttps://github.com/vllm-project/vllm/releases/tag/v0.31.0}

For the full-attention path here, Automatic Prefix Caching (APC) reuses only full blocks. R0's two cached blocks are A B C D in slot 12 and E F G H in slot 41. R1's prompt is A B C D E F X Y Z. The exact shared prefix is six tokens (A through F), but that's not one and a half cached blocks. Only the first four-token block is full and identical, so R1 can increment slot 12's reference count and must still prefill the remaining five tokens. A seven-token shared prefix would miss in the same way: the second block isn't an exact match, so it isn't reusable.^{[12]Reference 12Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

def reusable_tokens(shared, prompt_length, block_size=4):
    if any(type(count) is not int for count in (shared, prompt_length, block_size)):
        raise ValueError("Token counts and block size must be integers, not booleans")
    if block_size <= 0 or prompt_length <= 0 or not 0 <= shared <= prompt_length:
        raise ValueError("Require a nonempty prompt and valid shared-prefix length")
    # v0.28.0 full-attention path: leave a position for logits recomputation.
    limit = min(shared, prompt_length - 1)
    return (limit // block_size) * block_size

for shared, length in [(6, 9), (7, 9), (8, 9), (8, 8), (1, 1)]:
    skipped = reusable_tokens(shared, length)
    print(f"shared={shared}, prompt={length}: "
          f"skip {skipped}, compute {length - skipped}")

assert reusable_tokens(6, 9) == 4
assert reusable_tokens(8, 9) == 8
assert reusable_tokens(8, 8) == 4
assert reusable_tokens(1, 1) == 0
assert reusable_tokens(0, 9) == 0
for args in [(10, 9), (-1, 9), (0, 0), (6, 9, 0), (6.5, 9), (6, 9, True)]:
    try:
        reusable_tokens(*args)
    except ValueError:
        pass
    else:
        raise AssertionError(f"Invalid input accepted: {args}")

shared=6, prompt=9: skip 4, compute 5
shared=7, prompt=9: skip 4, compute 5
shared=8, prompt=9: skip 8, compute 1
shared=8, prompt=8: skip 4, compute 4
shared=1, prompt=1: skip 0, compute 1

If APC had been on in the earlier trace, R1 would have entered the scheduler with 5 tokens of prefill rather than 9. R0's one decode position plus those five positions would finish R1's prompt in step 0.

The shared=8, prompt=8 case is different. KV entries aren't the final logits needed to sample a response. In v0.28.0, get_computed_blocks limits a hit to at most prompt_length - 1; block alignment can therefore force the last whole block to be recomputed. The function models that full-attention path, assuming eligible cached blocks are present and hash context matches. It isn't a complete cache lookup for sliding-window, hybrid, offloaded, or prompt-logprob requests.^{[14]Reference 14vLLM v0.28.0: KV Cache Managerhttps://github.com/vllm-project/vllm/blob/v0.28.0/vllm/v1/core/kv_cache_manager.py}

Reference counts protect live requests. When a request finishes, cached blocks whose counts reach zero move to the tail of the free queue. In v0.28.0, non-cached blocks instead move to its head for earlier reuse. The next allocation pops the head and removes any cached hash before overwriting physical storage. Within the cached blocks, reverse-free ordering makes recently completed suffix blocks more likely to leave first because longer suffixes are less likely to be reused.^{[12]Reference 12Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/[15]Reference 15vLLM v0.28.0 KV Block Poolhttps://github.com/vllm-project/vllm/blob/v0.28.0/vllm/v1/core/block_pool.py}

This reduced pool starts from the figure's live slots, without its three spare slots. Try to allocate a fifth physical block before freeing anything. Then release R0 and ask which slot becomes reusable: 41, not the shared slot 12. The hash labels are symbolic fixture values, not production hash functions.

from collections import Counter, deque

tables = {"R0": [12, 41], "R1": [12, 73, 90]}
refs = Counter(slot for table in tables.values() for slot in table)
hashes = {12: "prefix-ABCD", 41: "prefix-ABCDEFGH", 73: "prefix-ABCDEFXY"}
free = deque()  # Slot 90 is live but partial, so it has no cache hash.

def allocate():
    if not free:
        raise MemoryError("No free KV block")
    slot = free.popleft()
    assert refs[slot] == 0
    hashes.pop(slot, None)  # Invalidate cache identity before overwriting storage.
    refs[slot] = 1
    return slot

def release(request_id):
    for slot in reversed(tables.pop(request_id)):
        refs[slot] -= 1
        assert refs[slot] >= 0
        if refs[slot] == 0:
            if slot in hashes:
                free.append(slot)
            else:
                free.appendleft(slot)

before = (dict(refs), dict(hashes), list(free))
try:
    allocate()
except MemoryError as error:
    print(error)
else:
    raise AssertionError("Allocation must fail while all four slots are live")
assert before == (dict(refs), dict(hashes), list(free))

release("R0")
assert refs[12] == 1 and list(free) == [41]
assert 41 in hashes  # Being free doesn't immediately erase reusable content.
tables["R2"] = [allocate()]
assert tables["R2"] == [41] and 41 not in hashes
assert refs == Counter(slot for table in tables.values() for slot in table)
print(f"R1 still owns {tables['R1']}; R2 reuses {tables['R2']}")
release("R1")
assert list(free) == [90, 73, 12]  # Partial slot first, then cached suffix/prefix.
assert refs[41] == 1  # R2's live slot remains unavailable.

No free KV block
R1 still owns [12, 73, 90]; R2 reuses [41]

This tests ownership and safe reuse, not vLLM's allocator. The real implementation also coordinates cache-hit touches, multiple cache groups, allocation failure, and scheduler preemption.

Cache salt changes reuse identity: matching text with different salts doesn't share prefix blocks. For tenant isolation, a trusted gateway should assign and enforce the salt for an authenticated trust group; arbitrary client-chosen public values aren't an authorization boundary. Use an unguessable salt where the threat is probing another group's cached prefixes. Salting reduces cross-group cache-timing exposure, but doesn't encrypt memory or replace access control.^{[12]Reference 12Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

From scheduler output to GPU kernels

The engine core emits a scheduler output containing request IDs, token counts, block tables, sampling metadata, and model-execution inputs. Treat that message as the handoff contract: it says which work is admitted and which physical KV blocks back it.

GPU workers receive the contract and the model runner prepares tensors for the selected model. The runner owns weight loading, input layout, CUDA graph capture or eager execution, forward calls, and sampling handoff.

At the attention boundary, a block-table entry is metadata, not a kernel. The runner supplies logical sequence lengths and physical block IDs; the attention backend turns them into addresses and loads the corresponding K/V data. At the model boundary, parallelism controls how weights and layers are distributed:

V1 exposes those costs. A larger TP group can reduce weight memory per GPU, leaving room for KV blocks, but decode steps pay communication costs. A larger PP group can fit the model while exposing bubbles at low concurrency. DP adds engine cores and routing. The simple replica description assumes no expert sharding across those ranks; combined DP and EP changes which weights are replicated. Benchmark each topology on target hardware because more GPUs can hurt small or low-concurrency workloads.^{[1]Reference 1vLLM Architecture Overviewhttps://docs.vllm.ai/en/stable/design/arch_overview/[9]Reference 9Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html}

The model runner is why a new family can land without a second engine. A shared configuration object carries model, scheduler, cache, and parallelism choices. Model implementations expose a common runner contract; backend registries pick hardware-specific kernels. Compatibility is still a matrix: architecture, dtype, quantization, attention backend, and feature flags can interact.

Current V1 docs list NVIDIA, AMD, Intel GPU, TPU, and CPU as functional hardware paths, with additional platforms through plugins. Kernel quality and feature coverage still vary by platform and dtype, so "supported" isn't the same as "already tuned for this model."^{[8]Reference 8vLLM V1 User Guidehttps://docs.vllm.ai/en/stable/usage/v1_guide/}

At v0.28.0, gpu_worker.py branches on self.use_v2_model_runner: the established runner is vllm/v1/worker/gpu_model_runner.py, and the modular runner is vllm/v1/worker/gpu/model_runner.py. Here V2 names a model runner inside the V1 engine, not a replacement V2 engine. Read the branch your configuration actually constructs.^{[4]Reference 4vLLM v0.28.0 Releasehttps://github.com/vllm-project/vllm/releases/tag/v0.28.0}

Kernel evidence needs a separate measurement. Warm up compilation and CUDA graphs, synchronize GPU timing, and hold model, shapes, batch, sequence lengths, dtype, block size, and sampling path constant. Separate launch time, KV-memory traffic, arithmetic, and collectives, then compare outputs against a reference. A busy GPU can still be waiting on the wrong memory path, while a quiet GPU can mean the scheduler never issued work.^{[9]Reference 9Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html[16]Reference 16Metricshttps://docs.vllm.ai/en/stable/design/metrics/}

APIs and the boundary of trust

The online server offers OpenAI-compatible JSON, but compatibility isn't the same as a complete security boundary. Current security docs describe --api-key (or VLLM_API_KEY) protection under /v1, /v2, /inference, and /cohere. Unprefixed aliases such as /score, /rerank, and SageMaker-style /invocations, plus health, metrics, and some operational routes, can remain unauthenticated on the same HTTP server. Protecting /v1/score doesn't protect /score. Put vLLM behind an authenticated gateway, restrict management routes, and expose only the registered API surface clients need.^{[17]Reference 17Securityhttps://docs.vllm.ai/en/stable/usage/security/[2]Reference 2Online Servinghttps://docs.vllm.ai/en/stable/serving/online_serving/}

For multimodal requests, media URLs are an input capability. Allowlist domains with --allowed-media-domains; retain download and decode limits, including VLLM_MAX_MEDIA_DOWNLOAD_SIZE_MB, VLLM_MAX_IMAGE_PIXELS, VLLM_MAX_AUDIO_CLIP_FILESIZE_MB, VLLM_MAX_AUDIO_DECODE_DURATION_S, and VLLM_MAX_AUDIO_DECODE_BYTES. Compressed-byte, duration, and decoded-byte caps protect different resources. An untrusted URL can target internal services (server-side request forgery, or SSRF), download a huge file, or expand compressed media into an out-of-memory event. Setting VLLM_MEDIA_URL_ALLOW_REDIRECTS=0 blocks redirect tricks that bypass the allowlist.^{[17]Reference 17Securityhttps://docs.vllm.ai/en/stable/usage/security/}

In v0.31.0, non-empty per-request mm_processor_kwargs and media_io_kwargs are rejected unless the server enables --trust-request-mm-kwargs. A client that previously changed image sizing or video sampling can now fail before inference. Keep deployment settings in server-level --mm-processor-kwargs and --media-io-kwargs; enable client overrides only for trusted callers. The offline LLM interface isn't subject to this API-server gate.^{[17]Reference 17Securityhttps://docs.vllm.ai/en/stable/usage/security/[5]Reference 5vLLM v0.31.0 Releasehttps://github.com/vllm-project/vllm/releases/tag/v0.31.0}

Multi-node communication is insecure by default. PyTorch distributed, KV transfer, and tensor, pipeline, or data-parallel links don't provide an authorization protocol or encryption suitable for an untrusted network. Isolate those ports, set explicit host addresses, and firewall internal interfaces. An API key on /v1/chat/completions can't protect a reachable process-group port.^{[17]Reference 17Securityhttps://docs.vllm.ai/en/stable/usage/security/}

Project identity

vLLM grew from UC Berkeley's Sky Computing Lab and the PagedAttention paper. Governance says committers earn authority through sustained contributions, reviews, and subsystem ownership. Companies can participate, but committer status belongs to individuals.^{[7]Reference 7Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180[18]Reference 18vLLM Governance Processhttps://github.com/vllm-project/vllm/blob/main/docs/governance/process.md}

The advertised compatibility surface is wide. In the checkout, that usually means a feature landed for one model, backend, or topology and can still be missing or untested for another. Read the feature matrix for the revision you run.

Strengths and weaknesses

Block-based KV, token scheduling, specialized kernels, and API processing fail at different boundaries. A prefix miss, a token-budget cap, and a slow collective can all lower tokens per second, so that one chart can't tell you which layer needs a change.

Code-reading route

Open the v0.28.0 tag in the public repository rather than treating moving main as the article's source snapshot. If you already have that checkout, these read-only commands locate the main boundaries without installing vLLM or loading weights:

git describe --tags --exact-match HEAD
rg -n 'def schedule|num_computed_tokens' vllm/v1/core/sched/scheduler.py
rg -n 'max_cache_hit_length|def allocate_slots' vllm/v1/core/kv_cache_manager.py
rg -n 'use_v2_model_runner|gpu.model_runner' vllm/v1/worker/gpu_worker.py

The first command should identify v0.28.0. A version mismatch means you must recheck the branch conditions and defaults before comparing behavior. These are source-navigation commands, not a server startup recipe.

Read these files in order, and keep R0's request ID in your head:

1. vllm/entrypoints/cli/main.py and vllm/entrypoints/openai/api_server.py: supported vllm serve CLI, HTTP parsing, input processing, and the streaming boundary.
2. vllm/v1/engine/core.py: engine-core loop and handoff to workers.
3. vllm/v1/core/sched/scheduler.py: token-budget scheduling, preemption, and scheduler output.
4. vllm/v1/core/kv_cache_manager.py: computed-block lookup, allocation, touch, and free operations.
5. vllm/v1/core/block_pool.py: physical block pool, free queue, hash map, and LRU behavior.
6. vllm/v1/worker/gpu_worker.py, then the constructed runner in gpu_model_runner.py or gpu/model_runner.py: input tensors, model execution, graphs, and output preparation.

While reading, write down three invariants:

• A request's logical token order is independent of physical block addresses.
• A cached block can be reused only when its full hash and extra context match.
• A worker can't execute tokens that the scheduler hasn't assigned and backed with block capacity.

A small failure drill

Suppose a dashboard shows rising ITL, low GPU utilization, and a growing preemption counter. Don't start by changing the attention kernel. Check, in order:

1. KV-cache usage and free-block count. If blocks are exhausted, recompute work may be stealing GPU time.
2. max_num_batched_tokens, max_num_seqs, and prompt-length distribution. An oversized active set can create a thrash loop.
3. Engine-core CPU saturation and scheduler latency. V1's process split can make CPU starvation look like a GPU problem.
4. TP and PP topology. A new parallelism size can add collective or pipeline wait.
5. Prefix-cache hit rate. A workload with unique prompts won't benefit from APC, no matter how large the cache is.

Prometheus metrics for KV usage, TTFT, and ITL live in the official metrics reference. Use those names rather than inventing dashboard fields.^{[16]Reference 16Metricshttps://docs.vllm.ai/en/stable/design/metrics/}

vllm:inter_token_latency_seconds measures gaps between streamed output events, which can bundle tokens. Retain request timelines to evaluate this lesson's per-request p99 ITL pass count; use the separate request-level TPOT metric when TPOT is the actual contract.^{[16]Reference 16Metricshttps://docs.vllm.ai/en/stable/design/metrics/}

For a request that never gets its first token, break TTFT into queue wait, scheduler delay, prefill compute, and worker launch time.

Keep one request ID through the client timeline, engine-core stats, KV events, and GPU trace. High queue time with short prefill points to admission or CPU scheduling. Short queue time with long prefill points to prompt work or its kernel path. Normal engine timings with late client output points outside the engine, such as detokenization or network delivery. That boundary test is more useful than replacing the attention backend on a low-utilization graph.

Research roots and current practice

The PagedAttention paper explains the original memory-management insight and its 2023 throughput evaluation.^{[7]Reference 7Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} The vLLM repository describes the practical engine that grew around it.^{[20]Reference 20vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttentionhttps://github.com/vllm-project/vllm} Current architecture and V1 guides document the process split and unified scheduler, while optimization and prefix-caching guides describe chunking, recompute, hashes, reference counts, and LRU behavior.^{[1]Reference 1vLLM Architecture Overviewhttps://docs.vllm.ai/en/stable/design/arch_overview/[8]Reference 8vLLM V1 User Guidehttps://docs.vllm.ai/en/stable/usage/v1_guide/[9]Reference 9Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html[12]Reference 12Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

Read those sources with dates in mind. vLLM V1 replaces older V0 assumptions, and backend support changes as models and accelerators change. A paper result can remain historically important without being a current service target. A documentation default can be correct for today's release and still need rechecking before a production rollout.

The guide still contains that restriction when checked October 10, 2026. Meanwhile, v0.30.0 advertises partial prefix checkpoints for a specific hybrid model, and v0.31.0 renames --enable-mamba-fine-grained-prefix-cache to --enable-mamba-shared-prefix-checkpoint. This is why the worked trace states its cache assumptions rather than promising universal APC behavior.^{[8]Reference 8vLLM V1 User Guidehttps://docs.vllm.ai/en/stable/usage/v1_guide/[4]Reference 4vLLM v0.28.0 Releasehttps://github.com/vllm-project/vllm/releases/tag/v0.28.0[6]Reference 6vLLM v0.30.0 Releasehttps://github.com/vllm-project/vllm/releases/tag/v0.30.0[5]Reference 5vLLM v0.31.0 Releasehttps://github.com/vllm-project/vllm/releases/tag/v0.31.0}

A wait on R0 can sit in the token budget, the free-block queue, or a process outside the GPU. Check those before you blame the attention kernel.

Architectural summary

• Serving is a joint scheduling and memory problem: GPU compute and KV blocks can stall independently.
• PagedAttention's logical-to-physical blocks reduce fragmentation. The name doesn't identify every current attention kernel.
• V1's API server, engine core, and GPU workers separate request I/O, scheduling, cache management, and execution.
• One token budget is how V1 mixes continuous batching, chunked prefill, and decode-first steps.
• APC hashes full blocks, protects live blocks with reference counts, and evicts zero-reference blocks through an LRU queue.
• TP, PP, DP, and EP solve different fit and throughput constraints, each with communication or bubble costs.
• Serving design must cover API keys, network isolation, media allowlists, and metrics.

Evaluation rubric

• Trace a request through API processing, engine-core scheduling, KV allocation, workers, and streaming without attributing every delay to attention.
• Compute the block-rounded APC hit, including complete-prompt recomputation, and explain why live reference counts prevent eviction.
• Read the pinned scheduler and runner branch before claiming a default, process topology, or kernel applies to your deployment.
• Compare configurations using passing completions per second, latency distributions, errors, and output correctness under the same workload.

Follow-up questions

The CPU examples establish arithmetic and ownership invariants only. They don't validate vLLM execution, GPU kernels, output parity, security enforcement, or throughput. A deployment review still needs the installed revision, resolved configuration, hardware, workload, correctness checks, and a measured serving trace.