Deep Dive - vLLM

R0 is halfway through a response when R1 arrives with a nine-token prompt. R0 needs one decode step, producing its next token; R1 needs its prompt processed in prefill. If R1 consumes the whole next step, R0's next token waits. If every request reserves its maximum attention state, short responses leave holes that can't serve another request. One serving decision has to keep both GPU work and growing memory moving.

vLLM is built around that conflict. Its public API can look like an OpenAI-compatible endpoint, but the work below HTTP schedules tokens, maps logical sequences to physical Key and Value (KV) blocks, chooses a model runner and attention kernel, and streams output back to the client.

Keep R0 and R1 in view as we trace the path. Time to first token (TTFT) is the wait for the first generated token, inter-token latency (ITL) is the gap between later tokens, and the KV cache is the per-request attention state that saves recomputation during decoding.

Before reading code, name the two resources vLLM must schedule together.

Answer

GPU compute and KV-cache memory. A request can fit in one resource and still stall in the other, so V1 schedules token work while reserving physical KV blocks.

Two entry paths, one engine loop

vLLM has two public fronts. The LLM Python class offers synchronous offline inference; that doesn't mean its engine and workers all run in the caller's process. The online server accepts requests and streams responses. Its route family includes chat completions, completions, embeddings, scoring, and reranking, but a deployment's model and runner determine which tasks it can actually serve. A generation endpoint isn't automatically an embedding service. The supported CLI is vllm serve; the older python -m vllm.entrypoints.openai.api_server entrypoint is deprecated.^{[1]Reference 1vLLM Architecture Overviewhttps://docs.vllm.ai/en/stable/design/arch_overview/[2]Reference 2Online Servinghttps://docs.vllm.ai/en/stable/serving/online_serving/}

The code-reading path below uses v0.28.0, released August 26, 2026. Living documentation was checked September 2, 2026. The worked examples are CPU-only accounting models, not vLLM executions or performance measurements. They assume a decoder-only, full-attention model with no speculative decoding, cache offload, or hybrid-state cache groups.^{[3]Reference 3vLLM v0.28.0 Releasehttps://github.com/vllm-project/vllm/releases/tag/v0.28.0}

The same scheduler and KV pool sit under several request shapes:

The last row is the next project's problem. A serving engine answers requests. SkyRL has to create, score, and learn from those requests while the sampler keeps changing.

The historical idea: page KV memory

Autoregressive generation appends one token at a time. In a full-attention Transformer, each new position attends to the Key and Value vectors of earlier positions. Reserving each request's maximum possible KV size wastes space inside its allocation when it finishes early: internal fragmentation. Allocating differently sized contiguous regions can also leave gaps between allocations that can't satisfy a larger request: external fragmentation. Both reduce the number of requests that fit.

PagedAttention borrowed a familiar operating-system idea: split each sequence's KV state into fixed-size blocks and keep a logical block table. A logical sequence can grow from block 0 to block 1 without owning adjacent physical addresses. A kernel follows the table to load the right physical blocks when it computes attention.^{[4]Reference 4Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}

Use a four-token block for arithmetic, not as a recommended GPU-backend setting. The letters stand for exact token IDs. R0 has already processed A B C D E F G H:

Request code sees one sequence. The GPU sees two physical block IDs. When the sequence grew past four tokens, vLLM could allocate any free physical block instead of searching for four adjacent cache rows. The table is the indirection layer that makes fragmentation manageable.

R1 now arrives with prompt A B C D E F X Y Z. The first four tokens match R0 exactly, so both requests can point at physical slot 12. The suffixes diverge, so they need distinct blocks. R1's ninth token Z sits in a partial third block that isn't full yet. Unreferenced blocks become candidates for least-recently-used (LRU) eviction rather than displacing either request's live prefix.

The paper's headline is 2x to 4x higher throughput than FasterTransformer and Orca at comparable latency.^{[4]Reference 4Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} Read that number as an experiment receipt, not a current service promise:

The paper used one-hour traces for most experiments and 15-minute traces for OPT-175B. That detail matters: a short run can miss queue growth and cache churn. The lasting contribution is the memory layout and its scheduling consequences, not an unconditional speedup for every current model, GPU, or vLLM release.^{[4]Reference 4Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}

PagedAttention isn't one current backend

The paper's name remains useful shorthand, but today's vLLM has a backend matrix. V1 model runners select attention implementations for the model, hardware, data type, and feature set. A backend can use paged KV addresses while relying on a fused attention kernel, a FlashAttention-family kernel, a Triton path, or a hardware plugin. Some model families, such as state-space or hybrid models, don't have the same KV semantics as a decoder-only Transformer.

The stable idea is the logical-to-physical contract. To learn which CUDA kernel a current request uses, inspect the V1 model runner and its attention backend registry, then the backend's block-table arguments and shape checks.

Why doesn't "vLLM uses PagedAttention" tell you which GPU kernel ran?

Answer

PagedAttention describes logical KV blocks and their physical mapping. V1 can select different fused or hardware-specific attention backends that consume that mapping, and some supported models use other state representations.

V1 request path: separate processes, one engine loop

V0 is fully deprecated. The rest of this walkthrough is V1.^{[5]Reference 5vLLM V1 User Guidehttps://docs.vllm.ai/en/stable/usage/v1_guide/}

V1 separates request handling from scheduling and GPU execution. The API server parses the request, tokenizes text, loads approved media, and streams results. It connects through ZeroMQ (ZMQ) to all engine-core processes, one per data-parallel (DP) rank. Each engine core owns a scheduler and KV-cache manager, then dispatches work to one worker process per GPU.^{[1]Reference 1vLLM Architecture Overviewhttps://docs.vllm.ai/en/stable/design/arch_overview/[5]Reference 5vLLM V1 User Guidehttps://docs.vllm.ai/en/stable/usage/v1_guide/}

With data parallelism, API-server count defaults to the DP size, and routing between API servers and engine cores is many-to-many. A coordinator process is added when DP > 1 so ranks can be balanced.^{[1]Reference 1vLLM Architecture Overviewhttps://docs.vllm.ai/en/stable/design/arch_overview/[5]Reference 5vLLM V1 User Guidehttps://docs.vllm.ai/en/stable/usage/v1_guide/}

Trace R1 across that split: the API server records arrival and tokenizes, an engine core queues the request, the scheduler assigns tokens only alongside enough KV blocks, GPU workers run prefill or decode and sample, and the result travels back for detokenization and streaming. TTFT spans those boundaries, so a slow first token isn't automatically a slow attention kernel.

For the online multiprocessing topology documented here, let A be API-server count, DP data-parallel size, and N total GPUs. The main serving processes total A + DP + N, plus one coordinator when DP > 1. This isn't a count of every OS process, nor a universal formula for offline, Ray, or externally load-balanced deployments.

This split makes failures easier to place. A slow tokenizer or media download is API-server work. Scheduler delay and block allocation are engine-core work. A kernel launch, CUDA graph, or tensor-parallel collective is worker work. CPU limits matter because every process participates in the hot path. V1's optimization guide calls out engine-core scheduling latency and worker CPU resources as throughput factors.^{[6]Reference 6Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html}

R0 and R1 don't care which process owns them until something stalls. The next question is how the engine core decides how much of each request runs in the next model step.

The V1 scheduler: one token budget

V1 keeps prefill and decode in one accounting system instead of giving them unrelated queues. It tracks how many tokens each request still needs and schedules against a fixed token budget. A decode request may need one new token. A new prompt may need hundreds, but chunked prefill lets the scheduler admit only a slice. The same allocation interface can therefore combine decode, prefill, prefix hits, and speculative work.^{[5]Reference 5vLLM V1 User Guidehttps://docs.vllm.ai/en/stable/usage/v1_guide/}

When V1 can use chunked prefill, it's enabled by default. The optimization guide describes its policy as decode-first, with remaining capacity available for prefill chunks. In the code, this is unified token accounting over running and waiting requests, not two independent schedulers. KV availability, scheduling policy, and model-specific constraints can prevent a request from running even when token budget remains.^{[6]Reference 6Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html}

Predict the first step before reading the trace. With max_num_batched_tokens = 6, R0 needing one decode token and R1 needing nine prefill tokens, which request should consume the first slot, and how many R1 tokens fit after it? Treat APC as off for a moment, so R1's whole prompt still needs prefill:

The budget counts input positions computed, not output tokens emitted. Step 1 finishes R1's prefill and produces the logits used to sample its first output token. Step 2 feeds that sampled token back through the model to produce the next one. Thus R1 decode = 0 in step 1 doesn't mean R1 emits nothing.

These counts assume sufficient KV capacity and two requests that remain active. The script models decode-first selection followed by one prefill slice; it omits priorities, speculative tokens, asynchronous scheduling, and preemption.

Continuous batching and chunked prefill

Continuous batching means a request can join or leave the active set between model steps. The batch isn't a fixed list of sequences built once at the beginning.

Tune max_num_batched_tokens against the service SLO. Smaller caps can protect ITL by limiting prefill work beside live decodes. Larger caps can finish prompts in fewer steps, but longer steps and queueing can erase that TTFT benefit. The optimization guide suggests trying values above 8192 for smaller models on large GPUs; v0.28.0 also raised a default token-budget setting to 16384. Neither number is a workload-independent optimum. Record the resolved configuration rather than assuming a default applies to every entrypoint and device.^{[6]Reference 6Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html[3]Reference 3vLLM v0.28.0 Releasehttps://github.com/vllm-project/vllm/releases/tag/v0.28.0}

Code-completion services with strict ITL may prefer a smaller cap. Batch document summarizers may prefer larger prefill slices. Frequent preemptions call for more KV capacity or fewer active sequences alongside any token-cap change.

Count useful requests, not only tokens

Raw output tokens per second can rise while a streaming service feels worse. Measure goodput as successful, SLO-compliant completions divided by the measurement duration. If you instead use offered rate times a pass fraction, that fraction must use all offered requests, not only successful completions. Otherwise errors and unfinished requests disappear from the denominator.

Choose the decode promise explicitly: this example uses each request's p99 ITL, while DistServe uses request-mean time per output token (TPOT). Those aren't interchangeable. Here every request asks for at least two output tokens; a real evaluator also needs a stated policy for requests with no inter-token gaps.^{[7]Reference 7DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670}

Predict before reading the table: which candidate serves more useful requests when the product requires TTFT under 300 ms and p99 ITL under 50 ms?

The counts below are a synthetic 100-second observation window, not a vLLM benchmark. Count only passing completions inside that window; report arrivals, failures, and outstanding requests separately. Use warmup and steady-state windows in a real load test so boundary effects don't dominate.

The following CPU check exposes the denominator error. Candidate A's completion-only pass fraction looks like 91.7%, but multiplying it by its arrival rate invents almost four useful requests per second.

Candidate B wins despite lower offered rate. Measure this curve at each load level, alongside p50 and p99 TTFT, ITL or TPOT, queue wait, completion errors, and KV usage. A larger token budget is a change worth keeping only when it raises goodput under the same workload and preserves output correctness.

What does chunked prefill protect, and what can it hurt?

Answer

It protects decode ITL by preventing one long prompt from consuming an entire step. It can increase TTFT for that prompt because prefill is spread across steps, and a cap that is too high can still let prompt work delay decodes.

When KV blocks run out, V1's default preemption mode is recompute. The engine releases a request's blocks, keeps request state, and schedules its prefix again when capacity returns; surviving cached blocks may still be reusable. The old V0 CPU-swap preemption mode isn't this path. Separately configured KV offloading is a different feature, not evidence that recompute preemption never occurs. Monitor preemption-counter increases, KV usage, queue wait, TTFT, and ITL together.^{[6]Reference 6Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html[8]Reference 8KV Offloading Usage Guidehttps://docs.vllm.ai/en/latest/features/kv_offloading_usage/}

The scheduler just spent two steps chewing R1's prompt. Did it need to? R0 already computed A B C D.

Block pool, prefix cache, and eviction

The block pool is preallocated when the KV-cache manager starts. Each KVCacheBlock has an immutable physical ID, a hash once full, a reference count, and pointers for a doubly linked free queue. The cache maps block hashes to physical IDs. A request maps its own ID to the block IDs it currently uses.^{[9]Reference 9Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

The prefix hash is chained. Conceptually:

Including the parent distinguishes identical block tokens after different prefixes. Extra hashes distinguish a multimodal input, adapter, or trust group. This defines reuse identity; it doesn't make finite hashes mathematically collision-free. The documented default is SHA-256. Choosing a faster non-cryptographic hash changes the collision and multi-tenant risk tradeoff.^{[9]Reference 9Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

Automatic Prefix Caching (APC) reuses only full blocks. R0's two cached blocks are A B C D in slot 12 and E F G H in slot 41. R1's prompt is A B C D E F X Y Z. The exact shared prefix is six tokens (A through F), but that's not one and a half cached blocks. Only the first four-token block is full and identical, so R1 can increment slot 12's reference count and must still prefill the remaining five tokens. A seven-token shared prefix would miss in the same way: the second block isn't an exact match, so it isn't reusable.^{[9]Reference 9Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

If APC had been on in the earlier trace, R1 would have entered the scheduler with 5 tokens of prefill rather than 9. R0's one decode position plus those five positions would finish R1's prompt in step 0.

The shared=8, prompt=8 case is different. KV entries aren't the final logits needed to sample a response. In v0.28.0, get_computed_blocks limits a hit to at most prompt_length - 1; block alignment can therefore force the last whole block to be recomputed. The function models that full-attention path, assuming eligible cached blocks are present and hash context matches. It isn't a complete cache lookup for sliding-window, hybrid, offloaded, or prompt-logprob requests.^{[10]Reference 10vLLM v0.28.0: KV Cache Managerhttps://github.com/vllm-project/vllm/blob/v0.28.0/vllm/v1/core/kv_cache_manager.py}

Reference counts protect live requests. When a request finishes, blocks with count zero move to the tail of the free queue. The next allocation pops the head, which is the least-recently-used (LRU) candidate. If that block is cached, vLLM removes its hash before reusing the physical storage. The reverse-free ordering makes recently completed suffix blocks more likely to leave first because longer suffixes are less likely to be reused.^{[9]Reference 9Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

The following reduced pool starts from the figure's live slots, without its three spare slots. Try to allocate a fifth physical block before freeing anything. Then release R0 and ask which slot becomes reusable: 41, not the shared slot 12. The hash labels are symbolic fixture values, not production hash functions.

This tests ownership and safe reuse, not vLLM's allocator. The real implementation also coordinates cache-hit touches, multiple cache groups, allocation failure, and scheduler preemption.

Cache salt changes reuse identity: matching text with different salts doesn't share prefix blocks. For tenant isolation, a trusted gateway should assign and enforce the salt for an authenticated trust group; arbitrary client-chosen public values aren't an authorization boundary. Use an unguessable salt where the threat is probing another group's cached prefixes. Salting reduces cross-group cache-timing exposure, but doesn't encrypt memory or replace access control.^{[9]Reference 9Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

Why is an exact-prefix APC hit a prefill optimization, not a decode optimization?

Answer

The hit skips computation for already processed prompt blocks. Every new output token still needs decode attention and sampling, so long generations with little shared prompt can see almost no APC gain.

From scheduler output to GPU kernels

The engine core emits a scheduler output containing request IDs, token counts, block tables, sampling metadata, and model-execution inputs. Treat that message as the handoff contract: it says which work is admitted and which physical KV blocks back it.

GPU workers receive the contract and the model runner prepares tensors for the selected model. The runner owns weight loading, input layout, CUDA graph capture or eager execution, forward calls, and sampling handoff.

At the attention boundary, a block-table entry is metadata, not a kernel. The runner supplies logical sequence lengths and physical block IDs; the attention backend turns them into addresses and loads the corresponding K/V data. At the model boundary, parallelism controls how weights and layers are distributed:

V1 exposes those costs. A larger TP group can reduce weight memory per GPU, leaving room for KV blocks, but decode steps pay communication costs. A larger PP group can fit the model while exposing bubbles at low concurrency. DP adds engine cores and routing. The simple replica description assumes no expert sharding across those ranks; combined DP and EP changes which weights are replicated. Benchmark each topology on target hardware because more GPUs can hurt small or low-concurrency workloads.^{[1]Reference 1vLLM Architecture Overviewhttps://docs.vllm.ai/en/stable/design/arch_overview/[6]Reference 6Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html}

The model runner is why a new family can land without a second engine. A shared configuration object carries model, scheduler, cache, and parallelism choices. Model implementations expose a common runner contract; backend registries pick hardware-specific kernels. Compatibility is still a matrix: architecture, dtype, quantization, attention backend, and feature flags can interact.

Current V1 docs list NVIDIA, AMD, Intel GPU, TPU, and CPU as functional hardware paths, with additional platforms through plugins. Kernel quality and feature coverage still vary by platform and dtype, so "supported" isn't the same as "already tuned for this model."^{[5]Reference 5vLLM V1 User Guidehttps://docs.vllm.ai/en/stable/usage/v1_guide/}

At v0.28.0, gpu_worker.py branches on self.use_v2_model_runner: the established runner is vllm/v1/worker/gpu_model_runner.py, and the modular runner is vllm/v1/worker/gpu/model_runner.py. Here V2 names a model runner inside the V1 engine, not a replacement V2 engine. Read the branch your configuration actually constructs.^{[3]Reference 3vLLM v0.28.0 Releasehttps://github.com/vllm-project/vllm/releases/tag/v0.28.0}

Kernel evidence needs a separate measurement. Warm up compilation and CUDA graphs, synchronize GPU timing, and hold model, shapes, batch, sequence lengths, dtype, block size, and sampling path constant. Separate launch time, KV-memory traffic, arithmetic, and collectives, then compare outputs against a reference. A busy GPU can still be waiting on the wrong memory path, while a quiet GPU can mean the scheduler never issued work.^{[6]Reference 6Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html[11]Reference 11Metricshttps://docs.vllm.ai/en/stable/design/metrics/}

APIs and the boundary of trust

The online server offers OpenAI-compatible JSON, but compatibility isn't the same as a complete security boundary. The --api-key flag (or VLLM_API_KEY) authenticates many /v1 inference routes, plus some /v2 and /inference paths. Health, metrics, SageMaker-style /invocations, pause/resume, and other operational routes can remain unauthenticated on the same HTTP server. Put vLLM behind an authenticated gateway, restrict management routes, and expose only the API surface clients need.^{[12]Reference 12Securityhttps://docs.vllm.ai/en/stable/usage/security/[2]Reference 2Online Servinghttps://docs.vllm.ai/en/stable/serving/online_serving/}

For multimodal requests, media URLs are an input capability. Allowlist domains with --allowed-media-domains and keep decode-size limits in place (VLLM_MAX_IMAGE_PIXELS, VLLM_MAX_AUDIO_CLIP_FILESIZE_MB, VLLM_MAX_AUDIO_DECODE_DURATION_S). Otherwise an untrusted URL can target internal services (server-side request forgery, or SSRF), download a huge file, or expand compressed media into an out-of-memory event. Setting VLLM_MEDIA_URL_ALLOW_REDIRECTS=0 blocks redirect tricks that bypass the allowlist.^{[12]Reference 12Securityhttps://docs.vllm.ai/en/stable/usage/security/}

Multi-node communication is insecure by default. PyTorch distributed, KV transfer, and tensor, pipeline, or data-parallel links don't provide an authorization protocol or encryption suitable for an untrusted network. Isolate those ports, set explicit host addresses, and firewall internal interfaces. An API key on /v1/chat/completions can't protect a reachable process-group port.^{[12]Reference 12Securityhttps://docs.vllm.ai/en/stable/usage/security/}

Project identity

vLLM grew from UC Berkeley's Sky Computing Lab and the PagedAttention paper. Governance says committers earn authority through sustained contributions, reviews, and subsystem ownership. Companies can participate, but committer status belongs to individuals.^{[4]Reference 4Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180[13]Reference 13vLLM Governance Processhttps://github.com/vllm-project/vllm/blob/main/docs/governance/process.md}

The advertised compatibility surface is wide. In the checkout, that usually means a feature landed for one model, backend, or topology and can still be missing or untested for another. Read the feature matrix for the revision you run.

Strengths and weaknesses

Block-based KV, token scheduling, specialized kernels, and API processing fail at different boundaries. A prefix miss, a token-budget cap, and a slow collective can all lower tokens per second, so that one chart can't tell you which layer needs a change.

Code-reading route

Open the v0.28.0 tag in the public repository rather than treating moving main as the article's source snapshot. If you already have that checkout, these read-only commands locate the main boundaries without installing vLLM or loading weights:

The first command should identify v0.28.0. A version mismatch means you must recheck the branch conditions and defaults before comparing behavior. These are source-navigation commands, not a server startup recipe.

Read these files in order, and keep R0's request ID in your head:

1. vllm/entrypoints/cli/main.py and vllm/entrypoints/openai/api_server.py: supported vllm serve CLI, HTTP parsing, input processing, and the streaming boundary.
2. vllm/v1/engine/core.py: engine-core loop and handoff to workers.
3. vllm/v1/core/sched/scheduler.py: token-budget scheduling, preemption, and scheduler output.
4. vllm/v1/core/kv_cache_manager.py: computed-block lookup, allocation, touch, and free operations.
5. vllm/v1/core/block_pool.py: physical block pool, free queue, hash map, and LRU behavior.
6. vllm/v1/worker/gpu_worker.py, then the constructed runner in gpu_model_runner.py or gpu/model_runner.py: input tensors, model execution, graphs, and output preparation.

While reading, write down three invariants:

• A request's logical token order is independent of physical block addresses.
• A cached block can be reused only when its full hash and extra context match.
• A worker can't execute tokens that the scheduler hasn't assigned and backed with block capacity.

A small failure drill

Suppose a dashboard shows rising ITL, low GPU utilization, and a growing preemption counter. Don't start by changing the attention kernel. Check, in order:

1. KV-cache usage and free-block count. If blocks are exhausted, recompute work may be stealing GPU time.
2. max_num_batched_tokens, max_num_seqs, and prompt-length distribution. An oversized active set can create a thrash loop.
3. Engine-core CPU saturation and scheduler latency. V1's process split can make CPU starvation look like a GPU problem.
4. TP and PP topology. A new parallelism size can add collective or pipeline wait.
5. Prefix-cache hit rate. A workload with unique prompts won't benefit from APC, no matter how large the cache is.

Prometheus metrics for KV usage, TTFT, and ITL live in the official metrics reference. Use those names rather than inventing dashboard fields.^{[11]Reference 11Metricshttps://docs.vllm.ai/en/stable/design/metrics/}

Which symptom points most directly to KV pressure: a low APC hit rate or repeated recompute preemptions?

Answer

Repeated recompute preemptions. A low APC hit rate says prefixes aren't reusable, but it doesn't prove the cache is full. Preemption counters and free-block telemetry point to capacity pressure.

For a request that never gets its first token, break TTFT into queue wait, scheduler delay, prefill compute, and worker launch time.

Keep one request ID through the client timeline, engine-core stats, KV events, and GPU trace. High queue time with short prefill points to admission or CPU scheduling. Short queue time with long prefill points to prompt work or its kernel path. Normal engine timings with late client output points outside the engine, such as detokenization or network delivery. That boundary test is more useful than replacing the attention backend on a low-utilization graph.

Research roots and current practice

The PagedAttention paper explains the original memory-management insight and its 2023 throughput evaluation.^{[4]Reference 4Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} The vLLM repository describes the practical engine that grew around it.^{[15]Reference 15vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttentionhttps://github.com/vllm-project/vllm} Current architecture and V1 guides document the process split and unified scheduler, while optimization and prefix-caching guides describe chunking, recompute, hashes, reference counts, and LRU behavior.^{[1]Reference 1vLLM Architecture Overviewhttps://docs.vllm.ai/en/stable/design/arch_overview/[5]Reference 5vLLM V1 User Guidehttps://docs.vllm.ai/en/stable/usage/v1_guide/[6]Reference 6Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html[9]Reference 9Automatic Prefix Cachinghttps://docs.vllm.ai/en/stable/design/prefix_caching/}

Read those sources with dates in mind. vLLM V1 replaces older V0 assumptions, and backend support changes as models and accelerators change. A paper result can remain historically important without being a current service target. A documentation default can be correct for today's release and still need rechecking before a production rollout.

Architectural summary

• Serving is a joint scheduling and memory problem: GPU compute and KV blocks can stall independently.
• PagedAttention's logical-to-physical blocks reduce fragmentation. The name doesn't identify every current attention kernel.
• V1's API server, engine core, and GPU workers separate request I/O, scheduling, cache management, and execution.
• One token budget is how V1 mixes continuous batching, chunked prefill, and decode-first steps.
• APC hashes full blocks, protects live blocks with reference counts, and evicts zero-reference blocks through an LRU queue.
• TP, PP, DP, and EP solve different fit and throughput constraints, each with communication or bubble costs.
• Serving design must cover API keys, network isolation, media allowlists, and metrics.

Evaluation rubric

• Trace a request through API processing, engine-core scheduling, KV allocation, workers, and streaming without attributing every delay to attention.
• Compute the block-rounded APC hit, including complete-prompt recomputation, and explain why live reference counts prevent eviction.
• Read the pinned scheduler and runner branch before claiming a default, process topology, or kernel applies to your deployment.
• Compare configurations using passing completions per second, latency distributions, errors, and output correctness under the same workload.

Follow-up questions

A server has 98% prefix-cache hits but still misses its ITL target on long responses. What would you investigate next?

Answer

APC saves prompt computation, not the attention and sampling required for each new output token. Check active sequence lengths, KV capacity and preemptions, decode batch size, worker or collective time, and CPU scheduling. Separate prompt-cache success from decode performance before increasing cache capacity.

R1 supplies a cache salt equal to R0's tenant name. Does that prevent it from probing R0's cached prefixes?

Answer

No. A caller who can choose the same namespace can attempt the same prefix lookups. A trusted gateway must bind cache identity to authenticated tenancy and prevent callers from selecting another group's salt; unguessable salts help against probing but don't replace authorization.

The CPU examples establish arithmetic and ownership invariants only. They don't validate vLLM execution, GPU kernels, output parity, security enforcement, or throughput. A deployment review still needs the installed revision, resolved configuration, hardware, workload, correctness checks, and a measured serving trace.