Deep Dive - vLLM

When a chat service gets busy, GPU arithmetic usually isn't the first problem. Requests arrive with different prompt lengths, generate for different amounts of time, and hold different amounts of Key and Value (KV) state. A serving engine has to keep all those partial jobs moving without turning GPU memory into a fragmented heap.

vLLM is an open-source inference and serving engine built around that systems problem. Its public API can look like an OpenAI-compatible endpoint, but its interesting work happens below the HTTP boundary: scheduling tokens, mapping logical sequences to physical KV blocks, launching model runners, and returning streamed output. This lesson follows that path from request to kernel and back.

Use time to first token (TTFT), inter-token latency (ITL), and KV-cache occupancy to follow that path. TTFT measures the wait for the first generated token, ITL measures gaps between later tokens, and the KV cache holds per-request attention state that saves recomputation during decoding.

What vLLM is for

The project supports two broad entry paths. The LLM Python class runs offline batches in one process. The online server accepts requests, streams responses, and exposes OpenAI-compatible routes such as chat completions, completions, embeddings, scoring, and reranking.^{[1]Reference 1vLLM Architecture Overviewhttps://docs.vllm.ai/en/stable/design/arch_overview/[2]Reference 2Online Servinghttps://docs.vllm.ai/en/stable/serving/online_serving/}

That makes vLLM useful for several workloads:

Project identity

The project began in UC Berkeley's Sky Computing Lab around the PagedAttention paper. A broad open-source community now maintains it. Governance says committers earn authority through sustained contributions, reviews, and subsystem ownership; companies can participate, but committer status belongs to individuals.^{[3]Reference 3Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180[4]Reference 4vLLM Governance Processhttps://github.com/vllm-project/vllm/blob/main/docs/governance/process.md}

That distinction matters when you evaluate vLLM. Its compatibility surface is intentionally wide, while its design values put performance, ease of use, hardware coverage, production readiness, and extensibility side by side. A feature can help one deployment and still need a different backend, topology, or security boundary in another.

The historical idea: page KV memory

Autoregressive generation appends one token at a time. For every request, attention needs the Key and Value vectors for all earlier tokens. A naive allocator reserves one contiguous KV-cache region for each request's maximum length. Most requests finish early, so those reservations leave holes unavailable to a different-length request. The memory is free in aggregate, but not free in the shape the allocator needs.

PagedAttention borrowed a familiar operating-system idea: split each sequence's KV state into fixed-size blocks and keep a logical block table. A logical sequence can grow from block 0 to block 1 without owning adjacent physical addresses. A kernel follows the table to load the right physical blocks when it computes attention.^{[3]Reference 3Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180}

Consider block size four and token IDs A B C D E F G:

Request code sees one sequence. The GPU sees two physical block IDs. When the sequence grows, vLLM can allocate any free physical block instead of searching for four adjacent cache rows. The table is the indirection layer that makes fragmentation manageable.

The paper's 2023 experiments reported 2x to 4x throughput over FasterTransformer and Orca at comparable latency on the evaluated workloads.^{[3]Reference 3Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} Those numbers come from the paper's workloads and don't predict every model, GPU, or vLLM release. The lasting contribution is the memory layout and its scheduling consequences.

PagedAttention isn't one current backend

The paper's name remains useful shorthand, but today's vLLM has a backend matrix. V1 model runners select attention implementations for the model, hardware, data type, and feature set. A backend can use paged KV addresses while relying on a fused attention kernel, FlashAttention family kernel, Triton path, or a hardware plugin. Some model families, such as state-space or hybrid models, don't have the same KV semantics as a decoder-only Transformer.

The stable idea is the logical-to-physical contract. To learn which CUDA kernel a current request uses, inspect the V1 model runner and its attention backend registry, followed by the backend's block-table arguments and shape checks.

V1 request path: separate processes, one engine loop

V1 separates request handling from scheduling and GPU execution. The API server parses the request, tokenizes text, loads approved media, and streams results. It connects through ZMQ to all engine-core processes, one per data-parallel rank. Each engine core owns a scheduler and KV-cache manager, then dispatches work to one worker process per GPU. With data parallelism, API-server count defaults to the DP size and routing between API servers and engine cores is many-to-many; a conditional coordinator can balance ranks.^{[1]Reference 1vLLM Architecture Overviewhttps://docs.vllm.ai/en/stable/design/arch_overview/[6]Reference 6vLLM V1 User Guidehttps://docs.vllm.ai/en/stable/usage/v1_guide/}

The process-count formula is concrete. Let A be API-server count, DP data-parallel size, and N total GPUs. V1 starts A + DP + N, plus one coordinator when DP > 1. A four-GPU, single-replica deployment is one API server, one engine core, and four GPU workers: six processes. With TP=2 and DP=4 on eight GPUs, the documented example has four API servers, four engine cores, eight workers, and one coordinator: 17 processes.^{[1]Reference 1vLLM Architecture Overviewhttps://docs.vllm.ai/en/stable/design/arch_overview/}

This split makes failures easier to place. A slow tokenizer or media download is API-server work. Scheduler delay and block allocation are engine-core work. A kernel launch, CUDA graph, or tensor-parallel collective is worker work. CPU limits matter because every process participates in the hot path; V1's optimization guide calls out engine-core scheduling latency and worker CPU resources as throughput factors.^{[7]Reference 7Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html}

The V1 scheduler: one token budget

Older serving designs often described prefill and decode as separate queues. V1 represents both with a request-to-token count and schedules against a fixed token budget. A decode request may need one new token. A new prompt may need hundreds, but chunked prefill lets the scheduler admit only a slice. The same allocation interface can therefore combine decode, prefill, prefix hits, and speculative work.^{[6]Reference 6vLLM V1 User Guidehttps://docs.vllm.ai/en/stable/usage/v1_guide/}

Here is a small trace with max_num_batched_tokens = 6:

These counts illustrate the scheduler's token budget; they are not benchmark results. The script models decode-first selection followed by one prefill slice, so changing the budget shows which work remains.

Continuous batching and chunked prefill

Continuous batching means a request can join or leave the active set between model steps. The batch isn't a fixed list of sequences built once at the beginning. V1 prioritizes pending decodes, then fills unused budget with prefills. If a prompt doesn't fit, it gets split into chunks. Smaller budgets tend to protect ITL; larger budgets tend to improve TTFT by doing more prompt work per step.^{[7]Reference 7Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html}

Tune max_num_batched_tokens against the service SLO. Code-completion services with strict ITL may prefer a smaller cap, while batch document summarizers may prefer larger prefill slices. Frequent preemptions call for more KV capacity or fewer active sequences alongside any token-cap change.

When KV blocks run out, V1's default preemption mode is recompute. The engine releases a request's blocks, keeps enough request state to retry, and recomputes the prefix when capacity returns. Recompute avoids swap traffic in this architecture, but it adds latency and duplicated compute. Monitor cumulative preemptions, KV usage, queue wait, TTFT, and ITL together.^{[7]Reference 7Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html}

Block pool, prefix cache, and eviction

The block pool is preallocated when the KV-cache manager starts. Each KVCacheBlock has an immutable physical ID, a hash once full, a reference count, and pointers for a doubly linked free queue. The cache maps block hashes to physical IDs. A request maps its own ID to the block IDs it currently uses.^{[8]Reference 8Automatic Prefix Cachinghttps://docs.vllm.ai/en/latest/features/automatic_prefix_caching/}

The prefix hash is chained. Conceptually:

Including the parent means the same block tokens after different prefixes don't collide semantically. Including exact tokens reduces accidental matches. Extra hashes distinguish a multimodal input, adapter, or trust group.

Automatic Prefix Caching (APC) reuses only full blocks. Suppose block size is four and two requests share the first eight tokens. The second request can touch the first two cached blocks, increment their reference counts, and allocate fresh blocks for its nonmatching suffix. A seven-token shared prefix still hits only the first four-token block; the partially matching second block must be recomputed.^{[8]Reference 8Automatic Prefix Cachinghttps://docs.vllm.ai/en/latest/features/automatic_prefix_caching/}

Reference counts protect live requests. When a request finishes, blocks with count zero move to the tail of the free queue. The next allocation pops the head, which is the least-recently-used (LRU) candidate. If that block is cached, vLLM removes its hash before reusing the physical storage. The reverse-free ordering makes recently completed suffix blocks more likely to leave first because longer suffixes are less likely to be reused.^{[8]Reference 8Automatic Prefix Cachinghttps://docs.vllm.ai/en/latest/features/automatic_prefix_caching/}

Cache salt adds a trust boundary. A tenant can send a salt that participates in the first block hash, so another tenant with the same text but a different salt won't reuse its KV data. Salt doesn't encrypt memory or replace authorization; it reduces timing-based prefix-content inference when callers share an engine.^{[8]Reference 8Automatic Prefix Cachinghttps://docs.vllm.ai/en/latest/features/automatic_prefix_caching/}

From scheduler output to GPU kernels

The engine core emits a scheduler output containing request IDs, token counts, block tables, sampling metadata, and model-execution inputs. GPU workers receive that output and the model runner prepares tensors for the selected model. The runner owns weight loading, input layout, CUDA graph capture or eager execution, forward calls, and sampling handoff.

At the attention boundary, the runner supplies logical sequence lengths and physical block IDs. The backend turns those into memory addresses. At the model boundary, parallelism controls how weights and layers are distributed:

V1 exposes those costs. A larger TP group can leave more memory for KV blocks, but every decode step may pay all-reduce latency. A larger PP group can fit the model while exposing bubbles at low concurrency. DP adds engine cores and API servers. Benchmark each topology on target hardware because more GPUs can hurt small or low-concurrency workloads.^{[1]Reference 1vLLM Architecture Overviewhttps://docs.vllm.ai/en/stable/design/arch_overview/[7]Reference 7Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html}

The model runner also explains why vLLM can support many model families without copying an entire engine. A shared configuration object carries model, scheduler, cache, and parallelism choices. Model implementations expose a common runner contract while backend registries select hardware-specific kernels. That modularity is a strength for contributors, but compatibility is a matrix: model architecture, dtype, quantization, attention backend, and feature flags can interact.

APIs and the boundary of trust

The online server offers OpenAI-compatible JSON, but compatibility isn't the same as a complete security boundary. The documented API-key flag protects many /v1 inference routes, yet health, metrics, and other endpoints can remain unauthenticated. Put vLLM behind an authenticated gateway, restrict management routes, and expose only the API surface clients need.^{[9]Reference 9Securityhttps://docs.vllm.ai/en/stable/usage/security/[2]Reference 2Online Servinghttps://docs.vllm.ai/en/stable/serving/online_serving/}

For multimodal requests, media URLs are an input capability. Allowlist domains and cap decoded image pixels, audio size, and audio duration. Otherwise an untrusted URL can target internal services (server-side request forgery, or SSRF), download a huge file, or expand compressed media into an out-of-memory event.^{[9]Reference 9Securityhttps://docs.vllm.ai/en/stable/usage/security/}

Multi-node communication is insecure by default. PyTorch distributed, KV transfer, and tensor, pipeline, or data-parallel links don't provide an authorization protocol or encryption suitable for an untrusted network. Isolate those ports, set explicit host addresses, and firewall internal interfaces. An API key on /v1/chat/completions can't protect a reachable process-group port.^{[9]Reference 9Securityhttps://docs.vllm.ai/en/stable/usage/security/}

Strengths and weaknesses

vLLM combines block-based KV memory, token scheduling, and specialized kernels. Each layer has its own tuning and failure modes, so production debugging needs traces and metrics rather than one throughput number.

Code-reading route

Use the local repository as a map. Read these files in order, and keep one request ID in your head:

1. vllm/entrypoints/openai/api_server.py: HTTP request parsing, input processing, and streaming boundary.
2. vllm/v1/engine/core.py: engine-core loop and handoff to workers.
3. vllm/v1/core/sched/scheduler.py: token-budget scheduling, preemption, and scheduler output.
4. vllm/v1/core/kv_cache_manager.py: computed-block lookup, allocation, touch, and free operations.
5. vllm/v1/core/block_pool.py: physical block pool, free queue, hash map, and LRU behavior.
6. vllm/v1/worker/gpu_model_runner.py: input tensors, model execution, graphs, and output preparation.

While reading, write down three invariants:

• A request's logical token order is independent of physical block addresses.
• A cached block can be reused only when its full hash and extra context match.
• A worker can't execute tokens that the scheduler hasn't assigned and backed with block capacity.

A small failure drill

Suppose a dashboard shows rising ITL, low GPU utilization, and a growing preemption counter. Don't start by changing the attention kernel. Check, in order:

1. KV-cache usage and free-block count. If blocks are exhausted, recompute work may be stealing GPU time.
2. max_num_batched_tokens, max_num_seqs, and prompt-length distribution. An oversized active set can create a thrash loop.
3. Engine-core CPU saturation and scheduler latency. V1's process split can make CPU starvation look like a GPU problem.
4. TP and PP topology. A new parallelism size can add collective or pipeline wait.
5. Prefix-cache hit rate. A workload with unique prompts won't benefit from APC, no matter how large the cache is.

For a request that never gets its first token, break TTFT into queue wait, scheduler delay, prefill compute, and worker launch time.

Research roots and current practice

The PagedAttention paper explains the original memory-management insight and its 2023 throughput evaluation.^{[3]Reference 3Efficient Memory Management for Large Language Model Serving with PagedAttention.https://arxiv.org/abs/2309.06180} The vLLM repository describes the practical engine that grew around it.^{[10]Reference 10vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttentionhttps://github.com/vllm-project/vllm} Current architecture and V1 guides document the process split and unified scheduler, while optimization and prefix-caching guides describe chunking, recompute, hashes, reference counts, and LRU behavior.^{[1]Reference 1vLLM Architecture Overviewhttps://docs.vllm.ai/en/stable/design/arch_overview/[6]Reference 6vLLM V1 User Guidehttps://docs.vllm.ai/en/stable/usage/v1_guide/[7]Reference 7Optimization and Tuning.https://docs.vllm.ai/en/latest/configuration/optimization.html[8]Reference 8Automatic Prefix Cachinghttps://docs.vllm.ai/en/latest/features/automatic_prefix_caching/}

Read those sources with dates in mind. vLLM V1 replaces older V0 assumptions, and backend support changes as models and accelerators change. A paper result can remain historically important without being a current service target. A documentation default can be correct for today's release and still need rechecking before a production rollout.

Key takeaways

• vLLM treats serving as a joint scheduling and memory problem.
• PagedAttention's logical-to-physical blocks reduce fragmentation; it doesn't name every current attention kernel.
• V1's API server, engine core, and GPU workers separate request I/O, scheduling, cache management, and execution.
• One token budget enables continuous batching, chunked prefill, and shared scheduling decisions.
• APC hashes full blocks, protects live blocks with reference counts, and evicts zero-reference blocks through an LRU queue.
• TP, PP, DP, and EP solve different fit and throughput constraints, each with communication or bubble costs.
• Serving design must cover API keys, network isolation, media allowlists, and metrics.

Before moving on, explain one request from HTTP parse to streamed token without looking at the Mermaid diagram. Then explain what changes when its first eight tokens hit APC and what still has to run for every generated token.