Distributed Inference Data Plane

Request req-8k-017 contains 8,192 prompt tokens. One mixed prefill/decode replica can process it and start generating, but a busy fleet makes its long prefill interrupt several already-streaming replies. Splitting prompt work from token generation removes that interference, then creates a new problem: roughly 2.50 gibibytes (GiB) of key-value (KV) cache state must reach the right decoder before the first token can safely leave the service.

That state is much larger than a routing message, and it isn't interchangeable across model revisions or cache layouts. A fast transfer to the wrong worker is still a failed request. A correct transfer that starts after the decode pool fills is wasted prefill work.

The GPU serving and autoscaling lesson decided how many workers should be ready. Model parallelism mapped one model across ranks and physical links. KV Cache and PagedAttention showed how token state grows and how logical blocks map to rank-local memory. This lesson connects those pieces into a distributed request path with explicit owners at every boundary.

Why isn't a successful network copy enough to declare a KV handoff correct?

Answer

The decoder also needs the same request identity, model and tokenizer revisions, adapter, token positions, KV dtype and layout, layer coverage, and shard mapping. Bytes that arrive intact under the wrong interpretation can produce invalid output or a runtime failure.

One request, two planes

A distributed serving system has two kinds of work.

The control plane makes decisions. It authenticates the caller, applies quotas, discovers healthy workers, chooses a compatible prefill/decode pair, reserves capacity, issues a fenced attempt, and decides whether failed work may retry.

The data plane carries and transforms request data. It tokenizes or receives the canonical prompt tokens, runs prefill, moves the KV cache, runs decode, and streams output tokens. Control messages are usually small. This request uses tensor parallelism (TP) across two ranks, and its KV payload stores 2-byte BFloat16 (BF16) values and occupies gigabytes.

The distinction is about ownership, not process names. A router belongs to the control plane while it chooses workers, but its request-forwarding socket participates in the data path. A worker reports health to control loops and executes model tensors in the data plane. Draw the boundary around each responsibility instead of labeling an entire product as one plane.

This split prevents a common design error. A transport library shouldn't decide tenant authorization, and a global router shouldn't manipulate GPU addresses. Each layer receives the smallest authority needed for its job.

Trace req-8k-017 from router to decode

Keep one immutable identity through the whole path:

The values describe a fixed teaching model with the same KV shape used in the prerequisite lesson. They aren't a claim about a newly released checkpoint. An actual service should derive this tuple from an immutable deployment manifest, not from client-supplied fields.

1. Authenticate and canonicalize once

The frontend authenticates tenant-73, applies prompt and output limits, assigns req-8k-017, and produces the canonical token sequence. If prefill and decode independently render chat templates, a template or tokenizer mismatch can change token positions even when the visible text looks identical. Current vLLM disaggregated-prefill documentation exposes one way to reuse prefill token IDs at decode through transfer parameters, which makes the identity requirement visible.^{[1]Reference 1Disaggregated Prefillinghttps://github.com/vllm-project/vllm/blob/main/docs/features/disagg_prefill.md}

Canonicalization has one owner. Downstream workers receive token IDs plus a digest and reject an incompatible tokenizer revision.

2. Reserve both phases before expensive work

The router selects prefill worker P7 and decode group D3. P7 has prompt-token capacity. D3 has enough KV bytes, active-sequence slots, and deadline headroom for the expected output.

D3 returns a short-lived reservation for attempt 4. Only then does P7 start the 8K prefill. This ordering is the simplest form of backpressure: downstream scarcity reaches the upstream admission point before compute is spent.

Mooncake's design makes the same systems issue concrete. Its Conductor considers KV placement and prefill/decode load, and its overload policy rejects work early because rejecting after prefill wastes the completed prompt computation.^{[2]Reference 2Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving.https://arxiv.org/abs/2407.00079}

Why reserve decode capacity before prefill begins?

Answer

Prefill creates a large, request-specific payload. If no decoder can admit it afterward, the prompt compute and transfer are wasted. A short-lived decode reservation turns downstream capacity into an upstream admission signal.

3. Prefill writes logical KV blocks

P7 processes all 8,192 prompt tokens. It writes 512 logical blocks because each block covers 16 token positions. Under tensor parallelism (TP) of 2, each prefill rank owns its deployed share of every logical block.

Physical block IDs are local allocator details. Prefill rank 0 might store logical block 31 in physical block 904, while decode rank 0 allocates physical block 118 for the same logical range. Correctness requires matching logical positions and shard meaning, not identical physical addresses.

Our request contract gives decode exclusive ownership of user-visible streaming. Prefill returns the prompt KV plus any engine-specific metadata needed to start decode, but the frontend doesn't expose a token until D3 has committed the transfer. Another runtime may divide first-token work differently. Whichever contract you choose, write it down so two components never emit token 1 or advance the sampling state twice.

4. Serialize a manifest, not 2.50 GiB into JSON

Prefill creates a transfer manifest, a small versioned record that describes the payload:

The JSON is explanatory, not a portable standard. D3 owns the lease and measures its expiry on a local monotonic clock, so the manifest carries an opaque reservation ID rather than a timestamp or TTL that another host might treat as authoritative. Production metadata also needs backend connection fields and rank-local buffer descriptors. Keep addresses, remote keys, and connection material out of ordinary application logs.

The KV tensor payload stays in registered memory. NVIDIA Inference Transfer Library (NIXL) describes a conductor-managed interface where agents register memory, exchange serialized connection metadata through a side channel or central service, and submit local and remote buffer descriptor lists for asynchronous transfer.^{[3]Reference 3NVIDIA Inference Transfer Library: Architecture and API Flowhttps://github.com/ai-dynamo/nixl/blob/main/docs/nixl.md} That division matters:

Turning the whole payload into a language-level byte string adds copies and allocation pressure. A scatter-gather transfer can instead describe the paged regions that already exist. The decode runtime may still need a layout permutation when prefill and decode use different supported tensor layouts. That conversion belongs in a versioned connector contract and in the latency receipt.

5. Move bytes along a measured topology

The transfer layer now moves attempt 4's rank-local KV shards to D3. NIXL is one transport abstraction. It can select among common backends for registered memory types, while the surrounding conductor still owns worker choice, metadata exchange, and failure policy.^{[3]Reference 3NVIDIA Inference Transfer Library: Architecture and API Flowhttps://github.com/ai-dynamo/nixl/blob/main/docs/nixl.md}

NVIDIA Dynamo documents the same separation in its disaggregated flow: a router selects prefill, receives backend-specific transfer metadata, routes decode, and lets NIXL coordinate direct KV movement between worker memory.^{[4]Reference 4Disaggregated Servinghttps://docs.nvidia.com/dynamo/dev/knowledge-base/concepts/system-architecture/disaggregated-serving} Mooncake uses a Conductor for global scheduling and a separate Messenger component for paged KV movement over RDMA (Remote Direct Memory Access), including storage tiers outside GPU memory.^{[2]Reference 2Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving.https://arxiv.org/abs/2407.00079}

These are implementations of compatible boundaries, not a ranking:

The stable lesson is the interface: decision, movement, and execution need separate contracts. Product names and supported backends will change.

Topology decides whether the split is sensible:

Don't route only on worker load. A lightly loaded decoder across a weak or congested path can lose to a busier decoder beside the prefiller. Dynamo's topology-aware router documentation explicitly treats rack or zone transfer domains as routing constraints for disaggregated serving.^{[5]Reference 5Disaggregated Serving: Prefill and Decode Routinghttps://docs.nvidia.com/dynamo/dev/knowledge-base/modular-components/router/disaggregated-serving}

6. Commit before decode streams

D3 allocates destination blocks under reservation d3-lease-772, validates the manifest, and waits for all expected rank and block ranges. Partial data stays invisible to attention. When completion arrives, decode performs these checks:

1. Attempt 4 is still the current fenced attempt for req-8k-017.
2. Reservation hasn't expired and belongs to tenant-73.
3. Model, tokenizer, adapter, sampling, KV layout, and TP rank map match.
4. All 512 logical blocks arrived on every required rank.
5. Transfer integrity checks and connector completion succeeded.

Only then does the decoder atomically change state from TRANSFERRING to DECODE_READY. It acknowledges the commit, becomes sole stream owner, and releases the prefiller's retained source blocks after the retry window closes.

That commit point is the most important correctness boundary in the data plane. A transport completion means the copy operation finished. A decode commit means the request may consume and expose the received state.

Size the bytes before choosing the fabric

The prerequisite KV lesson gave the payload formula. For each cached token:

$$B_{token} = 2 \times L \times H_{kv} \times D_h \times P$$

$L$ is layer count, $H_{kv}$ is KV-head count, $D_h$ is head dimension, and $P$ is bytes per stored value. The leading 2 stores both Key and Value.

For request req-8k-017:

$$B_{token} = 2 \times 80 \times 8 \times 128 \times 2 = 327{,}680\text{ bytes} = 320\text{ KiB}$$

Across 8,192 prompt tokens:

$$B_{prompt} = 327{,}680 \times 8{,}192 = 2{,}684{,}354{,}560\text{ bytes} = 2.50\text{ GiB}$$

A 128-token output reservation adds 40 MiB if the same KV shape continues. With TP=2 and ideal KV-head sharding, each rank transfers about 1.25 GiB for the prompt. The two rank transfers still total 2.50 GiB across the model. Replication or layout conversion can raise physical bytes, so inspect actual connector counters instead of dividing blindly by TP.

At block size 16, each global logical block contains 5 MiB of KV payload. There are 512 prompt blocks. Paging changes how those blocks are placed and described; it doesn't shrink their mathematical payload.

Does TP=2 cut the request's total KV transfer from 2.50 GiB to 1.25 GiB?

Answer

Not under ideal sharding. It cuts each rank's shard to about 1.25 GiB, but both shards must arrive, so model-wide transferred payload remains about 2.50 GiB. Replication or conversion can move more.

Transfer time is a budget term

For payload bytes $B$ and measured effective payload bandwidth $R$, the no-overlap transfer time is:

$$T_{xfer} = \frac{B}{R}$$

Use effective application payload bandwidth, not a link's marketing line rate. Include registration, staging, layout conversion, queueing, and completion signaling as separate terms when they aren't already inside the measurement.

The runnable exercise keeps the request shape fixed and treats phase timings as an illustrative measurement fixture. It calculates KV bytes, transfer time at three observed payload rates, and one no-overlap time to first token (TTFT) receipt. Replace the timing dictionary with numbers from your trace.

The 650 ms target and phase timings are not vendor results. They form a reviewable budget. At the fixture's 25 GB/s effective rate, transfer consumes 107.4 ms and leaves 150.6 ms of margin. If p99 transfer rises by 180 ms under congestion, the request misses TTFT even when prefill compute stays flat.

Layer-wise or chunk-wise transfer can overlap some movement with ongoing prefill. Mooncake describes layer-wise transfer for this purpose, while Dynamo documents non-blocking NIXL movement.^{[2]Reference 2Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving.https://arxiv.org/abs/2407.00079[4]Reference 4Disaggregated Servinghttps://docs.nvidia.com/dynamo/dev/knowledge-base/concepts/system-architecture/disaggregated-serving} When work overlaps, don't add both full durations. Trace the critical path and report overlap explicitly:

$$T_{handoff} = T_{prefill} + T_{xfer} - T_{overlap} + T_{commit}$$

Overlap doesn't erase bandwidth consumption. It can move waiting time off the critical path while still congesting links used by other requests.

Admission and backpressure need byte credits

Request counts hide the resource that decode must reserve. One 100-token prompt and one 8K prompt both count as one request, but their prompt KV bills differ by roughly 82 times under the same model shape.

Use credits that reflect the constrained resource:

• Prefill credits: prompt tokens or predicted prefill work inside a deadline window.
• Transfer credits: bytes in flight per topology domain, with bounded queue age.
• Decode credits: reserved KV bytes, active sequences, and expected output-token occupancy.
• Tenant credits: per-tenant prompt, byte, and concurrency limits before shared queues.

Suppose D3 has 8 GiB of uncommitted KV capacity. Each 8K request reserves 2.50 GiB plus 40 MiB for 128 output tokens. Three requests fit under the arithmetic bound. A fourth shouldn't start prefill and hope capacity appears later.

Reservations need a lease. If attempt 4 doesn't commit before its monotonic expiry, D3 returns those credits. A late transfer carrying the old fencing token is discarded. Leases prevent dead workers from pinning capacity; fencing prevents their delayed packets from reviving stale state.

Backpressure flows in one direction even though status flows both ways:

1. Decode publishes available byte and sequence credits.
2. Router grants a reservation only while credits and deadline margin remain.
3. Prefill starts only with a valid reservation.
4. Transfer queue stops accepting new payloads when its byte or age limit is reached.
5. Edge rejects or defers work with an explicit retry policy before internal queues become unbounded.

DistServe defines per-GPU goodput as maximum request rate served while meeting a service-level objective (SLO) attainment goal. It applies separate TTFT and decode-latency requirements while choosing resource allocation and placement.^{[6]Reference 6DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670} Raw completed tokens don't count as healthy capacity when their requests miss either user-facing objective.

Why should decode admission use KV bytes rather than only request count?

Answer

Requests have different prompt and output lengths, KV layouts, and precision. Byte reservations represent the memory that must exist after handoff, while a request counter treats a tiny prompt and an 8K prompt as equal.

Routing identity is a correctness protocol

A load balancer can send stateless HTTP traffic to any healthy replica. A disaggregated inference router must select a decoder that can interpret and own existing request state.

Use three identities for different jobs:

• Request ID: Stable across all attempts. It joins the user operation and idempotency record.
• Attempt or fencing token: Increases whenever ownership changes. Only highest live attempt may commit or stream.
• Transfer ID: Names one concrete byte movement. Retransferring same attempt gets a new transfer ID without creating a second stream owner.

Worker compatibility is an exact tuple, not a model family name:

Two groups can both advertise m42 yet disagree on tensor parallel (TP), pipeline parallel (PP), or expert parallel (EP) layout. A connector may support a documented conversion, but the router must select that conversion deliberately and budget its cost. Silent reinterpretation is never a fallback.

Cache locality adds another identity. Prefix hashes must include tenant or sharing scope, model tuple, adapter, and token sequence. A matching token prefix under a different adapter or tenant policy isn't reusable state.

Security starts at the metadata side channel

KV tensors are derived from user prompts and model activations. Treat them as sensitive workload data. Transfer descriptors can also grant access to registered memory regions, so they belong inside the trusted serving boundary.

NIXL's architecture assumes an external conductor handles user requests and exchanges serialized agent metadata through a side channel or central metadata service.^{[3]Reference 3NVIDIA Inference Transfer Library: Architecture and API Flowhttps://github.com/ai-dynamo/nixl/blob/main/docs/nixl.md} That means NIXL is a byte-movement abstraction, not the service's authentication and authorization protocol. The surrounding platform must secure the metadata path.

Build the trust boundary explicitly:

Register the smallest memory pool the connector needs. Bind descriptors to an authenticated worker identity and short attempt lifetime. Revoke or invalidate remote metadata when workers leave or fail; NIXL documents explicit remote-agent invalidation for dynamic removal and failures.^{[3]Reference 3NVIDIA Inference Transfer Library: Architecture and API Flowhttps://github.com/ai-dynamo/nixl/blob/main/docs/nixl.md}

Don't assume RDMA implies encryption or tenant isolation. The deployment owns network segmentation and any confidentiality layer required by its threat model. Measure end-to-end integrity checks instead of adding them blindly. Use authenticated transport integrity and add chunk checks when the connector doesn't provide the required guarantee. Sampled validation belongs in diagnostics, not commit proof; semantic manifest checks stay mandatory.

Failure and retry ownership

Retries are safe only while the protocol knows whether output became visible. The commit gate and first-byte boundary make that decision possible.

Before FirstByte, the router can create a new fenced attempt because the client hasn't observed output. After FirstByte, repeating the request may duplicate tokens or side effects in a caller that consumes the stream incrementally. Exact continuation needs an explicit resumable-stream protocol, including output-token position and sampling state. Without one, surface a terminal stream error.

Retain source KV until decode commit plus a short retry window. Releasing immediately after transport completion removes cheap retransmission. Retaining indefinitely turns prefiller memory into an accidental second cache. The lease and SLO budget should determine the window.

Every cleanup API accepts (request_id, attempt) and succeeds when state is already gone. Idempotency makes duplicate cancellation and delayed failure notices harmless. Fencing makes a late success from attempt 3 unable to delete or commit attempt 4.

Current vLLM KV-transfer configuration makes retry policy explicit with fail and recompute behavior for KV load failures.^{[7]Reference 7KVTransferConfig API Referencehttps://docs.vllm.ai/en/latest/api/vllm/config/kv_transfer/} Treat that runtime setting as one local mechanism. End-to-end retry ownership still belongs to the service protocol because only it knows whether user-visible streaming began.

A transfer times out after 70% of blocks arrive, but decode hasn't committed or streamed. What happens next?

Answer

Decode discards or quarantines the partial destination under that transfer ID. The router can retransmit from retained source KV or start a fenced recomputation attempt. Decode must never attend to the 70% partial cache.

Observe one request without logging its data

Give req-8k-017 one distributed trace and propagate its server-issued request ID and attempt through every control message. Keep request IDs out of metric labels because their cardinality is unbounded.

Useful spans follow ownership changes:

Each span records safe structural fields: attempt, worker pool, topology domain, model revision, prompt-token bucket, expected and transferred bytes, block count, retry reason, and deadline remaining. It shouldn't record prompt text, raw token IDs, remote memory keys, or full transfer descriptors.

Metrics need phase-specific denominators:

Separate logical payload bytes from physical wire bytes. A retry can move 5.00 GiB for one 2.50 GiB request. Replicated KV heads or a staging copy can also change the ratio. Without both counters, the network bill has no explanation.

Build the SLO from phase budgets

TTFT covers the full critical path until token 1 is visible:

$$TTFT = T_{edge} + T_{route/reserve} + Q_p + T_{prefill} + T_{handoff} + Q_d + T_{first-token}$$

After streaming begins, inter-token latency (ITL) belongs mainly to decode scheduling, model execution, and stream flushing. A transfer regression should raise TTFT while leaving steady-state ITL mostly unchanged for successfully committed requests. That symptom is a useful localization clue.

Use an SLO ledger rather than one end-to-end number:

These targets match the exercise fixture and remain illustrative. Set production budgets from user experience, then validate them with measured workload distributions. Splitwise and DistServe both motivate phase-specific provisioning because prefill and decode have different resource behavior, while state transfer becomes a new term that placement must minimize.^{[8]Reference 8Splitwise: Efficient Generative LLM Inference Using Phase Splitting.https://arxiv.org/abs/2311.18677[6]Reference 6DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670}

Goodput closes the loop: count only offered work that completes inside both first-token and decode objectives. Also report rejection rate. A system can protect latency by rejecting everything, which is healthy backpressure only when the allowed rejection budget says so.

TTFT rises, effective KV-transfer bandwidth falls, and ITL for committed streams stays flat. Which boundary should you inspect first?

Answer

Inspect the prefill-to-decode handoff and its topology. Stable ITL suggests decode execution is healthy after commit, while lower effective transfer bandwidth adds delay before the first token.

Review the contract before scaling it

For req-8k-017, the complete path is now auditable:

1. Frontend authenticates tenant-73, creates canonical token IDs, and assigns stable request identity.
2. Router selects compatible P7 and D3, then reserves decode bytes before prefill starts.
3. Prefill creates 512 logical KV blocks totaling 2.50 GiB across two ranks.
4. Handoff serializes identity and descriptors while the tensor payload remains in registered memory.
5. Transfer follows a measured topology and records payload, wire bytes, time, and retries.
6. Decode validates attempt, layout, completeness, integrity, and lease before one atomic commit.
7. Decode alone owns user-visible streaming; cancellation and cleanup reach every local owner.

Disaggregation earns its complexity when a tuned mixed pool can't keep both prompt and streaming latency inside their targets, and when the KV handoff fits the remaining first-token budget. DistServe and Splitwise show why separating resource plans can help; Mooncake shows why KV locality, early rejection, and transfer overlap become first-class scheduling concerns.^{[6]Reference 6DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670[8]Reference 8Splitwise: Efficient Generative LLM Inference Using Phase Splitting.https://arxiv.org/abs/2311.18677[2]Reference 2Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving.https://arxiv.org/abs/2407.00079}

If the mixed pool already meets its SLOs, keep it. The simplest correct data plane has no remote KV handoff. When evidence justifies a split, the design is ready only after bytes, identities, trust, failures, and observability all have named owners.