Distributed Inference Data Plane

Request req-8k-017 contains 8,192 prompt tokens. While one replica processes that prompt, already-streaming replies may wait for their next decode step. Moving prefill to a separate worker can reduce this interference, but creates another wait: the prompt's key-value (KV) cache must reach the decoder. For the teaching model below, that's 2.50 GiB, not a small routing message.

A fast copy to an incompatible decoder is useless. A correct copy into a buffer that's already been reassigned is dangerous. Distributed inference has to coordinate both the movement of bytes and the right to use them.

Model parallelism explained how ranks cooperate inside a model replica. KV Cache and PagedAttention explained cached token state and logical blocks. The GPU failure operations lesson covered worker failures. Here those pieces meet in one request's prefill-to-decode handoff.

Decisions, tensor traffic, and token streams

The control plane chooses workers, checks caller permissions, reserves capacity, and assigns request ownership. The data plane executes the model and carries its prompt, intermediate tensors, KV state, and output. These names describe responsibilities, not whole processes: a router makes placement decisions and also forwards request bytes.

Our request uses prefill group P7 and decode group D3, each with two tensor-parallel (TP) ranks. Communication inside either group combines partial model results. Communication between the groups transfers request state. These are different operations even if they share the same physical network.

Splitwise and DistServe study separating prefill and decode so their resources and latency objectives can be managed independently.^{[1]Reference 1Splitwise: Efficient Generative LLM Inference Using Phase Splitting.https://arxiv.org/abs/2311.18677[2]Reference 2DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670} A split isn't automatically faster. It adds transfer, coordination, and separately provisioned pools; a tuned mixed pool with chunked prefill may already meet the same objectives. vLLM still labels its disaggregated-prefill feature experimental.^{[3]Reference 3Disaggregated Prefillinghttps://github.com/vllm-project/vllm/blob/main/docs/features/disagg_prefill.md}

Collectives inside a replica must agree

Suppose one layer produces partial activation vectors [1,2] on rank 0 and [10,20] on rank 1. A sum all-reduce gives [11,22] to both ranks. An all-gather instead concatenates contributions in rank order. Reduce-scatter sums corresponding elements and gives each rank only its assigned output chunk.^{[4]Reference 4NCCL Collective Operationshttps://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html}

The CPU example distinguishes these contracts. The sequence labels are application-level diagnostics, not fields NVIDIA Collective Communications Library (NCCL) checks for you.

For a given collective, participating ranks must agree on the operation and its arguments, including element count and dtype. They must issue matching collective sequences. Swapping two same-shaped reductions can combine the wrong layers without an obvious shape error; mismatches can also hang or corrupt results. NCCL's group-call documentation makes the ordering requirement explicit.^{[4]Reference 4NCCL Collective Operationshttps://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html[5]Reference 5NCCL Group Callshttps://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/groups.html}

A successful NCCL enqueue isn't a completed GPU result. The consumer needs the appropriate CUDA stream dependency or completion event.^{[6]Reference 6NCCL CUDA Stream Semanticshttps://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/streams.html} Cancelling a request on just one rank mustn't make that rank skip a collective that its peers still execute. Coordinate scheduling changes across the group; a failed group needs the runtime's communicator recovery procedure, not an independent retry of one rank's next collective.

The script checks arithmetic and an abstract schedule. It doesn't launch NCCL, detect a real distributed hang, or validate CUDA stream synchronization.

How much state crosses the handoff?

Keep req-8k-017 fixed. Its hypothetical decoder-only model has 80 layers, 8 KV heads per layer, head dimension 128, and 2-byte BF16 cache values. Every cached token stores both a key and a value:

$$B_{token}=2 \times 80 \times 8 \times 128 \times 2 =327{,}680\text{ bytes}=320\text{ KiB}.$$

Multiplying by 8,192 tokens gives 2,684,354,560 bytes, or 2.50 GiB. Under ideal two-way KV-head sharding, each rank holds 1.25 GiB. Both shards must arrive, so TP=2 doesn't halve the model-wide payload. Replicated heads, padding, quantization metadata, or layout conversion can change physical bytes. This formula isn't a universal cache layout for every architecture.

At 16 tokens per logical block, the prompt occupies 512 blocks. Each model-wide block accounts for 5 MiB. Reserving another 128 token positions adds 40 MiB, giving 2,726,297,600 bytes before allocator and other runtime overhead. A partial final block rounds allocation up; the payload of real tokens and the reserved block capacity are different quantities.

The exercise separates those two quantities and uses assumed, not measured, effective bandwidth. GB/s uses decimal bytes; GiB uses powers of 1,024.

For serial transfer with aggregate effective payload rate $R$, the transfer term is $B/R$. At an assumed 25 GB/s, it's 107.4 ms. If two independent links each carry one 1.25 GiB shard concurrently at 25 GB/s, the transfer finishes in roughly 53.7 ms before overhead. If both streams share a 25 GB/s bottleneck, dividing each shard by 25 GB/s falsely doubles the available capacity.

Reserve capacity before creating expensive state

D3 has 8 GiB of available KV capacity in our simplified budget. Three requests fit; four don't. A request counter alone misses that limit: one 100-token prompt and one 8K prompt each count as one request but consume very different memory.

Our teaching protocol reserves downstream capacity before prefill starts. This reduces wasted prompt computation when decode is full. It's a design choice, not a claim that every engine implements this exact sequence. Mooncake discusses early rejection and KV-aware scheduling to avoid completing prefill for work that can't be served.^{[7]Reference 7Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving.https://arxiv.org/abs/2407.00079}

Admission must consider several constraints together:

An output reservation is a budget, not a prediction that generation ends there. Enforce an output cap or acquire more credit before crossing the reserved boundary. Cancellation, timeout, and completed requests must eventually release their charges, but only after the associated memory is safe to reuse.

A lease limits how long a reservation may become ready. Its expiry uses the destination's monotonic clock; the sender doesn't get to extend it by supplying a larger timestamp. After commit, active request ownership replaces this admission lease. Expiry revokes admission immediately. It doesn't prove that remote writes have stopped. Retired buffers can remain physically charged while their transfers drain.

Agree on the interpretation before moving bytes

The destination first compares the sender's manifest against a trusted reservation. The manifest describes the transfer; it doesn't authorize itself.

The short digests here are labels, not cryptographic examples. Use canonical token IDs from one owner rather than independently rendering chat templates at both workers. vLLM documents experimental reuse of prefill token IDs through kv_transfer_params for this purpose.^{[3]Reference 3Disaggregated Prefillinghttps://github.com/vllm-project/vllm/blob/main/docs/features/disagg_prefill.md} Sampling state, the last token or hidden state needed to start decode, and first-token ownership also belong in the engine-specific handoff. KV tensors alone aren't necessarily enough to resume generation.

Logical block 31 can live at physical block 904 in P7 and block 118 in D3. Local addresses needn't match. The mapping from logical positions to the correct destination shard must match. A connector may explicitly support repartitioning or layout conversion; budget and validate that conversion rather than silently reinterpreting bytes.

A transfer engine doesn't own the request

NIXL separates memory registration, connection-metadata exchange, buffer descriptors, and asynchronous transfer operations. An external conductor coordinates the workload and metadata path.^{[8]Reference 8NVIDIA Inference Transfer Library: Architecture and API Flowhttps://github.com/ai-dynamo/nixl/blob/main/docs/nixl.md} Serialize that small metadata, not gigabytes of tensors into JSON. Keep the payload in registered GPU or host memory supported by the connector.

The surrounding service still owns authorization, placement, retry policy, and output ownership. Protect the metadata channel, restrict descriptor exposure, and isolate registered regions. Remote keys and descriptors don't belong in application logs. RDMA isn't itself a promise of encryption or tenant isolation. Transport integrity and authenticated workload identity are separate from checking the model and layout fields.

Dynamo's documented flow returns backend-specific handoff metadata from prefill and uses it to coordinate decode and NIXL movement.^{[9]Reference 9Disaggregated Servinghttps://docs.nvidia.com/dynamo/dev/knowledge-base/concepts/system-architecture/disaggregated-serving} Its router can constrain decode placement to a compatible topology domain, such as a rack or zone.^{[10]Reference 10Disaggregated Serving: Prefill and Decode Routinghttps://docs.nvidia.com/dynamo/dev/knowledge-base/modular-components/router/disaggregated-serving} A lightly loaded decoder across a congested path can finish later than a busier nearby one. Evaluate the whole path, including GPU-to-NIC placement, staging, registration, conversion, and shared link contention.

NIXL's backend guide gives two especially relevant limits: distinct transfer requests aren't automatically ordered, and the application must prevent simultaneous transfers from corrupting the same region. Releasing a transfer can involve asynchronous cancellation that hasn't yet succeeded.^{[11]Reference 11NIXL Backend Guidehttps://github.com/ai-dynamo/nixl/blob/main/docs/BackendGuide.md} Treat “cancel requested,” “operation stopped,” and “buffer reusable” as different events.

Timeout revokes ownership, not physical access

Suppose 70% of attempt 4's KV has arrived when its lease expires. The destination must reject commit. But returning those blocks to the allocator immediately could let an old remote write overwrite attempt 5's cache.

A conservative whole-prompt protocol has four distinct boundaries:

1. Reserve: charge capacity and record the expected manifest under a server-issued attempt.
2. Transfer: keep incoming blocks unavailable to attention, even if some are complete.
3. Commit: after compatibility, complete unique coverage, integrity, and connector-required visibility checks, grant one decoder ownership.
4. Retire and reclaim: stop admission and streaming, then wait for successful cancellation or completion and GPU-reader quiescence before reusing memory.

Advanced connectors can overlap layer-wise transfer and compute. That requires dependency tracking for each consumed region; it doesn't permit attention to read a missing region. The whole-prompt gate here is intentionally simpler.

A duplicate arrival mustn't count as a new block. Counting 1,024 notifications isn't equivalent to receiving 512 distinct blocks on each of two ranks. Likewise, a completed byte count doesn't validate layer coverage or a manifest's interpretation.

Execute the ownership rules locally

Download handoff_model.py. It serializes transitions in one Python process and stores block identities, not tensor data. The expected manifest comes from its trusted reservation; a received manifest must match it. finish(), abort_ack(), and consumer_stopped() represent trusted runtime evidence, not measurements that this script obtains.

This short run expires a lease while writes are still possible. Credit remains unavailable until successful abort acknowledgement.

The longer checks reject eight manifest mismatches, exercise duplicate and missing blocks, reject streaming before commit, retain an old attempt's charge during replacement, and prevent retries after visible output. They also enumerate 720 orderings of two arrivals, completion, commit, cancellation, and reclamation.

These checks validate the model's transitions. They don't establish distributed atomicity, authentication, memory visibility, or cancellation semantics in NIXL, vLLM, Dynamo, CUDA, or an RDMA backend. A deployed coordinator needs authoritative attempt ownership, serialization or compare-and-swap at commit, durable recovery rules, and a frontend that fences stale streams. A higher attempt number in a message alone doesn't stop an old writer or stream owner.

Failure modes and retry boundaries

Before output is visible, the service may retry if it can fence the old owner, preserve request semantics, and stay within its deadline and retry budget. This still costs compute and network traffic. Current vLLM configuration exposes fail and recompute policies for KV-load failure; those are local engine behaviors, not a complete resumable-stream protocol.^{[12]Reference 12KVTransferConfig API Referencehttps://docs.vllm.ai/en/latest/api/vllm/config/kv_transfer/}

After output may have reached the client, don't transparently restart at token 1. A lost acknowledgement can make delivery uncertain. Exact continuation requires an agreed cursor, token or frame deduplication, sampling state, and authoritative ownership. Without that protocol, return an explicit interrupted-stream error.

Retaining source KV briefly after commit can make retransmission cheaper, but consumes prefill memory. The retained copy needs its own accounted capacity and expiry. Never release a source buffer while a transport may still read it. NIXL's remote-agent invalidation helps manage disappearing peers, but the application still has to coordinate safe buffer lifetime.^{[8]Reference 8NVIDIA Inference Transfer Library: Architecture and API Flowhttps://github.com/ai-dynamo/nixl/blob/main/docs/nixl.md}

Measure the critical path, not a sum of unrelated percentiles

For one request without overlap, time to first token (TTFT) is the sum of its edge, routing, queue, prefill, handoff, and first-token durations. For overlapping spans, use the union of their time intervals on the dependency path. Don't count prefill again inside a handoff term that already includes it.

The synthetic timeline assumes prefill runs from 0 to 310 ms. Transfer either starts afterward or starts at 240 ms and overlaps its final 70 ms. Commit plus the first decode step takes 28 ms after both finish. No network measurements are implied.

Phase p99 values don't generally add to end-to-end p99: the slowest prefill and slowest transfer may belong to different requests. Compute end-to-end quantiles from joined request traces. Keep request IDs in traces, not unbounded metric labels.

A host staging copy increases memory-system traffic, but doesn't necessarily increase network wire bytes. Specify where every byte counter is measured. Likewise, reduced handoff bandwidth may raise TTFT without changing steady-state inter-token latency (ITL), but shared-fabric contention can also slow collectives and hurt ITL. Treat that symptom as a clue, not proof.

Compare a mixed pool and a split pool under the same offered workload, output lengths, hardware budget, and SLO definitions. Report goodput, rejection, errors, and tail latency together. A service that rejects almost everything can have excellent latency for the few requests it accepts.

Try a changed workload

Follow-up questions

Prefill can finish 40 of these 8K requests per second, but all KV transfers share an effective 25 GB/s path. Can the transfer queue remain stable at that arrival rate?

Answer

Each request moves 2,684,354,560 bytes, so 40 requests/s demands about 107.4 GB/s. Even before other overhead, the 25 GB/s path supports only about 9.31 such transfers/s. A finite queue can absorb a short burst, not this sustained mismatch. Admission must slow or reject work, or the system needs less transferred state or more independent bandwidth. Faster prefill alone doesn't resolve it.

The prompt grows from 8,192 to 8,193 tokens while the 128-position output budget stays fixed. How much additional KV capacity does the 16-token block allocator reserve?

Answer

The total grows from 8,320 positions, exactly 520 blocks, to 8,321 positions, which requires 521 blocks. One model-wide block is 16 × 320 KiB = 5 MiB. Reserved capacity rises from 2,600 to 2,605 MiB, although the additional real token contains only 320 KiB of KV. The allocation bound and logical payload bound serve different purposes.

Evaluation rubric

• Calculate both logical KV payload and rounded allocation; distinguish aggregate link capacity from each stream's requested rate.
• Trace a timed-out attempt through fencing, late writes, quiescence, and reclamation without returning its byte credit early.
• Explain which guarantees the local model checks and which require actual connector, GPU, and coordinator evidence.