Distributed Inference Data Plane

Request req-8k-017 contains 8,192 prompt tokens. While one replica processes that prompt, already-streaming replies may wait for their next decode step. Moving prefill to a separate worker can reduce this interference, but creates another wait: the prompt's key-value (KV) cache must reach the decoder. For the teaching model below, that's 2.50 GiB, not a small routing message.

A fast copy to an incompatible decoder is useless. A correct copy into a buffer that's already been reassigned is dangerous. Distributed inference has to coordinate both the movement of bytes and the right to use them.

Model parallelism explained how ranks cooperate inside a model replica. KV Cache and PagedAttention explained cached token state and logical blocks. The GPU failure operations lesson covered worker failures. Here those pieces meet in one request's prefill-to-decode handoff.

Decisions, tensor traffic, and token streams

The control plane chooses workers, checks caller permissions, reserves capacity, and assigns request ownership. The data plane executes the model and carries its prompt, intermediate tensors, KV state, and output. These names describe responsibilities, not whole processes: a router makes placement decisions and also forwards request bytes.

Our request uses prefill group P7 and decode group D3, each with two tensor-parallel (TP) ranks. Communication inside either group combines partial model results. Communication between the groups transfers request state. These are different operations even if they share the same physical network.

Splitwise and DistServe study separating prefill and decode so their resources and latency objectives can be managed independently.^{[1]Reference 1Splitwise: Efficient Generative LLM Inference Using Phase Splitting.https://arxiv.org/abs/2311.18677[2]Reference 2DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670} A split isn't automatically faster. It adds transfer, coordination, and separately provisioned pools; a tuned mixed pool with chunked prefill may already meet the same objectives. vLLM still labels its disaggregated-prefill feature experimental.^{[3]Reference 3Disaggregated Prefillinghttps://github.com/vllm-project/vllm/blob/main/docs/features/disagg_prefill.md}

Collectives inside a replica must agree

Suppose one layer produces partial activation vectors [1,2] on rank 0 and [10,20] on rank 1. A sum all-reduce gives [11,22] to both ranks. An all-gather instead concatenates contributions in rank order. Reduce-scatter sums corresponding elements and gives each rank only its assigned output chunk.^{[4]Reference 4NCCL Collective Operationshttps://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html}

The CPU example distinguishes these contracts. The sequence labels are application-level diagnostics, not fields NVIDIA Collective Communications Library (NCCL) checks for you.

partials = [[1, 2], [10, 20]]
reduced = [sum(column) for column in zip(*partials, strict=True)]
all_reduce = [reduced.copy() for _ in partials]
reduce_scatter = [[reduced[0]], [reduced[1]]]
all_gather = [value for shard in reduce_scatter for value in shard]
assert all_reduce == [[11, 22], [11, 22]]
assert all_gather == reduced
print("all-reduce:", all_reduce)
print("reduce-scatter:", reduce_scatter, "then gather:", all_gather)

def matching_schedule(rank_logs):
    return bool(rank_logs) and all(log == rank_logs[0] for log in rank_logs)

# (model step, layer, operation, element count, dtype)
a = (0, 3, "all_reduce_sum", 2, "float32")
b = (0, 4, "all_reduce_sum", 2, "float32")
assert matching_schedule([[a, b], [a, b]])
assert not matching_schedule([[a, b], [b, a]])
assert not matching_schedule([[a, b], [a]])
wrong_count = (0, 3, "all_reduce_sum", 3, "float32")
assert not matching_schedule([[a], [wrong_count]])
wrong_dtype = (0, 3, "all_reduce_sum", 2, "float16")
assert not matching_schedule([[a], [wrong_dtype]])
print("reordered, missing, mismatched-count and mismatched-dtype calls: detected")

all-reduce: [[11, 22], [11, 22]]
reduce-scatter: [[11], [22]] then gather: [11, 22]
reordered, missing, mismatched-count and mismatched-dtype calls: detected

For a given collective, participating ranks must agree on the operation and its arguments, including element count and dtype. They must issue matching collective sequences. Swapping two same-shaped reductions can combine the wrong layers without an obvious shape error; mismatches can also hang or corrupt results. NCCL's group-call documentation makes the ordering requirement explicit.^{[4]Reference 4NCCL Collective Operationshttps://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/collectives.html[5]Reference 5NCCL Group Callshttps://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/groups.html}

A successful NCCL enqueue isn't a completed GPU result. The consumer needs the appropriate CUDA stream dependency or completion event.^{[6]Reference 6NCCL CUDA Stream Semanticshttps://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/streams.html} Cancelling a request on just one rank mustn't make that rank skip a collective that its peers still execute. Coordinate scheduling changes across the group; a failed group needs the runtime's communicator recovery procedure, not an independent retry of one rank's next collective.

Grouping adds an earlier boundary: individual calls inside a group can return before enqueue. Wait for ncclGroupEnd; with a nonblocking communicator, ncclInProgress requires checking asynchronous communicator state until issuing succeeds before using related CUDA synchronization. Issued still doesn't mean GPU execution completed.^{[5]Reference 5NCCL Group Callshttps://docs.nvidia.com/deeplearning/nccl/user-guide/docs/usage/groups.html}

The script checks arithmetic and an abstract schedule. It doesn't launch NCCL, detect a real distributed hang, or validate CUDA stream synchronization.

How much state crosses the handoff?

Keep req-8k-017 fixed. Its hypothetical decoder-only model has 80 layers, 8 KV heads per layer, head dimension 128, and 2-byte BF16 cache values. Every cached token stores both a key and a value:

Multiplying by 8,192 tokens gives 2,684,354,560 bytes, or 2.50 GiB. Under ideal two-way KV-head sharding, each rank holds 1.25 GiB. Both shards must arrive, so TP=2 doesn't halve the model-wide payload. Replicated heads, padding, quantization metadata, or layout conversion can change physical bytes. This formula isn't a universal cache layout for every architecture.

At 16 tokens per logical block, the prompt occupies 512 blocks. Each model-wide block accounts for 5 MiB. Reserving another 128 token positions adds 40 MiB, giving 2,726,297,600 bytes before allocator and other runtime overhead. A partial final block rounds allocation up; the payload of real tokens and the reserved block capacity are different quantities.

The exercise separates those two quantities and uses assumed, not measured, effective bandwidth. GB/s uses decimal bytes; GiB uses powers of 1,024.

BYTES_PER_TOKEN = 2 * 80 * 8 * 128 * 2
BLOCK_TOKENS = 16

def reserved_bytes(prompt_tokens, output_tokens):
    if any(type(n) is not int or n < 0 for n in (prompt_tokens, output_tokens)):
        raise ValueError("token counts must be nonnegative integers")
    blocks = (prompt_tokens + output_tokens + BLOCK_TOKENS - 1) // BLOCK_TOKENS
    return blocks * BLOCK_TOKENS * BYTES_PER_TOKEN

payload = 8192 * BYTES_PER_TOKEN
reservation = reserved_bytes(8192, 128)
assert payload == 2_684_354_560
assert reservation == 2_726_297_600
assert reserved_bytes(8193, 128) - reservation == 16 * BYTES_PER_TOKEN
assert reserved_bytes(0, 0) == 0
capacity = 8 * 1024**3
assert capacity // reservation == 3
print(f"payload: {payload / 1024**3:.2f} GiB; reservation: {reservation / 1024**2:.0f} MiB")
print("8 GiB capacity fits", capacity // reservation, "requests before other overhead")
for assumed_gb_s in (12.5, 25.0, 50.0):
    milliseconds = payload / (assumed_gb_s * 1e9) * 1000
    print(f"assumed {assumed_gb_s:4.1f} GB/s -> {milliseconds:6.1f} ms")

payload: 2.50 GiB; reservation: 2600 MiB
8 GiB capacity fits 3 requests before other overhead
assumed 12.5 GB/s ->  214.7 ms
assumed 25.0 GB/s ->  107.4 ms
assumed 50.0 GB/s ->   53.7 ms

For serial transfer with aggregate effective payload rate $R$, the transfer term is $B/R$. At an assumed 25 GB/s, it's 107.4 ms. If two independent links each carry one 1.25 GiB shard concurrently at 25 GB/s, the transfer finishes in roughly 53.7 ms before overhead. If both streams share a 25 GB/s bottleneck, dividing each shard by 25 GB/s falsely doubles the available capacity.

Reserve capacity before creating expensive state

D3 has 8 GiB of available KV capacity in our simplified budget. Three requests fit; four don't. A request counter alone misses that limit: one 100-token prompt and one 8K prompt each count as one request but consume very different memory.

Our teaching protocol reserves downstream capacity before prefill starts. This reduces wasted prompt computation when decode is full. It's a design choice, not a claim that every engine implements this exact sequence. Mooncake discusses early rejection and KV-aware scheduling to avoid completing prefill for work that can't be served.^{[7]Reference 7Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving.https://arxiv.org/abs/2407.00079}

Free capacity now also isn't a forecast of decode load when prefill finishes. Mooncake describes oscillation caused by that scheduling delay and uses a system-level future-load prediction to improve early rejection. A byte reservation protects a memory bound; it doesn't predict individual output length or guarantee a future latency target.^{[7]Reference 7Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving.https://arxiv.org/abs/2407.00079}

Admission must consider several constraints together:

An output reservation is a budget, not a prediction that generation ends there. Enforce an output cap or acquire more credit before crossing the reserved boundary. Cancellation, timeout, and completed requests must eventually release their charges, but only after the associated memory is safe to reuse.

A lease limits how long a reservation may become ready. Its expiry uses the destination's monotonic clock; the sender doesn't get to extend it by supplying a larger timestamp. After commit, active request ownership replaces this admission lease. Expiry revokes admission immediately. It doesn't prove that remote writes have stopped. Retired buffers can remain physically charged while their transfers drain.

Agree on the interpretation before moving bytes

The destination first compares the sender's manifest against a trusted reservation. The manifest describes the transfer; it doesn't authorize itself.

The short digests here are labels, not cryptographic examples. Use canonical token IDs from one owner rather than independently rendering chat templates at both workers. vLLM documents experimental reuse of prefill token IDs through kv_transfer_params for this purpose. Its documentation, rechecked October 10, 2026, scopes this path to /v1/chat/completions: messages remains required, and non-text content or echo causes the supplied IDs to be ignored. A proxy therefore can't assume those IDs bypass preprocessing for every request.^{[3]Reference 3Disaggregated Prefillinghttps://github.com/vllm-project/vllm/blob/main/docs/features/disagg_prefill.md} Sampling state, the last token or hidden state needed to start decode, and first-token ownership also belong in the engine-specific handoff. KV tensors alone aren't necessarily enough to resume generation.

Logical block 31 can live at physical block 904 in P7 and block 118 in D3. Local addresses needn't match. The mapping from logical positions to the correct destination shard must match. A connector may explicitly support repartitioning or layout conversion; budget and validate that conversion rather than silently reinterpreting bytes.

A transfer engine doesn't own the request

NIXL separates memory registration, connection-metadata exchange, buffer descriptors, and asynchronous transfer operations. An external conductor coordinates the workload and metadata path.^{[8]Reference 8NVIDIA Inference Transfer Library: Architecture and API Flowhttps://github.com/ai-dynamo/nixl/blob/main/docs/nixl.md} Serialize that small metadata, not gigabytes of tensors into JSON. Keep the payload in registered GPU or host memory supported by the connector.

The surrounding service still owns authorization, placement, retry policy, and output ownership. Protect the metadata channel, restrict descriptor exposure, and isolate registered regions. Remote keys and descriptors don't belong in application logs. RDMA isn't itself a promise of encryption or tenant isolation. Transport integrity and authenticated workload identity are separate from checking the model and layout fields.

Dynamo's development documentation describes backend-specific handoff metadata from prefill used to coordinate decode and NIXL movement.^{[9]Reference 9Disaggregated Servinghttps://docs.nvidia.com/dynamo/dev/knowledge-base/concepts/system-architecture/disaggregated-serving} Its experimental topology-aware routing can require the same transfer domain or prefer it through a cost bias. A preference doesn't guarantee same-rack or same-zone placement.^{[10]Reference 10Topology-Aware KV Transfer (experimental)https://docs.nvidia.com/dynamo/dev/knowledge-base/modular-components/router/topology-aware-kv-transfer} A lightly loaded decoder across a congested path can finish later than a busier nearby one. Evaluate the whole path, including GPU-to-NIC placement, staging, registration, conversion, and shared link contention.

That documentation scopes automatic prefill-router activation to the integrated dynamo.frontend path. Standalone Python routers need explicitly connected stages and handoff handling. Once a request has been dispatched to prefill, a prefill or handoff failure is returned to the request; it doesn't silently fall back through an aggregated decode-only path.^{[11]Reference 11Disaggregated Serving: Prefill and Decode Routinghttps://docs.nvidia.com/dynamo/dev/knowledge-base/modular-components/router/disaggregated-serving}

NIXL's backend guide gives two especially relevant limits: distinct transfer requests aren't automatically ordered, and the application must prevent simultaneous transfers from corrupting the same region. Releasing a transfer can involve asynchronous cancellation that hasn't yet succeeded.^{[12]Reference 12NIXL Backend Guidehttps://github.com/ai-dynamo/nixl/blob/main/docs/BackendGuide.md} Treat “cancel requested,” “operation stopped,” and “buffer reusable” as different events.

Backend configuration also changes failure evidence. The current UCX plugin defaults ucx_error_handling_mode to peer. Selecting none can make additional transport lanes eligible, but removes peer-failure reporting guarantees; pending operations may remain incomplete and unreported. A timeout or missing disconnect error is therefore no proof that a writer stopped.^{[12]Reference 12NIXL Backend Guidehttps://github.com/ai-dynamo/nixl/blob/main/docs/BackendGuide.md}

Timeout revokes ownership, not physical access

Suppose 70% of attempt 4's KV has arrived when its lease expires. The destination must reject commit. But returning those blocks to the allocator immediately could let an old remote write overwrite attempt 5's cache.

A conservative whole-prompt protocol has four distinct boundaries:

1. Reserve: charge capacity and record the expected manifest under a server-issued attempt.
2. Transfer: keep incoming blocks unavailable to attention, even if some are complete.
3. Commit: after compatibility, complete unique coverage, integrity, and connector-required visibility checks, grant one decoder ownership.
4. Retire and reclaim: stop admission and streaming, then wait for successful cancellation or completion and GPU-reader quiescence before reusing memory.

Advanced connectors can overlap layer-wise transfer and compute. That requires dependency tracking for each consumed region; it doesn't permit attention to read a missing region. The whole-prompt gate here is intentionally simpler.

A duplicate arrival mustn't count as a new block. Counting 1,024 notifications isn't equivalent to receiving 512 distinct blocks on each of two ranks. Likewise, a completed byte count doesn't validate layer coverage or a manifest's interpretation.

Execute the ownership rules locally

Download handoff_model.py. It serializes transitions in one Python process and stores block identities, not tensor data. The expected manifest comes from its trusted reservation; a received manifest must match it. finish(), abort_ack(), and consumer_stopped() represent trusted runtime evidence, not measurements that this script obtains.

The simulation supplies integer ticks from one trusted destination clock. It doesn't read a real clock or accept sender times as authority. Invalid durations and block identities are rejected before changing ownership or coverage. In a real service, bind every callback to the actual transfer handle and device region.

This short run expires a lease while writes are still possible. Credit remains unavailable until successful abort acknowledgement.

from assets.handoff_model import Manifest, Receiver, rejected

manifest = Manifest()
pool = Receiver(manifest.reserved)
key = pool.reserve(manifest, now=0, ttl=10)
pool.begin(key, manifest)
pool.arrive(key, rank=0, block=0)
rejected(lambda: pool.commit(key, now=10))  # exact expiry boundary
pool.arrive(key, rank=1, block=0)           # late write remains possible
rejected(lambda: pool.reclaim(key))
print("after expiry, free bytes:", pool.free)
pool.abort_ack(key)                       # assumed connector confirmation
pool.reclaim(key)
pool.reclaim(key)                         # duplicate cleanup is harmless
print("after quiescence, free bytes:", pool.free)

after expiry, free bytes: 0
after quiescence, free bytes: 2726297600

The longer checks reject eight manifest mismatches, exercise duplicate and missing blocks, reject streaming before commit, retain an old attempt's charge during replacement, and prevent retries after visible output. They also enumerate 720 orderings of two arrivals, completion, commit, cancellation, and reclamation.

from assets.handoff_model import run_checks

run_checks()

8 manifest mismatches, duplicate/missing blocks, stale attempts and expiry: pass
720 event orders: no partial commit or premature credit reuse
invalid leases, clock ticks, manifests and block identities: rejected

These checks validate the model's transitions. They don't establish distributed atomicity, authentication, memory visibility, or cancellation semantics in NIXL, vLLM, Dynamo, CUDA, or an RDMA backend. A deployed coordinator needs authoritative attempt ownership, serialization or compare-and-swap at commit, durable recovery rules, and a frontend that fences stale streams. A higher attempt number in a message alone doesn't stop an old writer or stream owner.

Failure modes and retry boundaries

Before output is visible, the service may retry if it can fence the old owner, preserve request semantics, and stay within its deadline and retry budget. This still costs compute and network traffic. Current vLLM configuration exposes fail and recompute policies for KV-load failure; those are local engine behaviors, not a complete resumable-stream protocol.^{[13]Reference 13KVTransferConfig API Referencehttps://docs.vllm.ai/en/latest/api/vllm/config/kv_transfer/}

After output may have reached the client, don't transparently restart at token 1. A lost acknowledgement can make delivery uncertain. Exact continuation requires an agreed cursor, token or frame deduplication, sampling state, and authoritative ownership. Without that protocol, return an explicit interrupted-stream error.

Retaining source KV briefly after commit can make retransmission cheaper, but consumes prefill memory. The retained copy needs its own accounted capacity and expiry. Never release a source buffer while a transport may still read it. NIXL's remote-agent invalidation helps manage disappearing peers, but the application still has to coordinate safe buffer lifetime.^{[8]Reference 8NVIDIA Inference Transfer Library: Architecture and API Flowhttps://github.com/ai-dynamo/nixl/blob/main/docs/nixl.md}

Measure the critical path, not a sum of unrelated percentiles

Measure time to first token (TTFT) from the defined request-start boundary to the defined first-token boundary. Joined spans explain that elapsed time. Without overlap, complete edge, routing, queue, prefill, handoff, and first-token terms can sum to it. With overlap, follow dependencies and waits; don't count prefill again inside a handoff term that already includes it. The union of active spans alone can omit uninstrumented queueing gaps, so it isn't an end-to-end timing measurement.

The synthetic timeline assumes prefill runs from 0 to 310 ms. Transfer either starts afterward or starts at 240 ms and overlaps its final 70 ms. Commit plus the first decode step takes 28 ms after both finish. No network measurements are implied.

prefill_end = 310.0
transfer_ms = 2_684_354_560 / 25e9 * 1000
commit_and_first_token = 28.0
for transfer_start in (310.0, 240.0):
    ready = max(prefill_end, transfer_start + transfer_ms)
    first_token = ready + commit_and_first_token
    print(f"transfer starts {transfer_start:.0f} ms: first token at {first_token:.1f} ms")
assert abs((310 + transfer_ms + 28) - (240 + transfer_ms + 28) - 70) < 1e-9

# Same overlapped work, shifted by an otherwise uninstrumented 25 ms queue.
request_start = 0.0
prefill_start = 25.0
prefill_end = prefill_start + 310.0
transfer_start = prefill_end - 70.0
first_token = max(prefill_end, transfer_start + transfer_ms) + 28.0
ttft = first_token - request_start
active_union = first_token - prefill_start  # these active intervals overlap
assert abs(ttft - active_union - 25.0) < 1e-9
print(f"with 25 ms queue: TTFT {ttft:.1f} ms; active-span union {active_union:.1f} ms")

transfer starts 310 ms: first token at 445.4 ms
transfer starts 240 ms: first token at 375.4 ms
with 25 ms queue: TTFT 400.4 ms; active-span union 375.4 ms

Phase p99 values don't generally add to end-to-end p99: the slowest prefill and slowest transfer may belong to different requests. Compute end-to-end quantiles from joined request traces. Keep request IDs in traces, not unbounded metric labels.

A host staging copy increases memory-system traffic, but doesn't necessarily increase network wire bytes. Specify where every byte counter is measured. Likewise, reduced handoff bandwidth may raise TTFT without changing steady-state inter-token latency (ITL), but shared-fabric contention can also slow collectives and hurt ITL. Treat that symptom as a clue, not proof.

Compare a mixed pool and a split pool under the same offered workload, output lengths, hardware budget, and SLO definitions. Report goodput, rejection, errors, and tail latency together. A service that rejects almost everything can have excellent latency for the few requests it accepts.

Try a changed workload

Follow-up questions

Evaluation rubric

• Calculate both logical KV payload and rounded allocation; distinguish aggregate link capacity from each stream's requested rate.
• Trace a timed-out attempt through fencing, late writes, quiescence, and reclamation without returning its byte credit early.
• Explain which guarantees the local model checks and which require actual connector, GPU, and coordinator evidence.