Deep Dive - DeepEP

A routed mixture-of-experts (MoE) layer can send one token to two experts on different GPUs. Each expert is a specialized neural-network block, and a gate weight determines how much its output contributes. Returning both outputs isn't enough: they must return to the correct source row, with that weight applied exactly once. A route can be wrong while every CUDA kernel reports success, and communication that consumes too many streaming multiprocessors (SMs) leaves fewer resources for the expert matrix multiply.

The previous chapter followed slime's rollout and training loop. That loop still has to move hidden-state rows whenever its policy is a sparse MoE. DeepEP (DeepEveryParallel) is DeepSeek's communication library for that hop. V2 wraps expert-parallel dispatch and combine in an ElasticBuffer backed by NCCL Gin, NVIDIA Collective Communications Library's device-side network interface. Its NVLink and remote direct memory access (RDMA) paths share routing metadata but differ in transport and synchronization. The archived NVIDIA SHMEM (NVSHMEM)-based V1 has a different API and resource model.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d[2]Reference 2DeepEP V1 Documentationhttps://github.com/deepseek-ai/DeepEP/blob/dd758caf451848bd150e1046af3d0a73e5fff38d/docs/legacy.md}

The code walkthrough follows V2 at dd758caf451848bd150e1046af3d0a73e5fff38d. That historical snapshot is retained for the routing and kernel examples. The V2.5 migration boundary below was checked on October 10, 2026. The runnable examples below exercise CPU routing and arithmetic only. CUDA snippets show integration contracts, not a completed DeepEP installation or a measured network run.

That reading snapshot isn't a deployment recommendation. At the September 23 check, main was a56d6156febcd9976e55adc85b5155bfac9f28f8, three commits ahead.

The first two add and document a system-scope memory fence before the Gin barrier when a logical scale-up domain spans NVLink and RDMA. A completed transport operation and visibility of preceding writes are different guarantees. The mixed-fabric ordering fix matters even when API shapes stay unchanged.

The third fixes request submission in experimental Engram fetching. We'll return to that separate path below. The public EP API and README benchmark table are unchanged across these three commits.^{[3]Reference 3DeepEP Mixed-Fabric Gin Barrier Orderinghttps://github.com/deepseek-ai/DeepEP/blob/01dc3aaac82068020353dce2c302e38153c0bfaa/deep_ep/include/deep_ep/common/comm.cuh[4]Reference 4DeepEP Engram RDMA Get Doorbell Fixhttps://github.com/deepseek-ai/DeepEP/commit/a56d6156febcd9976e55adc85b5155bfac9f28f8}

Current boundary: V2.5 changes the integration contract

On October 10, upstream main was 93eb6eb238127e96c6d7a4a625a6dad158348509. V2.5 landed on September 29. The four-token ledger, caller-owned gate multiplication, and source identity still teach the EP problem, but installing current main doesn't provide the historical ElasticBuffer interface. V2.5 splits it into dedicated buffers and removes V1, its legacy documentation, and the NVSHMEM dependency.^{[5]Reference 5DeepEP V2.5 API and Requirementshttps://github.com/deepseek-ai/DeepEP/blob/93eb6eb238127e96c6d7a4a625a6dad158348509/README.md[6]Reference 6DeepEP V2.5 Releasehttps://github.com/deepseek-ai/DeepEP/commit/def865146e95aa9ea223e277c0a8a4c6b81e5ac7}

The new event contract is an observable failure boundary. Keep the old five-value unpack while enabling deferred epilogues, and Python raises TypeError: cannot unpack non-iterable EventOverlap object. Wait once and retain the returned tensors and handle: the hook runs only on the first wait, so a second .wait() returns None instead of another copy of the results. Waiting still establishes a stream dependency, not CPU completion.^{[5]Reference 5DeepEP V2.5 API and Requirementshttps://github.com/deepseek-ai/DeepEP/blob/93eb6eb238127e96c6d7a4a625a6dad158348509/README.md}

This V2.5 fragment assumes an initialized EP group, BF16 CUDA input x, valid routing tensors, and caller-provided expert functions and alignment. It was checked against the public API, not executed on GPUs:

from deep_ep import EPBuffer

buffer = EPBuffer(
    group, num_max_tokens_per_rank=4096, hidden=7168, num_topk=8,
)
ready = buffer.dispatch(
    x, topk_idx=topk_idx, topk_weights=topk_weights, num_experts=128,
    expert_alignment=expert_alignment, do_expand=True,
    do_zero_padding=True, do_cpu_sync=True,
    async_with_compute_stream=True, defer_epilogue=True,
)
launch_independent_work()
recv_x, _, recv_w, handle = ready.wait()
# Expanded weights are one value per expert row; caller applies them once.
expert_output = launch_weighted_expanded_experts(recv_x, recv_w, handle)
combined_ready = buffer.combine(
    expert_output, handle=handle,
    async_with_compute_stream=True, defer_epilogue=True,
)
combined_x, _ = combined_ready.wait()

For decoding without CPU count synchronization, use do_cpu_sync=False and consume only the valid expert ranges in the handle. Zero padding clears alignment gaps; it doesn't make unused capacity valid tokens. Fresh gating still needs a fresh route handle.^{[5]Reference 5DeepEP V2.5 API and Requirementshttps://github.com/deepseek-ai/DeepEP/blob/93eb6eb238127e96c6d7a4a625a6dad158348509/README.md}

V2.5 also adds redundant-expert weight prefetch and FP32 gradient reduction over NVLink. These are collective exchanges within the NVLink domain, including ranks with no replicas. The caller still supplies the replication map, reroutes tokens, manages expert computation, and keeps storage alive. They don't turn the old balanced-gate SM estimate into a planner for skewed experts. Bucket collectives, PP, and Engram remain experimental; validate each specialized buffer separately.^{[5]Reference 5DeepEP V2.5 API and Requirementshttps://github.com/deepseek-ai/DeepEP/blob/93eb6eb238127e96c6d7a4a625a6dad158348509/README.md}

Why expert parallelism needs a communication layer

Start from one rank's view. A rank is a process in the communication group; here each rank controls one GPU. An MoE architecture replaces one dense feed-forward block with many expert blocks. A router chooses top-k experts for each token. Expert parallelism (EP) places different experts on different ranks, so each rank owns only a slice of the expert weights. That saves per-rank model memory, but every MoE layer now has to exchange token rows.

For one local rank, write hidden states as a matrix X with one row per token. The router returns topk_idx[t, j] and topk_weight[t, j]. Dispatch transports a token once per selected destination rank, even if that rank owns two of its selected experts. An optional expansion then creates one slot per expert. Local destinations still need layout and memory work, but not a remote link. After expert computation, combine reverses the route and sums supplied contributions. The caller must apply gate weights before that sum; passing topk_weights doesn't multiply activation rows automatically.

A fixed-size all-to-all assumes known per-peer counts. Variable-split collectives can handle uneven sizes, but callers must still compute those splits and pack the rows. Sparse routing also creates per-expert expansion and reverse-reduction work. DeepEP specializes that whole exchange rather than introducing the possibility of uneven communication. Four questions remain:

DeepEP doesn't implement the router or expert GEMMs. It accepts their rows and routing tensors, builds a layout handle, executes the exchange, and returns data in the layout the next kernel expects. The router, expert GEMMs, and training loop remain in the caller. That ownership boundary will matter when we diagnose a slow route versus a wrong route.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

View larger

A four-token routing ledger

Use two ranks and four tokens. Rank 0 owns tokens a and b; rank 1 owns c and d. There are four experts, two per rank, and each token selects two experts. Here the two selected experts always live on different ranks, so eight expert slots also require eight source-token/destination-rank rows, including local rows.

Rank 0 sends one copy of a to expert 3 and one copy of b to expert 2. It also receives c for expert 1 and d for expert 0. Rank 1 performs the mirror exchange. The counts happen to balance here; a real gate can send more rows from one rank than that rank receives.

Read each copy in two coordinate systems:

1. Source coordinates: (source_rank, token_index, topk_slot) identify the original row and expert choice.
2. Expert coordinates: (destination_rank, local_expert, expert_slot) identify where a receiving GEMM expects the row.

Dispatch creates this mapping. The returned EPHandle retains route indices and reverse-slot metadata, not gate values. Received gate tensors are separate outputs. Keep them aligned with expert rows when applying weights, then use the handle to return the weighted contributions.

View larger

V2's single interface: ElasticBuffer

V2 gives high-throughput and latency-oriented expert-parallel dispatch one interface: ElasticBuffer. The caller can pass an explicit byte count, or derive a 2 MiB-aligned size from maximum tokens per rank, hidden dimension, top-k, 8-bit floating-point (FP8) dispatch, topology, and reduction settings. Here 2 MiB means 2,097,152 bytes. The buffer then owns symmetric memory registered with NCCL.

Elastic describes this memory abstraction, not automatic infinite growth. The pinned V2 uses GPU-backed storage and can reserve CPU bytes for experimental Engram use. The README says V2 consumes more buffer space than V1 and that mixed GPU/CPU backing is still planned. Size the buffer from a real worst-case envelope rather than assuming it will grow on demand.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

Before dispatch can run, a caller needs a capacity estimate. get_buffer_size_hint gives a planning number, not an allocation. This unexecuted integration fragment assumes an initialized NCCL process group, BF16 or correctly scaled FP8 input, and valid routing tensors on a supported GPU:

from deep_ep import ElasticBuffer

required = ElasticBuffer.get_buffer_size_hint(
    group,
    num_max_tokens_per_rank=4096,
    hidden=7168,
    num_topk=8,
    use_fp8_dispatch=True,
)
buffer = ElasticBuffer(
    group,
    num_max_tokens_per_rank=4096,
    hidden=7168,
    num_topk=8,
    use_fp8_dispatch=True,
)
recv_x, recv_idx, recv_w, handle, event = buffer.dispatch(
    x,
    topk_idx=topk_idx,
    topk_weights=topk_weights,
    num_experts=128,
    async_with_compute_stream=True,
)

If num_sms or num_qps is 0, dispatch asks the analytical helpers for defaults. Pass an integer to override them; the pinned implementation doesn't accept None as the automatic sentinel. EventOverlap marks communication readiness when asynchronous mode is requested. Launch independent work, then wait on the event before reading recv_x. EP2048 is a documented domain limit; measure saturation and topology support on the target deployment.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

Reusing a buffer can avoid allocation and communicator setup. Reusing a handle is a stronger promise: the route itself must remain unchanged. The Python API asserts selected dimensions and reuses handle.topk_idx; it doesn't compare a newly computed route because cached dispatch requires topk_idx=None. Equal shapes don't establish equal routing.

Dispatch: layout, move, and epilogue

Ask what dispatch must preserve while rows are moving. V2 has three conceptual stages, even though kernels can overlap them: count and lay out rows by scale-up rank and local expert; move rows through local symmetric memory, NVLink, or NCCL Gin RDMA; then copy or expand rows into the receive layout used by expert GEMM.

The input contract makes those stages concrete. topk_idx has shape [num_tokens, num_topk], and topk_weights has the same shape with float values. Hidden states have shape [num_tokens, hidden] in BF16, or arrive as FP8 values plus scale factors. An expert index of -1 means that selection is unused. V2 returns received indices and weights beside the data so the next kernel can keep routing metadata attached.

The receive layout can be non-expanded or expanded. In non-expanded mode, one received row represents a source token with top-k metadata. In expanded mode, a token can occupy one slot per selected expert, and each local expert's segment is aligned for its GEMM. Alignment padding regularizes expert matrix shapes, but padding isn't a token and must not enter a combined row. Callers can use do_zero_padding when those gaps need to be cleared.

The copy epilogue turns metadata into expert input. In expanded mode, psum_num_recv_tokens_per_expert[i] is the aligned start of expert i plus its actual count, not simply an inclusive sum of padded counts. For counts [3, 1] and alignment 4, starts are [0, 4], these endpoints are [3, 5], and allocated expert segments total 8 slots. num_unaligned_recv_tokens_per_expert retains [3, 1]. Exact CPU counts require the CPU-synchronized path; no-sync callers must respect valid device counts rather than treating capacity as data.

The C++ launch path chooses dispatch for one scale-out domain and hybrid_dispatch when ranks span multiple scale-out groups. It passes the NCCL device communicator, symmetric window, buffer pointers, prefix arrays, slot metadata, timeout cycles, SM count, QP count, and topology indices into a just-in-time (JIT)-generated kernel. Reading csrc/kernels/elastic/dispatch.hpp beside deep_ep/buffers/elastic.py exposes the Python-to-kernel boundary.

Combine: reverse routing plus reduction

Once each receiving rank runs its local experts, rows are in expert order rather than source-token order. combine consumes handle.recv_src_metadata, per-scale-up prefix sums, and optional hybrid channel metadata to reverse that mapping and sum contributions. Activation reduction and optional return of top-k weights are separate operations in combine_reduce_epilogue.cuh. Local bypasses skip a remote transfer, but still participate in reduction.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

For token a, the caller supplies weighted outputs 0.70 × (1, 0) and 0.30 × (4, 0). Combine should return (1.90, 0). Supplying raw expert outputs returns (5, 0); weighting twice returns (0.85, 0). All three tensors have the same shape. Check numerical values as well as counts in a transport test.

The API exposes allow_multiple_reduction, which controls staging. Reducing within a rank or node can send fewer rows onward, but intermediate BF16 materialization introduces extra rounding. Disabling it avoids those additional reduction stages, potentially moving more data. The general reduction path accumulates in FP32 before casting to BF16; a no-bias path with at most two inputs uses packed BF16 addition. Neither "BF16 output" nor "FP32 accumulation" alone describes every path. Treat staging as a bandwidth/numerical tradeoff.

For a concrete rounding case, consider already-weighted contributions 256, 1, and -256. Accumulating all three in FP32 gives 1, which BF16 can represent. If a local stage first materializes 256 + 1 as BF16, round-to-nearest-even produces 256; adding -256 then gives 0. This is cancellation: two large terms nearly cancel, so losing a small term changes the answer.

The standard-library example below rounds FP32 bit patterns to BF16 without requiring PyTorch or a GPU. Its inputs are finite normal numbers chosen for this calculation; it isn't a general-purpose quantizer for NaNs or overflow. Notice that changing the intermediate storage changes the result even though the route and final dtype match:

import struct

def bf16(value):
    bits = struct.unpack("<I", struct.pack("<f", value))[0]
    # Keep the upper 16 bits. At a tie, retain an even low bit.
    rounded = (bits + 0x7FFF + ((bits >> 16) & 1)) & 0xFFFF0000
    return struct.unpack("<f", struct.pack("<I", rounded))[0]

single_stage = bf16(256.0 + 1.0 - 256.0)
local_stage = bf16(256.0 + 1.0)
two_stages = bf16(local_stage - 256.0)
assert single_stage == 1.0 and local_stage == 256.0 and two_stages == 0.0
print(f"one final BF16 cast: {single_stage:.0f}")
print(f"BF16 local subtotal: {local_stage:.0f}; final result: {two_stages:.0f}")

one final BF16 cast: 1
BF16 local subtotal: 256; final result: 0

This calculation demonstrates intermediate rounding. It doesn't execute DeepEP's transport or prove which reduction stages a particular GPU topology selects. Compare the deployed kernel's staging with the upstream reference before choosing a tolerance.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

Backward duality provides a compact consistency check:

For the unweighted route operators, dispatch and combine are transposes: copying rows forward becomes summing their gradients backward. The surrounding autograd wrapper must also differentiate expert functions and gate multiplication. It can't omit gate gradients merely because the communication calls reverse each other.

topk_weights can be two-dimensional in non-expanded mode or one-dimensional in expanded mode. bias supports zero, one, or two BF16 bias tensors for the combine epilogue. Its combined activation output is BF16. If a caller supplies a different layout or dtype, failure should happen at the contract boundary rather than after a silent reinterpretation.

A correct reduction still doesn't say which road the row took. Scale-up inside a node and scale-out across nodes wait on different links.

Topology: scale-up and scale-out are different roads

The four-token ledger now meets hardware. Scale-up means GPUs inside one node, normally connected by NVLink. Scale-out means GPUs in different nodes, normally connected through RDMA network interfaces. A row that moves quickly over NVLink may wait on a network interface card (NIC), while a path that saves SM work across RDMA can add needless synchronization inside a node.

V2 derives physical domains from NCCL. With hybrid mode enabled, num_scaleout_ranks represents RDMA groups and num_scaleup_ranks represents the local NVLink domain. With hybrid mode disabled, DeepEP treats the whole communicator as one logical scale-up domain and uses a direct path. The logical view lets kernels choose a hierarchy while retaining original global rank IDs.

View larger

Hybrid dispatch uses channel linked lists and per-channel token metadata. It can forward rows through a local scale-up group after the RDMA phase. The code budgets CUDA warps, groups of 32 scheduled GPU threads, for scale-out and forward work separately, then launches a clustered kernel to overlap communication with nearby compute. A flat all-to-all comparison misses those extra ownership and synchronization costs.

When a network has multiple rails, NCCL Gin's connection type matters. The V2 backend asks NCCL for a NCCL_GIN_CONNECTION_RAIL context in hybrid mode and NCCL_GIN_CONNECTION_FULL in direct mode. If Gin isn't available for the selected topology, initialization asserts instead of quietly choosing an incompatible path, unless EP_DISABLE_GIN explicitly changes behavior. Treat that assertion as a topology validation step. A path that initializes cleanly can still publish a bandwidth number that doesn't match the NIC. The next question is which bytes that number counts.

Logical bandwidth is not a NIC counter

DeepEP's README calls its performance numbers logical bandwidth. Logical bytes count the communication contract seen by the algorithm, including local-rank traffic. Physical link counters count bytes that crossed NVLink or a NIC. Those measurements answer different questions, so a profiler needs both labels.

Suppose four ranks exchange 100 MiB of token payload according to an EP layout. If half the rows stay on source GPUs, the logical operation still accounts for 100 MiB while the RDMA port may see only the remote half. If that payload crosses an NVLink hop during a hybrid forward, a physical NVLink counter can count copies that logical bandwidth reports once. A speedup without its byte denominator is not a reproducible measurement.

For a hypothetical 1 ms interval, 100 MiB corresponds to 104.8576 decimal GB/s; 50 MiB corresponds to 52.4288 GB/s. A difference between those figures may come from byte accounting. Real counters also include their own protocol and direction conventions. Use dispatch-plus-expert-plus-combine latency to evaluate layer performance alongside communication bandwidth.

Read the README tables as reports for their stated shapes: 8K tokens per batch, hidden size 7168, top-8 routing, FP8 dispatch, and BF16 combine. For reproduction, inspect the test harness too; its per-rank count can be max(1, args.num_tokens - rank). Keep the README's batch description separate from the harness's rank-specific counts. The table reports these logical-bandwidth snapshots:^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

These rows don't establish a V1-to-V2 speedup. A comparison also needs the exact V1 workload, topology, tuning, timing and correctness settings. Record token count, hidden size, top-k, dtype, SM count, queue-pair count, and bandwidth denominator before comparing runs.

NCCL Gin V2 backend

NCCL Gin is a device-side communication interface for issuing network operations from GPU code. V2 reuses an existing NCCL communicator instead of asking an application to construct a separate transport world. The C++ backend queries communicator properties, requests Gin contexts and signals, registers a symmetric memory window, and stores a device communicator pointer for JIT kernels.

Follow initialization as a chain of ownership transfers:

1. Create or receive the host NCCL communicator.
2. Query physical rank domains and Gin capability.
3. Request QP contexts, queue depth, traffic class, and signal count.
4. Choose rail or full connection based on direct or hybrid mode.
5. Allocate symmetric GPU and optional CPU memory.
6. Register a window and obtain NVLink peer pointers.
7. Build a device communicator that JIT kernels can call.

The signal count includes rank barriers and custom communication notifications. EP_OVERRIDE_RDMA_SL can place V2 traffic on a chosen InfiniBand service level, while NCCL_GIN_CROSS_NIC=0 changes symmetric-memory handle reuse behavior for multi-plane systems. These are cluster integration knobs, not per-request model settings.

Keep ownership explicit. DeepEP owns layout, kernels, and its window lifecycle. NCCL owns communicator and network primitives. A model runtime still owns process-group formation, rank assignment, and shutdown order. Aborting a communicator while an EventOverlap hook or JIT kernel still references its window is a lifecycle bug, not a routing bug.

EPHandle: route metadata with a lifetime

After dispatch, ask what survives the network transfer. EPHandle is returned by dispatch and consumed by combine. Its fields expose the route in a form that bridges device kernels, CPU scheduling, and cached inference:

A cached handle can skip layout recomputation for a replay of the same route, such as a corresponding backward operation. A new decode token usually has a newly computed route; equal batch size doesn't make the old handle reusable. The cached path skips CPU count discovery, so the caller must establish route and layout compatibility before using it.

V2's deterministic mode sorts received rows after the kernel. Non-expanded mode sorts data, weights, indices, and source metadata. Expanded mode sorts rows within each expert and updates slot pointers without permuting the metadata table in the same way. The sort key uses source-global order and expert identity, so equal inputs can present the same expert order even when arrival order differs.

Determinism has a cost: sorting and metadata work can constrain asynchronous behavior. Enable it for correctness tests, reproducible training slices, or debugging route drift. Measure it separately from the non-deterministic throughput path.

EventOverlap: a stream contract

CUDA streams let communication and compute overlap, but a Python return doesn't make an output tensor safe to read. EventOverlap wraps a CUDA event and can register a hook to run after the event is waited on. First launch independent work, then make the compute stream wait before it reads received rows:

recv_x, recv_idx, recv_w, handle, event = buffer.dispatch(
    x,
    topk_idx=topk_idx,
    topk_weights=topk_weights,
    num_experts=num_experts,
    async_with_compute_stream=True,
)

launch_independent_work()
event.current_stream_wait()
# Non-expanded layout: caller computes experts, applies each gate once,
# and sums experts belonging to this destination rank into one BF16 row.
expert_output = launch_weighted_local_experts(recv_x, recv_idx, recv_w)

combined_x, _, combine_event = buffer.combine(
    expert_output,
    handle=handle,
    async_with_compute_stream=True,
)
combine_event.current_stream_wait()

previous_event can make a communication kernel wait for an upstream CUDA event. allocate_on_comm_stream controls ownership of newly allocated tensors when that dependency is used. A host synchronization inserted between the event and expert work can erase the overlap the caller intended to gain. Profile stream waits beside kernel durations, and keep readiness separate from route correctness.

This code sketch leaves launch_weighted_local_experts as the caller's expert-integration placeholder. current_stream_wait() enqueues a dependency on the current stream; it does not synchronize the host. In deterministic mode it also triggers a sorting hook on the waiting stream. Establish readiness on each consumer stream before releasing the wrapper. Keep allocation lifetimes and all consumer dependencies explicit. A waited tensor can still hold incorrect values if the wire format and reduction format disagree.

Precision: FP8 on the wire, BF16 for the reduction

DeepEP accepts BF16 hidden states for the baseline path. In FP8 dispatch mode, input is a tuple of FP8 values and scale factors. The receiving expert integration consumes that representation according to its GEMM contract. Combine takes and returns BF16 activation rows; gate multiplication belongs in the caller's expert path, not in the transport.

Lower payload precision doesn't remove numerical validation. Scale-factor layout, hidden alignment, and expert GEMM expectations must agree. A kernel can move bytes correctly while a mismatched scale stride corrupts activations. The reference quantizer uses one float scale per 128 values: a 7,168-element FP8 row takes 7,168 payload bytes plus 224 scale bytes, versus 14,336 bytes for BF16. Routing metadata and protocol overhead are additional, so this isn't exactly a 2× wire-byte reduction.

use_tma_aligned_col_major_sf selects an optional scale-factor layout suitable for Tensor Memory Accelerator (TMA) paths. Read the model's quantization and GEMM contract before enabling it. DeepEP's source handles scale metadata, but it doesn't decide which quantization scheme an entire model should use.

For post-training or reinforcement learning (RL) on large sparse models, this boundary matters twice. Rollout inference may favor low-latency FP8 dispatch, while training may need BF16 or a different accumulator policy. Keep dtype and scale metadata in the experiment receipt. Otherwise, a reward-curve change after a dtype switch confounds communication changes with model numerics.

Analytical SM and QP sizing

V2's get_theoretical_num_sms estimates communication SMs from topology bandwidth and a balanced gate model. Before trusting that number, ask what it can observe: expected top-k destinations, read and write work, NVLink traffic, RDMA traffic, and the link bottleneck, but not your future gate histogram.

Treat the helper as an analytical starting point and measure the workload before selecting a runtime value. It multiplies the raw bandwidth-derived estimate by 1.25, rounds up to an even count of at least 4, and clamps to the device's multiprocessor count. When compute-communication overlap is preferred, it stays near that calculated minimum; otherwise, it raises the floor to 64 SMs for maximum standalone throughput. The implementation assumes balanced routing, whereas production DeepSeek-style gates can be group-limited and asymmetric. Inspect the actual token distribution before trusting the estimate, and pass num_sms explicitly when your application overlap budget demands tighter bounds.

Queue-pair (QP) sizing follows a related rule. Direct mode encourages roughly one QP per communication SM, capped at nine in the pinned helper (min(num_sms, 8 + 1)). Hybrid mode encourages num_sms * 16 + 1 QPs so channels and notification work can have independent queues, then caps that value by allocated QPs. More QPs consume resources and can increase RDMA doorbell ringing. Use the theoretical count as a starting point, then measure queue pressure and tail latency.

The balanced-gate model counts distinct destinations, not expert selections. With four experts split evenly across two ranks and two distinct experts sampled uniformly, six expert pairs are possible. Two pairs stay on one destination rank; four touch both. Expected destination count is therefore $(2×1 + 4×2)/6 = 5/3$, not 2. The helper uses the equivalent formula $G[1-\binom{E-E/G}{k}/\binom{E}{k}]$, for $E$ experts evenly divided among $G$ groups.

This standard-library check compares that formula with exhaustive enumeration and reproduces the pinned QP helper's arithmetic. Measure queue count and throughput on the target hardware before selecting a configuration:

from itertools import combinations
from math import comb, isclose

def expected_destinations(experts, groups, topk):
    if (any(type(v) is not int for v in (experts, groups, topk))
            or experts < 1 or groups < 1 or experts % groups
            or not 0 <= topk <= experts):
        raise ValueError("Invalid balanced routing model")
    return groups * (1 - comb(experts - experts // groups, topk) / comb(experts, topk))

def queue_count(sms, allocated, hybrid):
    if type(sms) is not int or type(allocated) is not int or min(sms, allocated) < 1:
        raise ValueError("Expected positive SM and QP counts")
    requested = sms * 16 + 1 if hybrid else min(sms, 9)
    return min(requested, allocated)

pairs = list(combinations(range(4), 2))
enumerated = sum(len({expert // 2 for expert in pair}) for pair in pairs) / len(pairs)
assert isclose(enumerated, expected_destinations(4, 2, 2))
print(f"expected destination ranks: {enumerated:.6f}")
print(f"12 SMs, 128 allocated QPs: direct={queue_count(12, 128, False)}, hybrid={queue_count(12, 128, True)}")

expected destination ranks: 1.666667
12 SMs, 128 allocated QPs: direct=9, hybrid=128

The hybrid request is 193 queues, but only 128 are allocated. The helper caps at 128; it doesn't allocate the missing 65. Group-limited or skewed gates violate the uniform-subset assumption behind the expected-destination formula. The source explicitly warns against using its balanced SM model for V3.0's group-limited gate.

JIT compilation is part of the runtime

DeepEP V2 compiles kernels at runtime. The Python or C++ launch code assembles template parameters such as topology, expansion mode, hidden bytes, expert alignment, number of warps, SM count, and QP count. The JIT compiler parses required headers, invokes NVCC or the configured compiler, caches the resulting module, and launches it with a generated configuration.

That choice avoids compiling every shape specialization during installation, but moves a failure boundary into first use. A missing CUDA compiler, incompatible NCCL headers, or unsupported instruction can surface at the first dispatch. The pinned JIT selects its architecture from the active device and compiler support; TORCH_CUDA_ARCH_LIST controls extension/build targets, not every runtime JIT specialization. Separate compile, cold-start, and steady-state timings.

Relevant environment switches include:

For a launch, pre-warm every required shape on every worker or include compilation time in startup readiness. A cache mounted from one host may contain code for a different device capability. Treat cache keys and compiler versions as part of the deployment artifact. Those steps still assume the V2 API. What does the integration do if it was written against the archived library?

V2 and V1 are different products

DeepEP V1 is archived documentation, not a hidden compatibility mode for the V2 API. V1 exposes a Buffer, uses NVSHMEM for internode and low-latency methods, asks callers to provide separate NVLink and RDMA sizes, and has distinct normal and low-latency kernels. Its low-latency path can use a hook to defer an RDMA receive without occupying SMs. V2 uses ElasticBuffer, NCCL Gin, unified high-throughput and low-latency APIs, and larger scale-up and scale-out domains.^{[2]Reference 2DeepEP V1 Documentationhttps://github.com/deepseek-ai/DeepEP/blob/dd758caf451848bd150e1046af3d0a73e5fff38d/docs/legacy.md}

V1 source remains relevant when reading old DeepSeek-V3 integrations or reproducing a historical benchmark. Check the current V2 source and deployment prerequisites for CUDA, PyTorch, NVSHMEM, and low-latency support. V1 performance tables use H800 and CX7 setups; compare them with V2's SM90 and SM100 tables only after matching shapes and links.^{[2]Reference 2DeepEP V1 Documentationhttps://github.com/deepseek-ai/DeepEP/blob/dd758caf451848bd150e1046af3d0a73e5fff38d/docs/legacy.md[7]Reference 7DeepSeek-V3 Technical Report.https://arxiv.org/abs/2412.19437}

Migration order matters: port the route contract first, then port the transport. Replace Buffer layout calls with ElasticBuffer handles, verify NCCL Gin availability, resize memory, and rerun deterministic correctness tests before measuring throughput. Don't infer compatibility from a successful import alone.

Experimental PP, CP, and Engram boundaries

The pinned V2 repository includes experimental primitives beyond EP. Treat each as a separate boundary rather than as another dispatch mode:

• Pipeline parallel (PP): send and receive tensors through symmetric memory. The README lists 0-SM PP with RDMA as an experimental path. It doesn't replace Megatron's full pipeline scheduler.
• Context parallel (CP): use copy-engine-oriented paths for context slices, including a 0-SM copy-engine variant. It doesn't define attention semantics or sequence partitioning policy.
• Engram: fetch remote embedding entries by integer index through RDMA, with optional CPU-backed storage and TMA-aligned scale factors. Its API doesn't expose key search or an attention KV-cache manager. FP8 scale factors are globally replicated at write time and gathered locally at fetch time; the remote get transfers the entry data.
• AGRS: experimental all-gather and reduce-scatter sessions over NVLink symmetric memory.

The README labels Engram, PP, and CP experimental. Their shared allocation and event machinery with ElasticBuffer makes them useful systems experiments; validate each primitive's maturity, failure handling, and target workload before production use.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

The September 16 Engram change is a useful example of a transport failure that a routing ledger can't detect. A NIC doorbell tells the network device that queued requests are ready to submit. The older fetch loop could leave the tail of aggregated gets without that notification. The newer kernel defers one get per used peer, issues it without aggregation after the other gets, and then records the asynchronous flush. Correct indices don't prove those bytes were submitted. Check the fetch implementation and test actual arrival on your cluster before relying on this experimental path.^{[4]Reference 4DeepEP Engram RDMA Get Doorbell Fixhttps://github.com/deepseek-ai/DeepEP/commit/a56d6156febcd9976e55adc85b5155bfac9f28f8}

For a large RL system, keep these boundaries explicit. A PP send can be healthy while EP RDMA is congested. An Engram fetch can consume the same QP or memory budget that dispatch sizing assumed was free. A CP copy can change stream dependencies. Share a buffer only when its maximum concurrent sessions and lifetimes are accounted for.

Requirements and cluster checks

The pinned V2 README lists Hopper (SM90) GPUs or architectures with SM90 Parallel Thread Execution (PTX) support, CUDA 12.3 or newer for SM90, PyTorch 2.10 or newer, and NCCL 2.30.4 or newer. NVLink is expected for intranode communication and RDMA for internode communication. The repository reports full InfiniBand testing and theoretical compatibility with RDMA over Converged Ethernet (RoCE). Recheck these requirements against the source revision you deploy; later revisions can change the support floor.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

Before a multi-node job starts, check the path from hardware to buffer:

The NCCL minimum isn't the whole version contract. In the pinned backend, a build against headers older than NCCL 2.31 must load exactly the same NCCL version at runtime. The 2.31-and-newer branch requests a runtime-version device communicator instead. Two installations that both satisfy 2.30.4+ can therefore still fail initialization if headers and the loaded library differ. Inspect the loaded version in buffer debugging output, rather than checking only the wheel installed in the image.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

Don't run pip install on a login node and assume the worker image matches. The source's JIT cache and NCCL library path are runtime dependencies. Include their versions, topology dump, and EP_BUFFER_DEBUG=1 initialization output in a launch receipt. Redact credentials and private host details before sharing it.

Applications in giant-model post-training

Every generated token in a sparse MoE policy or teacher can trigger another dispatch and combine. The previous slime study showed that rollout, reward, and training must agree on tokens and weight versions. Those sampled trajectories still pass through MoE layers. Rollout needs fast expert routing. Training needs exact route metadata and reproducible gradients. A slow all-to-all multiplies across generated tokens, while an incorrect or stale handle can corrupt every subsequent update.

Use DeepEP to compare three placement patterns. For each one, ask which phase owns the communicator and which receipt would reveal its cost:

The communication library doesn't solve policy staleness, reward hacking, or checkpoint synchronization. It can become the lowest-level source of latency and numerical mismatch in those systems. Pair route metrics with rollout weight versions, expert-load histograms, and reward receipts. If reward falls after a topology change, separate transport delay, dtype error, and algorithm behavior before changing the policy.

A runnable routing ledger

Work token a by hand before touching code. Its hidden state is (1.0, 0.0). Give expert $k$ a toy map that multiplies by $k + 1$, so expert 0 returns (1.0, 0.0) and expert 3 returns (4.0, 0.0). Predict the combined first coordinate before reading the equation: gate 0.70 keeps most of expert 0's value, while gate 0.30 adds a smaller expert 3 contribution. The caller weights each expert output before combine sums the results:

Expert 0 lives on rank 0, so that copy of a is local. Expert 3 lives on rank 1, so the second copy is remote. The standard-library model below builds expanded receive slots, pads each expert to four rows, computes and weights outputs in expert order, and combines them using the saved inverse map. It never recomputes the original routes during combine. Shuffling the returned rows must not change the answer.

from dataclasses import dataclass
import math

HIDDEN = {
    "a": (1.0, 0.0),
    "b": (0.0, 1.0),
    "c": (1.0, 1.0),
    "d": (2.0, -1.0),
}
ROUTES = {
    "a": [(0, 0.70), (3, 0.30)],
    "b": [(2, 0.40), (1, 0.60)],
    "c": [(1, 0.55), (3, 0.45)],
    "d": [(0, 0.25), (2, 0.75)],
}
SOURCE_RANK = {"a": 0, "b": 0, "c": 1, "d": 1}
EXPERTS_PER_RANK = 2

def owner_rank(expert):
    return expert // EXPERTS_PER_RANK

def expert_map(expert, hidden):
    scale = expert + 1
    return (hidden[0] * scale, hidden[1] * scale)

@dataclass(frozen=True)
class Row:
    token: str
    topk_slot: int
    hidden: tuple[float, float]
    weight: float
    path: str

def dispatch(hidden, routes, sources, alignment=4):
    if type(alignment) is not int or alignment < 1:
        raise ValueError("Invalid expert alignment")
    if set(hidden) != set(routes) or set(hidden) != set(sources):
        raise ValueError("Token sets differ")
    recv = {expert: [] for expert in range(4)}
    inverse = {}
    rank_rows = set()
    for token, choices in routes.items():
        if sources[token] not in (0, 1):
            raise ValueError("Invalid source rank")
        if len(hidden[token]) != 2 or not all(math.isfinite(v) for v in hidden[token]):
            raise ValueError("Expected a finite two-coordinate row")
        seen = set()
        for topk_slot, (expert, weight) in enumerate(choices):
            if expert == -1:  # unused router slot, not expert Python index -1
                continue
            if type(expert) is not int or not 0 <= expert < 4 or expert in seen:
                raise ValueError("Invalid or duplicate expert")
            if not math.isfinite(weight) or weight < 0:
                raise ValueError("Invalid gate weight")
            seen.add(expert)
            destination = owner_rank(expert)
            rank_rows.add((token, destination))
            slot = len(recv[expert])
            inverse[expert, slot] = (token, topk_slot)
            path = "local" if destination == sources[token] else "remote"
            recv[expert].append(Row(token, topk_slot, hidden[token], weight, path))
    for rows in recv.values():
        rows.extend([None] * (-len(rows) % alignment))
    return recv, inverse, rank_rows

def run_experts(recv):
    results = []
    for expert, rows in recv.items():
        for slot, row in enumerate(rows):
            if row is None:  # padding never acquires a source identity
                continue
            raw = expert_map(expert, row.hidden)
            weighted = tuple(row.weight * value for value in raw)
            results.append(((expert, slot), weighted))
    return results

def combine(results, inverse, source_tokens):
    combined = {token: [0.0, 0.0] for token in source_tokens}
    seen = set()
    for key, value in results:
        if key not in inverse or key in seen:
            raise ValueError("Unexpected, padded, or duplicate result slot")
        if len(value) != 2 or not all(math.isfinite(v) for v in value):
            raise ValueError("Invalid expert output")
        seen.add(key)
        token, _topk_slot = inverse[key]
        for column in range(2):
            combined[token][column] += value[column]
    if seen != set(inverse):
        raise ValueError("Missing expert results")
    return {token: tuple(value) for token, value in combined.items()}

recv, inverse, rank_rows = dispatch(HIDDEN, ROUTES, SOURCE_RANK)
results = run_experts(recv)
combined = combine(list(reversed(results)), inverse, HIDDEN)
assert math.isclose(combined["a"][0], 1.90)
assert len(inverse) == 8 and sum(map(len, recv.values())) == 16

for token in HIDDEN:
    x, y = combined[token]
    print(f"{token}: ({x:.2f}, {y:.2f})")
for expert in range(4):
    copies = ", ".join(f"{row.token} {row.path}" for row in recv[expert] if row is not None)
    print(f"E{expert} <- {copies}")

a: (1.90, 0.00)
b: (0.00, 2.40)
c: (2.90, 2.90)
d: (5.00, -2.50)
E0 <- a local, d remote
E1 <- b local, c remote
E2 <- b remote, d local
E3 <- a remote, c local

This inverse map is a Python teaching structure that makes routing steps visible; DeepEP's binary EPHandle uses its own layout. Gate values live in the received rows and are applied in run_experts; combine only sums supplied vectors. Missing, duplicated, or padded return slots fail explicitly. GPU operations avoid these expensive Python-style checks, so retain an independent reference in integration tests.

Now move a's first selection from expert 0 to expert 2. Experts 2 and 3 both live on rank 1: there are still eight expert assignments, but only seven source-token/destination-rank rows. The receiver expands a for the two local experts after transport. A -1 selection contributes neither a transport destination nor an expert slot; don't accidentally treat it as Python's last expert.

Run this continuation in the same session as routing_ledger.py. It checks that deduplication changes transport rows without dropping expert work, then removes the second selection for a. Finally, it deliberately returns an alignment-padding slot. Predict whether combine can assign that slot a source token:

same_rank_routes = {**ROUTES, "a": [(2, 0.70), (3, 0.30)]}
recv2, inverse2, destinations2 = dispatch(HIDDEN, same_rank_routes, SOURCE_RANK)
assert len(inverse2) == 8 and len(destinations2) == 7
assert math.isclose(combine(run_experts(recv2), inverse2, HIDDEN)["a"][0], 3.30)

masked_routes = {**ROUTES, "a": [(0, 0.70), (-1, 0.30)]}
recv3, inverse3, destinations3 = dispatch(HIDDEN, masked_routes, SOURCE_RANK)
assert len(inverse3) == 7 and len(destinations3) == 7
assert combine(run_experts(recv3), inverse3, HIDDEN)["a"] == (0.70, 0.0)
# The reference doesn't renormalize the remaining gate; masking is separate.

try:
    combine(results + [((0, 2), (999.0, 999.0))], inverse, HIDDEN)
except ValueError as error:
    assert str(error) == "Unexpected, padded, or duplicate result slot"
else:
    raise AssertionError("Padding acquired a source identity")
print("same-rank experts: 8 expert assignments, 7 destination rows")
print("masked selection: a=(0.70, 0.00); padded result rejected")

same-rank experts: 8 expert assignments, 7 destination rows
masked selection: a=(0.70, 0.00); padded result rejected

The same-rank route still needs both experts: 0.70 × 3 + 0.30 × 4 = 3.30. Sending one row to a rank saves transport duplication; it doesn't turn top-2 expert routing into top-1. A masked expert and a padded storage slot are different absences, but neither should create a returned contribution.

Strengths and weaknesses

DeepEP stays small at its boundary. It exposes a compact EP API and specializes kernels for current GPU and RDMA paths. V2's unified buffer, analytical resource estimates, NCCL communicator reuse, deterministic option, and JIT compilation make that boundary inspectable. Each choice also leaves a limit to measure:

A test failure can expose a deployment contract or a software defect. Larger V2 buffers may trade memory for broader topology support, while minimum SM counts can favor overlap over standalone benchmark scores. A failed Gin assertion can reveal that network deployment does not satisfy the selected mode. Diagnose against the contract before patching a kernel.

Team, contributors, governance, and licenses

The repository is published under the DeepSeek organization. Its citation lists Chenggang Zhao, Shangyan Zhou, Liyue Zhang, Chengqi Deng, Zhean Xu, Yuxuan Liu, Kuai Yu, Jiashi Li, and Liang Zhao. The README also thanks NCCL contributors and the NCCL team for V2 Gin support. These names describe the pinned source snapshot's credited contributors, not a permanent ranking of current maintainers.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

The source's MIT license permits use, modification, and distribution with the required notice. It doesn't grant a license to NCCL or NVSHMEM binaries, model checkpoints, benchmark datasets, or network firmware. An image should inventory those dependencies separately. At the old V2 pin, NVSHMEM remains needed for legacy methods, so removing every NVSHMEM package from that image can break a V1 compatibility path even when V2 EP itself uses NCCL Gin. This dependency warning applies to that historical image.^{[8]Reference 8DeepEP MIT Licensehttps://github.com/deepseek-ai/DeepEP/blob/dd758caf451848bd150e1046af3d0a73e5fff38d/LICENSE[9]Reference 9DeepEP NVSHMEM Installation Guidehttps://github.com/deepseek-ai/DeepEP/blob/dd758caf451848bd150e1046af3d0a73e5fff38d/docs/nvshmem.md}

Use the pinned README for project identity and its benchmark tables. The DeepSeek-V3 report motivates expert-parallel communication and group-limited routing; the later V2 kernels need sources from their own revision.^{[7]Reference 7DeepSeek-V3 Technical Report.https://arxiv.org/abs/2412.19437}

Benchmark and failure-mode checklist

A throughput number is evidence only when another engineer can reconstruct its bytes, route, timing, and correctness check. Before trusting a number, capture a receipt with:

1. DeepEP commit and JIT/compiler versions.
2. GPU architecture, GPU count, NVLink map, NIC model, and rail topology.
3. CUDA, PyTorch, NCCL, and optional NVSHMEM versions.
4. Tokens per rank, hidden size, top-k, expert count, and expert alignment.
5. Gate histogram, local-bypass rate, and deterministic setting.
6. FP8/BF16 scale layout, SM count, QP count, and hybrid/direct mode.
7. Warm-up policy, JIT time, dispatch time, combine time, and end-to-end layer time.
8. Logical-byte formula and physical-link counters.

Common failures map to specific boundaries:

The most dangerous failure is a plausible tensor with wrong provenance. A profiler shows kernel time, stream waits, and link traffic. Verify row-to-source-token mapping with route checksums, per-expert counts, and deterministic small-cluster tests before launching a long RL job.

Source-reading map

This map targets the historical V2 commit, where these paths exist. Read one vertical slice instead of opening every kernel at once. Begin with the public contract, follow one route into the kernels, then check the legacy boundary:

At each boundary, write down the authoritative state. Router output owns expert choices. EPHandle owns source-to-slot metadata. NCCL owns communicator and window state. The event owns readiness. The expert GEMM owns its local output. Combine should never guess any of those values from decoded text or a second route calculation. That ownership map is the shortest path from a profiler symptom to the component that can explain it.

Reviewing an integration

Evaluation rubric

• Foundational: Trace token a through both experts and derive 1.90 after exactly one gate multiplication. Distinguish received gate tensors from the handle's route metadata.
• Intermediate: Explain destination-rank deduplication, expert padding, cached-route validity, and the stream wait that must precede reading dispatched rows.
• Advanced: Reconcile logical bytes with physical traffic, qualify balanced-gate resource estimates, and separate CPU reference agreement from GPU memory-ordering and distributed correctness.

Follow-up questions