Deep Dive - DeepEP

A routed mixture-of-experts (MoE) layer can send one token to two experts on different GPUs. Returning both outputs isn't enough: they must return to the correct source row, with the gate applied exactly once. A route can be wrong while every CUDA kernel reports success, and communication that consumes too many streaming multiprocessors (SMs) leaves fewer resources for the expert matrix multiply.

The previous chapter followed slime's rollout and training loop. That loop still has to move hidden-state rows whenever its policy is a sparse MoE. DeepEP (DeepEveryParallel) is DeepSeek's communication library for that hop. V2 wraps expert-parallel dispatch and combine in an ElasticBuffer backed by NCCL Gin, NVIDIA Collective Communications Library's device-side network interface. Its NVLink and remote direct memory access (RDMA) paths share routing metadata but differ in transport and synchronization. The archived NVIDIA SHMEM (NVSHMEM)-based V1 has a different API and resource model.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d[2]Reference 2DeepEP V1 Documentationhttps://github.com/deepseek-ai/DeepEP/blob/dd758caf451848bd150e1046af3d0a73e5fff38d/docs/legacy.md}

The code walkthrough is pinned to dd758caf451848bd150e1046af3d0a73e5fff38d, checked on September 2, 2026. The runnable examples below exercise CPU routing and arithmetic only. CUDA snippets show integration contracts, not a completed DeepEP installation or a measured network run.

That reading snapshot isn't a deployment recommendation. At the same check, main was 01dc3aaac82068020353dce2c302e38153c0bfaa, two commits ahead. Those commits add and document a system-scope memory fence before the Gin barrier when a logical scale-up domain spans NVLink and RDMA. A completed transport operation and visibility of preceding writes are different guarantees; the mixed-fabric ordering fix matters even when API shapes stay unchanged.^{[3]Reference 3DeepEP Mixed-Fabric Gin Barrier Orderinghttps://github.com/deepseek-ai/DeepEP/blob/01dc3aaac82068020353dce2c302e38153c0bfaa/deep_ep/include/deep_ep/common/comm.cuh}

Before reading kernels, answer one question: what must cross ranks in an MoE layer?

Answer

Hidden-state rows and enough routing metadata to identify destination experts and restore source positions. Gate values can travel as metadata too, but the caller decides where to apply them. Expert parameter matrices stay on their owning ranks.

Why expert parallelism needs a communication layer

Start from one rank's view. An MoE architecture replaces one dense feed-forward block with many expert blocks. A router chooses top-k experts for each token. Expert parallelism (EP) places different experts on different ranks, so each rank owns only a slice of the expert weights. That saves per-rank model memory, but every MoE layer now has to exchange token rows.

For one local rank, write hidden states as a matrix X with one row per token. The router returns topk_idx[t, j] and topk_weight[t, j]. Dispatch transports a token once per selected destination rank, even if that rank owns two of its selected experts. An optional expansion then creates one slot per expert. Local destinations still need layout and memory work, but not a remote link. After expert computation, combine reverses the route and sums supplied contributions. The caller must apply gate weights before that sum; passing topk_weights doesn't multiply activation rows automatically.

A fixed-size all-to-all assumes known per-peer counts. Variable-split collectives can handle uneven sizes, but callers must still compute those splits and pack the rows. Sparse routing also creates per-expert expansion and reverse-reduction work. DeepEP specializes that whole exchange rather than introducing the possibility of uneven communication. Four questions remain:

DeepEP doesn't implement the router or expert GEMMs. It accepts their rows and routing tensors, builds a layout handle, executes the exchange, and returns data in the layout the next kernel expects. The router, expert GEMMs, and training loop remain in the caller. That ownership boundary will matter when we diagnose a slow route versus a wrong route.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

A four-token routing ledger

Use two ranks and four tokens. Rank 0 owns tokens a and b; rank 1 owns c and d. There are four experts, two per rank, and each token selects two experts. Here the two selected experts always live on different ranks, so eight expert slots also require eight source-token/destination-rank rows, including local rows.

Rank 0 sends one copy of a to expert 3 and one copy of b to expert 2. It also receives c for expert 1 and d for expert 0. Rank 1 performs the mirror exchange. The counts happen to balance here; a real gate can send more rows from one rank than that rank receives.

Read each copy in two coordinate systems:

1. Source coordinates: (source_rank, token_index, topk_slot) identify the original row and expert choice.
2. Expert coordinates: (destination_rank, local_expert, expert_slot) identify where a receiving GEMM expects the row.

Dispatch creates this mapping. The returned EPHandle retains route indices and reverse-slot metadata, not gate values. Received gate tensors are separate outputs. Keep them aligned with expert rows when applying weights, then use the handle to return the weighted contributions.

For token b, which rank receives its first dispatched copy, and who applies its gate weight?

Answer

Rank 1 receives the copy for expert 2. The caller must multiply that expert's output by 0.40 before DeepEP combines the contributions for source token b.

V2's single interface: ElasticBuffer

V2 gives high-throughput and latency-oriented expert-parallel dispatch one interface: ElasticBuffer. The caller can pass an explicit byte count, or derive a 2 MiB-aligned size from maximum tokens per rank, hidden dimension, top-k, 8-bit floating-point (FP8) dispatch, topology, and reduction settings. Here 2 MiB means 2,097,152 bytes. The buffer then owns symmetric memory registered with NCCL.

Elastic describes this memory abstraction, not automatic infinite growth. Current V2 uses GPU-backed storage and can reserve CPU bytes for experimental Engram use. The README says V2 consumes more buffer space than V1 and that mixed GPU/CPU backing is still planned. Size the buffer from a real worst-case envelope rather than assuming it will grow on demand.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

Before dispatch can run, a caller needs a capacity estimate. get_buffer_size_hint gives a planning number, not an allocation. This unexecuted integration fragment assumes an initialized NCCL process group, BF16 or correctly scaled FP8 input, and valid routing tensors on a supported GPU:

If num_sms or num_qps is 0, dispatch asks the analytical helpers for defaults. Pass an integer to override them; the pinned implementation doesn't accept None as the automatic sentinel. EventOverlap marks communication readiness when asynchronous mode is requested. Launch independent work, then wait on the event before reading recv_x. V2 documents domains up to EP2048, which isn't a measured saturation result at that width.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

Reusing a buffer can avoid allocation and communicator setup. Reusing a handle is a stronger promise: the route itself must remain unchanged. The Python API asserts selected dimensions and reuses handle.topk_idx; it doesn't compare a newly computed route because cached dispatch requires topk_idx=None. Equal shapes don't establish equal routing.

Dispatch: layout, move, and epilogue

Ask what dispatch must preserve while rows are moving. V2 has three conceptual stages, even though kernels can overlap them: count and lay out rows by scale-up rank and local expert; move rows through local symmetric memory, NVLink, or NCCL Gin RDMA; then copy or expand rows into the receive layout used by expert GEMM.

The input contract makes those stages concrete. topk_idx has shape [num_tokens, num_topk], and topk_weights has the same shape with float values. Hidden states have shape [num_tokens, hidden] in BF16, or arrive as FP8 values plus scale factors. An expert index of -1 means that selection is unused. V2 returns received indices and weights beside the data so the next kernel can keep routing metadata attached.

The receive layout can be non-expanded or expanded. In non-expanded mode, one received row represents a source token with top-k metadata. In expanded mode, a token can occupy one slot per selected expert, and each local expert's segment is aligned for its GEMM. Alignment padding regularizes expert matrix shapes, but padding isn't a token and must not enter a combined row. Callers can use do_zero_padding when those gaps need to be cleared.

The copy epilogue turns metadata into expert input. In expanded mode, psum_num_recv_tokens_per_expert[i] is the aligned start of expert i plus its actual count, not simply an inclusive sum of padded counts. For counts [3, 1] and alignment 4, starts are [0, 4], these endpoints are [3, 5], and allocated expert segments total 8 slots. num_unaligned_recv_tokens_per_expert retains [3, 1]. Exact CPU counts require the CPU-synchronized path; no-sync callers must respect valid device counts rather than treating capacity as data.

The C++ launch path chooses dispatch for one scale-out domain and hybrid_dispatch when ranks span multiple scale-out groups. It passes the NCCL device communicator, symmetric window, buffer pointers, prefix arrays, slot metadata, timeout cycles, SM count, QP count, and topology indices into a just-in-time (JIT)-generated kernel. Reading csrc/kernels/elastic/dispatch.hpp beside deep_ep/buffers/elastic.py exposes the Python-to-kernel boundary.

Why does V2 keep both aligned prefix sums and unaligned expert counts?

Answer

Expert GEMM wants aligned offsets, while combine and correctness checks need the actual number of rows. The two values let the kernel use regular storage without treating alignment padding as a real token.

Combine: reverse routing plus reduction

Once each receiving rank runs its local experts, rows are in expert order rather than source-token order. combine consumes handle.recv_src_metadata, per-scale-up prefix sums, and optional hybrid channel metadata to reverse that mapping and sum contributions. Activation reduction and optional return of top-k weights are separate operations in combine_reduce_epilogue.cuh. Local bypasses skip a remote transfer, but still participate in reduction.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

For token a, the caller supplies weighted outputs 0.70 × (1, 0) and 0.30 × (4, 0). Combine should return (1.90, 0). Supplying raw expert outputs returns (5, 0); weighting twice returns (0.85, 0). All three tensors have the same shape. A transport test therefore needs a numerical reference, not just matching counts.

The API exposes allow_multiple_reduction, which controls staging. Reducing within a rank or node can send fewer rows onward, but intermediate BF16 materialization introduces extra rounding. Disabling it avoids those additional reduction stages, potentially moving more data. The general reduction path accumulates in FP32 before casting to BF16; a no-bias path with at most two inputs uses packed BF16 addition. Neither "BF16 output" nor "FP32 accumulation" alone describes every path. Treat staging as a bandwidth/numerical tradeoff.

For a concrete rounding case, consider already-weighted contributions 256, 1, and -256. Accumulating all three in FP32 gives 1, which BF16 can represent. If a local stage first materializes 256 + 1 as BF16, round-to-nearest-even produces 256; adding -256 then gives 0. The pinned CPU reference reproduces this difference when the first two contributions share a local reduction stage. This checks the reference's numerical contract, not a GPU kernel's latency or correctness.

Backward duality provides a compact consistency check:

For the unweighted route operators, dispatch and combine are transposes: copying rows forward becomes summing their gradients backward. The surrounding autograd wrapper must also differentiate expert functions and gate multiplication. It can't omit gate gradients merely because the communication calls reverse each other.

topk_weights can be two-dimensional in non-expanded mode or one-dimensional in expanded mode. bias supports zero, one, or two BF16 bias tensors for the combine epilogue. Its combined activation output is BF16. If a caller supplies a different layout or dtype, failure should happen at the contract boundary rather than after a silent reinterpretation.

Topology: scale-up and scale-out are different roads

The four-token ledger now meets hardware. Scale-up means GPUs inside one node, normally connected by NVLink. Scale-out means GPUs in different nodes, normally connected through RDMA network interfaces. A row that moves quickly over NVLink may wait on a network interface card (NIC), while a path that saves SM work across RDMA can add needless synchronization inside a node.

V2 derives physical domains from NCCL. With hybrid mode enabled, num_scaleout_ranks represents RDMA groups and num_scaleup_ranks represents the local NVLink domain. With hybrid mode disabled, DeepEP treats the whole communicator as one logical scale-up domain and uses a direct path. The logical view lets kernels choose a hierarchy while retaining original global rank IDs.

Hybrid dispatch uses channel linked lists and per-channel token metadata. It can forward rows through a local scale-up group after the RDMA phase. The code budgets CUDA warps, groups of 32 scheduled GPU threads, for scale-out and forward work separately, then launches a clustered kernel to overlap communication with nearby compute. A flat all-to-all comparison misses those extra ownership and synchronization costs.

When a network has multiple rails, NCCL Gin's connection type matters. The V2 backend asks NCCL for a NCCL_GIN_CONNECTION_RAIL context in hybrid mode and NCCL_GIN_CONNECTION_FULL in direct mode. If Gin isn't available for the selected topology, initialization asserts instead of quietly choosing an incompatible path, unless EP_DISABLE_GIN explicitly changes behavior. Treat that assertion as a topology validation step.

Logical bandwidth is not a NIC counter

DeepEP's README calls its performance numbers logical bandwidth. Logical bytes count the communication contract seen by the algorithm, including local-rank traffic. Physical link counters count bytes that crossed NVLink or a NIC. Those measurements answer different questions, so a profiler needs both labels.

Suppose four ranks exchange 100 MiB of token payload according to an EP layout. If half the rows stay on source GPUs, the logical operation still accounts for 100 MiB while the RDMA port may see only the remote half. If that payload crosses an NVLink hop during a hybrid forward, a physical NVLink counter can count copies that logical bandwidth reports once. A speedup without its byte denominator is not a reproducible measurement.

For a hypothetical 1 ms interval, 100 MiB corresponds to 104.8576 decimal GB/s; 50 MiB corresponds to 52.4288 GB/s. That difference can be bookkeeping, not speed loss. Real counters also include their own protocol and direction conventions. Use dispatch-plus-expert-plus-combine latency to evaluate layer performance, not a communication-only bandwidth number.

Read the README tables as shape-specific project reports, not generic hardware ceilings. It states 8K tokens per batch, hidden size 7168, top-8 routing, FP8 dispatch, and BF16 combine. For reproduction, also inspect the test harness: its per-rank count can be max(1, args.num_tokens - rank). Don't silently replace the README's batch wording with an exact per-rank claim. These are its reported logical-bandwidth snapshots:^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

The same README reports up to 1.3× peak performance versus V1 while using up to 4× fewer SMs. It also reports a V3-like legacy-training configuration dropping from 24 SMs to 4-6 while keeping equivalent or better performance. These are project-reported envelopes, not a warranty for your topology. Record token count, hidden size, top-k, dtype, SM count, queue-pair count, and bandwidth denominator before comparing runs.

An experiment reports 90 GB/s logical RDMA bandwidth, but your NIC counter shows 45 GB/s. Is that automatically a regression?

Answer

No. The logical figure can include local traffic and can use a different byte denominator. Reconcile token counts, local bypass, and measured physical bytes before judging the result.

NCCL Gin V2 backend

NCCL Gin is a device-side communication interface for issuing network operations from GPU code. V2 reuses an existing NCCL communicator instead of asking an application to construct a separate transport world. The C++ backend queries communicator properties, requests Gin contexts and signals, registers a symmetric memory window, and stores a device communicator pointer for JIT kernels.

Follow initialization as a chain of ownership transfers:

1. Create or receive the host NCCL communicator.
2. Query physical rank domains and Gin capability.
3. Request QP contexts, queue depth, traffic class, and signal count.
4. Choose rail or full connection based on direct or hybrid mode.
5. Allocate symmetric GPU and optional CPU memory.
6. Register a window and obtain NVLink peer pointers.
7. Build a device communicator that JIT kernels can call.

The signal count includes rank barriers and custom communication notifications. EP_OVERRIDE_RDMA_SL can place V2 traffic on a chosen InfiniBand service level, while NCCL_GIN_CROSS_NIC=0 changes symmetric-memory handle reuse behavior for multi-plane systems. These are cluster integration knobs, not per-request model settings.

Keep ownership explicit. DeepEP owns layout, kernels, and its window lifecycle. NCCL owns communicator and network primitives. A model runtime still owns process-group formation, rank assignment, and shutdown order. Aborting a communicator while an EventOverlap hook or JIT kernel still references its window is a lifecycle bug, not a routing bug.

EPHandle: route metadata with a lifetime

After dispatch, ask what survives the network transfer. EPHandle is returned by dispatch and consumed by combine. Its fields expose the route in a form that bridges device kernels, CPU scheduling, and cached inference:

A cached handle can skip layout recomputation for a replay of the same route, such as a corresponding backward operation. A new decode token usually has a newly computed route; equal batch size doesn't make the old handle reusable. The cached path skips CPU count discovery, so the caller must establish route and layout compatibility before using it.

V2's deterministic mode sorts received rows after the kernel. Non-expanded mode sorts data, weights, indices, and source metadata. Expanded mode sorts rows within each expert and updates slot pointers without permuting the metadata table in the same way. The sort key uses source-global order and expert identity, so equal inputs can present the same expert order even when arrival order differs.

Determinism has a cost: sorting and metadata work can constrain asynchronous behavior. Enable it for correctness tests, reproducible training slices, or debugging route drift. Measure it separately from the non-deterministic throughput path.

EventOverlap: a stream contract

CUDA streams let communication and compute overlap, but a Python return doesn't make an output tensor safe to read. EventOverlap wraps a CUDA event and can register a hook to run after the event is waited on. First launch independent work, then make the compute stream wait before it reads received rows:

previous_event can make a communication kernel wait for an upstream CUDA event. allocate_on_comm_stream controls ownership of newly allocated tensors when that dependency is used. A host synchronization inserted between the event and expert work can erase the overlap the caller intended to gain. Profile stream waits beside kernel durations, and keep readiness separate from route correctness.

This fragment is not an executable CUDA test: launch_weighted_local_experts stands for the caller's expert integration. current_stream_wait() enqueues a dependency, not a host-wide synchronization. In deterministic mode it also triggers a sorting hook on the waiting stream. Releasing the wrapper after one wait doesn't establish readiness on every other consumer stream; keep allocation lifetimes and all consumer dependencies explicit.

Precision: FP8 on the wire, BF16 for the reduction

DeepEP accepts BF16 hidden states for the baseline path. In FP8 dispatch mode, input is a tuple of FP8 values and scale factors. The receiving expert integration consumes that representation according to its GEMM contract. Combine takes and returns BF16 activation rows; gate multiplication belongs in the caller's expert path, not in the transport.

Lower payload precision doesn't remove numerical validation. Scale-factor layout, hidden alignment, and expert GEMM expectations must agree. A kernel can move bytes correctly while a mismatched scale stride corrupts activations. The reference quantizer uses one float scale per 128 values: a 7,168-element FP8 row takes 7,168 payload bytes plus 224 scale bytes, versus 14,336 bytes for BF16. Routing metadata and protocol overhead are additional, so this isn't exactly a 2× wire-byte reduction.

use_tma_aligned_col_major_sf selects an optional scale-factor layout suitable for Tensor Memory Accelerator (TMA) paths. Read the model's quantization and GEMM contract before enabling it. DeepEP's source handles scale metadata, but it doesn't decide which quantization scheme an entire model should use.

For post-training or RL on large sparse models, this boundary matters twice. Rollout inference may favor low-latency FP8 dispatch, while training may need BF16 or a different accumulator policy. Keep dtype and scale metadata in the experiment receipt. Otherwise, a reward-curve change after a dtype switch confounds communication changes with model numerics.

Analytical SM and QP sizing

V2's get_theoretical_num_sms estimates communication SMs from topology bandwidth and a balanced gate model. Before trusting that number, ask what it can observe: expected top-k destinations, read and write work, NVLink traffic, RDMA traffic, and the link bottleneck, but not your future gate histogram.

This is an analytical model, not a runtime-tuner oracle. The helper multiplies the raw bandwidth-derived estimate by 1.25, rounds up to an even count of at least 4, and clamps to the device's multiprocessor count. When compute-communication overlap is preferred, it stays near that calculated minimum; otherwise, it raises the floor to 64 SMs for maximum standalone throughput. The implementation assumes balanced routing, whereas production DeepSeek-style gates can be group-limited and asymmetric. Inspect the actual token distribution before trusting the estimate, and pass num_sms explicitly when your application overlap budget demands tighter bounds.

Queue-pair (QP) sizing follows a related rule. Direct mode encourages roughly one QP per communication SM, capped at nine in the current helper (min(num_sms, 8 + 1)). Hybrid mode encourages num_sms * 16 + 1 QPs so channels and notification work can have independent queues, then caps that value by allocated QPs. More QPs consume resources and can increase RDMA doorbell ringing. Use the theoretical count as a starting point, then measure queue pressure and tail latency.

The balanced-gate model counts distinct destinations, not expert selections. With four experts split evenly across two ranks and two distinct experts sampled uniformly, six expert pairs are possible. Two pairs stay on one destination rank; four touch both. Expected destination count is therefore $(2×1 + 4×2)/6 = 5/3$, not 2. The helper uses the equivalent formula $G[1-\binom{E-E/G}{k}/\binom{E}{k}]$, for $E$ experts evenly divided among $G$ groups.

This standard-library check compares that formula with exhaustive enumeration. It also reproduces the pinned QP helper's arithmetic without claiming that the resulting queue count is optimal on hardware:

The hybrid request is 193 queues, but only 128 are allocated. The helper caps at 128; it doesn't allocate the missing 65. Group-limited or skewed gates violate the uniform-subset assumption behind the expected-destination formula. The source explicitly warns against using its balanced SM model for V3.0's group-limited gate.

Why can a balanced-gate SM estimate be wrong for a production MoE model?

Answer

The gate distribution can be asymmetric or group-limited. That changes read, write, RDMA, and NVLink traffic, so the modeled bottleneck and chosen SM count no longer match the real route.

JIT compilation is part of the runtime

DeepEP V2 compiles kernels at runtime. The Python or C++ launch code assembles template parameters such as topology, expansion mode, hidden bytes, expert alignment, number of warps, SM count, and QP count. The JIT compiler parses required headers, invokes NVCC or the configured compiler, caches the resulting module, and launches it with a generated configuration.

That choice avoids compiling every shape specialization during installation, but moves a failure boundary into first use. A missing CUDA compiler, incompatible NCCL headers, or unsupported instruction can surface at the first dispatch. The pinned JIT selects its architecture from the active device and compiler support; TORCH_CUDA_ARCH_LIST controls extension/build targets, not every runtime JIT specialization. Separate compile, cold-start, and steady-state timings.

Relevant environment switches include:

For a launch, pre-warm every required shape on every worker or include compilation time in startup readiness. A cache mounted from one host may contain code for a different device capability. Treat cache keys and compiler versions as part of the deployment artifact.

V2 and V1 are different products

DeepEP V1 is archived documentation, not a hidden compatibility mode for the V2 API. V1 exposes a Buffer, uses NVSHMEM for internode and low-latency methods, asks callers to provide separate NVLink and RDMA sizes, and has distinct normal and low-latency kernels. Its low-latency path can use a hook to defer an RDMA receive without occupying SMs. V2 uses ElasticBuffer, NCCL Gin, unified high-throughput and low-latency APIs, and larger scale-up and scale-out domains.^{[2]Reference 2DeepEP V1 Documentationhttps://github.com/deepseek-ai/DeepEP/blob/dd758caf451848bd150e1046af3d0a73e5fff38d/docs/legacy.md}

V1 source remains relevant when reading old DeepSeek-V3 integrations or reproducing a historical benchmark. That history doesn't show that a current V2 deployment supports the same CUDA, PyTorch, NVSHMEM, or low-latency assumptions. V1 performance tables use H800 and CX7 setups, which shouldn't be compared with V2's SM90 and SM100 tables until shapes and links match.^{[2]Reference 2DeepEP V1 Documentationhttps://github.com/deepseek-ai/DeepEP/blob/dd758caf451848bd150e1046af3d0a73e5fff38d/docs/legacy.md[4]Reference 4DeepSeek-V3 Technical Report.https://arxiv.org/abs/2412.19437}

Migration order matters: port the route contract first, then port the transport. Replace Buffer layout calls with ElasticBuffer handles, verify NCCL Gin availability, resize memory, and rerun deterministic correctness tests before measuring throughput. Don't infer compatibility from a successful import alone.

Experimental PP, CP, and Engram boundaries

The current repository includes experimental primitives beyond EP. Treat each as a separate boundary rather than as another dispatch mode:

• Pipeline parallel (PP): send and receive tensors through symmetric memory. The README lists 0-SM PP with RDMA as an experimental path. It doesn't replace Megatron's full pipeline scheduler.
• Context parallel (CP): use copy-engine-oriented paths for context slices, including a 0-SM copy-engine variant. It doesn't define attention semantics or sequence partitioning policy.
• Engram: fetch remote key-value entries through RDMA, with optional CPU-backed storage and TMA-aligned scale factors. It doesn't provide a model-level retrieval index.
• AGRS: experimental all-gather and reduce-scatter sessions over NVLink symmetric memory.

These APIs share allocation and event machinery with ElasticBuffer, which makes them relevant systems experiments. They aren't proof that the whole feature set has the same production maturity as V2 EP. The README labels Engram, PP, and CP experimental.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

For a large RL system, keep these boundaries explicit. A PP send can be healthy while EP RDMA is congested. An Engram fetch can consume the same QP or memory budget that dispatch sizing assumed was free. A CP copy can change stream dependencies. Share a buffer only when its maximum concurrent sessions and lifetimes are accounted for.

Requirements and cluster checks

The current README lists Hopper (SM90) GPUs or architectures with SM90 Parallel Thread Execution (PTX) support, CUDA 12.3 or newer for SM90, PyTorch 2.10 or newer, and NCCL 2.30.4 or newer. NVLink is expected for intranode communication and RDMA for internode communication. The repository reports full InfiniBand testing and theoretical compatibility with RDMA over Converged Ethernet (RoCE). These are requirements for the pinned source, not a guarantee that a different revision keeps the same floor.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

Before a multi-node job starts, check the path from hardware to buffer:

Don't run pip install on a login node and assume the worker image matches. The source's JIT cache and NCCL library path are runtime dependencies. Include their versions, topology dump, and EP_BUFFER_DEBUG=1 initialization output in a launch receipt. Redact credentials and private host details before sharing it.

Applications in giant-model post-training

DeepEP matters when a policy or teacher is a sparse MoE model. The previous slime study showed that rollout, reward, and training must agree on tokens and weight versions. Those sampled trajectories still pass through MoE layers, so every generated token can trigger another dispatch and combine. Rollout needs fast expert routing; training needs exact route metadata and reproducible gradients. A slow all-to-all multiplies across generated tokens, while an incorrect or stale handle can corrupt every subsequent update.

Use DeepEP to compare three placement patterns. For each one, ask which phase owns the communicator and which receipt would reveal its cost:

The communication library doesn't solve policy staleness, reward hacking, or checkpoint synchronization. It can become the lowest-level source of latency and numerical mismatch in those systems. Pair route metrics with rollout weight versions, expert-load histograms, and reward receipts. If reward falls after a topology change, separate transport delay, dtype error, and algorithm behavior before changing the policy.

A runnable routing ledger

Work token a by hand before touching code. Its hidden state is (1.0, 0.0). Give expert $k$ a toy map that multiplies by $k + 1$, so expert 0 returns (1.0, 0.0) and expert 3 returns (4.0, 0.0). Predict the combined first coordinate before reading the equation: gate 0.70 keeps most of expert 0's value, while gate 0.30 adds a smaller expert 3 contribution. The caller weights each expert output before combine sums the results:

$0.70\cdot(1.0,\ 0.0) + 0.30\cdot(4.0,\ 0.0) = (1.90,\ 0.0)$

Expert 0 lives on rank 0, so that copy of a is local. Expert 3 lives on rank 1, so the second copy is remote. The standard-library model below builds expanded receive slots, pads each expert to four rows, computes and weights outputs in expert order, and combines them using the saved inverse map. It never recomputes the original routes during combine. Shuffling the returned rows must not change the answer.

This inverse map is a Python teaching structure, not DeepEP's binary EPHandle layout. Gate values live in the received rows and are applied in run_experts; combine only sums supplied vectors. Missing, duplicated, or padded return slots fail explicitly. DeepEP doesn't promise these expensive Python-style checks on every GPU operation, so retain an independent reference in integration tests.

Now move a's first selection from expert 0 to expert 2. Experts 2 and 3 both live on rank 1: there are still eight expert assignments, but only seven source-token/destination-rank rows. The receiver expands a for the two local experts after transport. A -1 selection contributes neither a transport destination nor an expert slot; don't accidentally treat it as Python's last expert.

Strengths and weaknesses

DeepEP stays small at its boundary. It exposes a compact EP API and specializes kernels for current GPU and RDMA paths. V2's unified buffer, analytical resource estimates, NCCL communicator reuse, deterministic option, and JIT compilation make that boundary inspectable. Each choice also leaves a limit to measure:

Weakness isn't the same as a bug. Larger V2 buffers may trade memory for broader topology support, while minimum SM counts can favor overlap over standalone benchmarks. A failed Gin assertion can expose a network deployment that doesn't satisfy the chosen mode. Diagnose against the contract before patching a kernel.

Team, contributors, governance, and licenses

The repository is published under the DeepSeek organization. Its citation lists Chenggang Zhao, Shangyan Zhou, Liyue Zhang, Chengqi Deng, Zhean Xu, Yuxuan Liu, Kuai Yu, Jiashi Li, and Liang Zhao. The README also thanks NCCL contributors and the NCCL team for V2 Gin support. These names describe the pinned source snapshot's credited contributors, not a permanent ranking of current maintainers.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

The source's MIT license permits use, modification, and distribution with the required notice. It doesn't grant a license to NCCL or NVSHMEM binaries, model checkpoints, benchmark datasets, or network firmware. An image should inventory those dependencies separately. The V2 repository says NVSHMEM remains needed for legacy methods, so removing every NVSHMEM package from an image can break a V1 compatibility path even when V2 EP itself uses NCCL Gin.^{[5]Reference 5DeepEP MIT Licensehttps://github.com/deepseek-ai/DeepEP/blob/dd758caf451848bd150e1046af3d0a73e5fff38d/LICENSE[6]Reference 6DeepEP NVSHMEM Installation Guidehttps://github.com/deepseek-ai/DeepEP/blob/dd758caf451848bd150e1046af3d0a73e5fff38d/docs/nvshmem.md}

The pinned README supplies a project citation and benchmark tables, not a paper establishing every V2 result. The DeepSeek-V3 report motivates expert-parallel communication and group-limited routing; it doesn't document all of these later V2 kernels.^{[4]Reference 4DeepSeek-V3 Technical Report.https://arxiv.org/abs/2412.19437}

Benchmark and failure-mode checklist

A throughput number is evidence only when another engineer can reconstruct its bytes, route, timing, and correctness check. Before trusting a number, capture a receipt with:

1. DeepEP commit and JIT/compiler versions.
2. GPU architecture, GPU count, NVLink map, NIC model, and rail topology.
3. CUDA, PyTorch, NCCL, and optional NVSHMEM versions.
4. Tokens per rank, hidden size, top-k, expert count, and expert alignment.
5. Gate histogram, local-bypass rate, and deterministic setting.
6. FP8/BF16 scale layout, SM count, QP count, and hybrid/direct mode.
7. Warm-up policy, JIT time, dispatch time, combine time, and end-to-end layer time.
8. Logical-byte formula and physical-link counters.

Common failures map to specific boundaries:

The most dangerous failure is a plausible tensor with wrong provenance. A profiler can show kernel time, stream waits, and link traffic, but it can't establish that each row returned to its source token. Add route checksums, per-expert counts, and deterministic small-cluster tests before launching a long RL job. Communication success isn't model correctness.

Source-reading map

Read one vertical slice instead of opening every kernel at once. Begin with the public contract, follow one route into the kernels, then check the legacy boundary:

At each boundary, write down the authoritative state. Router output owns expert choices. EPHandle owns source-to-slot metadata. NCCL owns communicator and window state. The event owns readiness. The expert GEMM owns its local output. Combine should never guess any of those values from decoded text or a second route calculation. That ownership map is the shortest path from a profiler symptom to the component that can explain it.

Reviewing an integration

Evaluation rubric

• Foundational: Trace token a through both experts and derive 1.90 after exactly one gate multiplication. Distinguish received gate tensors from the handle's route metadata.
• Intermediate: Explain destination-rank deduplication, expert padding, cached-route validity, and the stream wait that must precede reading dispatched rows.
• Advanced: Reconcile logical bytes with physical traffic, qualify balanced-gate resource estimates, and separate CPU reference agreement from GPU memory-ordering and distributed correctness.

Follow-up questions

The route changes from experts [0, 3] to [2, 3] without changing tensor shapes. What must change before dispatch?

Answer

Rebuild the route handle. Both new experts live on rank 1, so the token has one destination-rank row rather than two, followed by two expert slots on rank 1. Reusing the old handle silently preserves the old route; matching shapes aren't a validity check.

An integration agrees with the CPU routing ledger but hangs only when a direct logical domain spans NVLink and RDMA. What remains untested?

Answer

The ledger doesn't execute transport, remote visibility, barriers, event lifetimes, or GPU kernels. Check the deployed commit against the mixed-fabric ordering fix, then reproduce on the actual topology with upstream distributed correctness tests. A CPU match doesn't establish that a missing system-scope fence is harmless.