Deep Dive - DeepEP

A mixture-of-experts (MoE) layer can route one token to two experts on another GPU. The neural-network math is small; moving that token, its routing metadata, and its return value across a cluster is not. If communication burns too many streaming multiprocessors (SMs), it steals cycles from the expert matrix multiply. If the route is wrong, the model trains on the wrong token even when every kernel reports success.

DeepEP (DeepEveryParallel) is DeepSeek's communication library for that boundary. Its current V2 API turns expert-parallel dispatch and combine into an ElasticBuffer contract backed by NCCL Gin, NVIDIA Collective Communications Library's device-side network interface. This lesson starts with four tokens and two ranks, then follows the same metadata through the NVLink GPU interconnect, remote direct memory access (RDMA), a fused kernel, and a reverse combine. It keeps V2 separate from the archived NVIDIA SHMEM (NVSHMEM)-based V1 path, because the two versions have different interfaces, resource costs, and failure modes.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d[2]Reference 2DeepEP V1 Documentationhttps://github.com/deepseek-ai/DeepEP/blob/dd758caf451848bd150e1046af3d0a73e5fff38d/docs/legacy.md}

Why expert parallelism needs a communication layer

An MoE architecture replaces one dense feed-forward block with many expert blocks. A router chooses top-k experts for each token. Expert parallelism (EP) places different experts on different ranks, so each rank owns only a slice of the expert weights. That reduces per-rank model memory, but it turns each MoE layer into an all-to-all exchange.

For one local rank, write the hidden states as a matrix X with one row per token. The router returns topk_idx[t, j] and topk_weight[t, j]. The dispatch operation expands this logical table into rows grouped by destination expert. Local rows can stay on the current GPU; remote rows need a transport path. Each expert then runs its matrix multiplications. Combine reverses the route and applies gate weights while reducing duplicate expert results into the original token rows.

The operation is not the same as an ordinary all_to_all of a dense rectangular tensor. Token counts differ by expert, top-k can duplicate a token for several experts, and a rank may receive no rows for one expert but many for another. A complete communication layer therefore owns four jobs:

DeepEP doesn't implement the router or expert GEMMs. It accepts their token rows and routing tensors, provides a layout handle, executes the exchange, and returns data in a layout that the next kernel can consume. The boundary is deliberately narrow enough to embed in a training or inference runtime.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

A four-token routing ledger

Start with two ranks and four tokens. Rank 0 owns tokens a and b; rank 1 owns c and d. There are four experts, two per rank. Each token selects two experts:

Rank 0 must send one copy of a to expert 3 and one copy of b to expert 2. It also receives c for expert 1 and d for expert 0. Rank 1 performs the mirror exchange. The number of rows sent by one rank is not necessarily the number received by another rank.

Use two coordinate systems:

1. Source coordinates: (source_rank, token_index, topk_slot) identify the original row and expert choice.
2. Expert coordinates: (destination_rank, local_expert, expert_slot) identify where a receiving GEMM expects the row.

Dispatch creates the mapping. The returned EPHandle stores enough metadata to map expert coordinates back to source coordinates. Combine consumes that handle after expert computation. If the handle is discarded, an implementation would need to reconstruct source positions and weights, which is slower and prone to mistakes.

V2's single interface: ElasticBuffer

DeepEP V2 replaces separate high-throughput and low-latency expert-parallel interfaces with one ElasticBuffer. Its constructor can receive explicit bytes or calculate a size from maximum tokens per rank, hidden dimension, top-k, 8-bit floating-point (FP8) dispatch, topology, and reduction settings. Allocation is aligned to 2 MiB, and the buffer owns a symmetric memory window registered with NCCL.

The word elastic describes the planned memory abstraction, not automatic infinite growth. Current V2 allocates GPU-backed storage and may reserve CPU bytes for experimental Engram use. The README warns that V2 consumes more buffer space than V1. It also says elastic GPU and CPU backing is still on the roadmap, so production code should size the buffer from a real worst-case envelope instead of assuming it will grow on demand.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

The public path looks like this:

The example leaves num_sms and num_qps at zero, asking V2 to compute them. dispatch returns an EventOverlap object because communication may still be in flight when Python regains control. A caller can launch independent expert-side work, then wait on the event before reading recv_x. combine receives the expert outputs and the same handle.

The buffer is a resource and a protocol. Reusing it across layers can save allocation and communicator setup, but a stale handle is only valid when the dimensions, expert layout, and routing assumptions still match. The Python API asserts those invariants when a cached handle is passed.

Dispatch: layout, move, and epilogue

V2 dispatch has three conceptual stages even though kernels can overlap them:

1. Count and layout rows by scale-up rank and local expert.
2. Move rows through local symmetric memory, NVLink, or NCCL Gin RDMA.
3. Copy or expand rows into the receive layout used by expert GEMM.

topk_idx has shape [num_tokens, num_topk]. topk_weights has the same shape and uses float values. The hidden-state input is [num_tokens, hidden] in BF16, or a tuple containing FP8 values and scale factors. -1 expert indices represent unused selections. V2 returns received indices and weights alongside data so an expert implementation can preserve routing metadata.

The receive layout can be non-expanded or expanded. In non-expanded mode, one received row represents a source token with top-k metadata. In expanded mode, a token can occupy one slot per selected expert, and each local expert's segment is aligned for its GEMM. Alignment padding makes the expert matrix shapes regular, but padding must not become a trainable or combined row. The do_zero_padding option exists for callers that need those gaps cleared.

The copy epilogue is where metadata becomes a contiguous expert input. EPHandle.psum_num_recv_tokens_per_expert stores prefix sums with alignment rules. num_unaligned_recv_tokens_per_expert records actual counts in expanded mode. num_recv_tokens_per_expert_list is a CPU-side list used to launch one GEMM per local expert. These fields are not redundant bookkeeping: they connect irregular network arrival to regular compute tiles.

The C++ launch path selects dispatch for one scale-out domain and hybrid_dispatch when ranks span multiple scale-out groups. It passes the NCCL device communicator, symmetric window, buffer pointers, prefix arrays, slot metadata, timeout cycles, SM count, QP count, and topology indices into a just-in-time (JIT)-generated kernel. Reading csrc/kernels/elastic/dispatch.hpp beside deep_ep/buffers/elastic.py makes the Python-to-kernel boundary concrete.

Combine: the reverse map, not a second all-to-all

After each receiving rank runs its local experts, combine returns rows to source ranks. It consumes handle.recv_src_metadata, per-scale-up prefix sums, optional hybrid channel metadata, and gate weights. The kernel reduces duplicates that originated from one source token. Local bypasses can avoid a network transfer, but they still participate in the same logical reduction.

The API exposes allow_multiple_reduction. When enabled, a hybrid path can perform more than one reduction stage. When disabled, the final combine epilogue performs one reduction for better precision, potentially moving more data. This is a bandwidth and numerical-contract tradeoff, not a universal "fast" flag.

Backward duality provides a compact consistency check:

DeepEP's README example calls the backward of dispatch a combine, and the backward of combine a dispatch. This follows the same route metadata, not a claim that automatic differentiation is hidden inside the communication library. The surrounding training framework still computes matrix gradients and gate gradients.

topk_weights can be a two-dimensional tensor in non-expanded mode or a one-dimensional tensor in expanded mode. bias supports zero, one, or two BF16 bias tensors for the combine epilogue. The receiving output is BF16. If the caller uses a different layout or dtype, the failure should happen at the contract boundary rather than after a silent reinterpretation.

Topology: scale-up and scale-out are different roads

A GPU cluster has at least two transport scales. Scale-up means GPUs inside one node, normally connected by NVLink. Scale-out means GPUs in different nodes, normally connected through RDMA network interfaces. A route that is fast on NVLink can be a poor route across a network interface card (NIC), and a route that minimizes SM work for RDMA can add unnecessary synchronization inside a node.

V2 derives physical domains from NCCL. With hybrid mode enabled, num_scaleout_ranks represents RDMA groups and num_scaleup_ranks represents the local NVLink domain. With hybrid mode disabled, DeepEP treats the whole communicator as one logical scale-up domain and uses a direct path. This logical view lets the kernel choose hierarchy while retaining the original global rank IDs.

Hybrid dispatch uses channel linked lists and per-channel token metadata. It can forward rows through a local scale-up group after the RDMA phase. The code computes warps for scale-out and forward work separately, then launches a clustered kernel to overlap communication with nearby compute. That extra machinery is why a flat all-to-all comparison misses important costs.

When a network has multiple rails, NCCL Gin's railed connection type matters. The V2 backend asks NCCL for a NCCL_GIN_CONNECTION_RAIL context in hybrid mode and NCCL_GIN_CONNECTION_FULL in direct mode. If Gin isn't available for the selected topology, initialization asserts instead of quietly using an incompatible path unless EP_DISABLE_GIN explicitly changes behavior. A deployment should treat this as a topology validation step.

Logical bandwidth is not a NIC counter

DeepEP's README calls its performance numbers logical bandwidth. Logical bytes count the communication contract seen by the algorithm, including local-rank traffic. Physical link counters measure bytes that crossed NVLink or a NIC. Those values answer different questions.

Suppose four ranks exchange 100 MiB of token payload according to an EP layout. If half the rows stay on source GPUs, the logical operation still moves 100 MiB in its model, while the RDMA port may see only the remote half. If the same payload crosses an NVLink hop during a hybrid forward, a physical NVLink counter can count copies that logical bandwidth reports once. Comparing the numbers without naming the denominator produces a misleading speedup.

Use the README tables as shape-specific project reports: V2's stated configuration is 8K tokens per rank, hidden size 7168, top-8 routing, FP8 dispatch, and BF16 combine. It reports examples for SM90 and SM100, ConnectX-7 (CX7) RDMA, EP 8×2, EP 8×4, and EP 8. The headline comparison says up to 1.3× peak performance and up to 4× fewer SMs than V1, but those are not universal promises. Record topology, token count, hidden size, top-k, dtype, SM count, QP count, and whether bandwidth is logical before comparing runs.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

NCCL Gin V2 backend

NCCL Gin is a device-side communication interface for issuing network operations from GPU code. DeepEP V2 reuses an existing NCCL communicator instead of requiring an application to construct a separate transport world. The C++ backend queries communicator properties, requests Gin contexts and signals, registers a symmetric memory window, and stores a device communicator pointer for JIT kernels.

The V2 backend's initialization path is worth reading in order:

1. Create or receive the host NCCL communicator.
2. Query physical rank domains and Gin capability.
3. Request QP contexts, queue depth, traffic class, and signal count.
4. Choose rail or full connection based on direct or hybrid mode.
5. Allocate symmetric GPU and optional CPU memory.
6. Register a window and obtain NVLink peer pointers.
7. Build a device communicator that JIT kernels can call.

The signal count includes rank barriers and custom communication notifications. EP_OVERRIDE_RDMA_SL can place V2 traffic on a chosen InfiniBand service level. NCCL_GIN_CROSS_NIC=0 changes symmetric-memory handle reuse behavior for multi-plane systems. These are cluster integration knobs, not per-request model settings.

The important boundary is ownership. DeepEP owns layout, kernels, and its window lifecycle. NCCL owns communicator and network primitives. A model runtime still owns process-group formation, rank assignment, and shutdown order. Aborting a communicator while an EventOverlap hook or JIT kernel still references its window is a lifecycle bug, not a routing bug.

EPHandle: route metadata with a lifetime

EPHandle is returned by dispatch and consumed by combine. Its fields expose the route in a form that bridges device kernels, CPU scheduling, and cached inference:

For decode, a cached handle can skip layout recomputation when expert assignments remain unchanged. The cached path doesn't perform a CPU synchronization to discover new counts and must reuse matching dimensions. This can help stable decoding patterns, but it changes the correctness proof: test that the gate layout is reusable, and invalidate the handle when routing or capacity changes.

V2's deterministic mode sorts received rows after the kernel. Non-expanded mode sorts data, weights, indices, and source metadata. Expanded mode sorts rows within each expert and updates slot pointers without permuting the metadata table in the same way. The sort key uses source-global order and expert identity, so two runs with the same inputs can present the same expert order even when arrival order differs.

Determinism has a cost. It adds sorting and metadata work and can constrain asynchronous behavior. Enable it for correctness tests, reproducible training slices, or debugging route drift. Measure it separately from the non-deterministic throughput path.

EventOverlap: a stream contract

CUDA streams let communication and compute overlap, but a Python return doesn't make an output tensor safe to read. EventOverlap wraps a CUDA event and can register a hook to run after the event is waited on. The common pattern is:

previous_event can make a communication kernel wait for an upstream CUDA event. allocate_on_comm_stream controls ownership of newly allocated tensors when that dependency is used. If the caller combines a stream event with an ordinary host synchronization, it can erase the overlap it intended to gain. Profile stream waits along with kernel durations.

The same contract appears in V2's barrier and experimental PP, CP, and Engram APIs. Reusing one event wrapper across unrelated buffers is unsafe because event readiness says nothing about which allocation or route it protects.

Precision: FP8 on the wire, BF16 for the reduction

DeepEP accepts brain floating point 16 (BF16) hidden states for the baseline path. In FP8 dispatch mode, the input is a tuple of FP8 values and scale factors. The transport moves the low-precision payload, while scale factors describe how to recover a value for the receiving expert computation. Combine returns BF16 and applies gate-weight reductions there.

This split targets bandwidth and memory pressure without forcing the final reduction to accumulate in FP8. It doesn't remove the need to validate numerical error. Scale-factor layout, hidden dimension alignment, and expert GEMM expectations must agree. A kernel can move bytes correctly while a mismatched scale stride corrupts activations.

use_tma_aligned_col_major_sf selects an optional scale-factor layout suitable for Tensor Memory Accelerator (TMA) paths. Read the model's quantization and GEMM contract before enabling it. DeepEP's source handles scale metadata, but it doesn't decide which quantization scheme an entire model should use.

For post-training or RL on large sparse models, this boundary matters twice. Rollout inference may favor low-latency FP8 dispatch, while training may need BF16 or a different accumulator policy. Keep dtype and scale metadata in the experiment receipt. Comparing reward curves across a dtype change without recording it confounds communication changes with model numerics.

Analytical SM and QP sizing

V2's get_theoretical_num_sms estimates communication SMs from topology bandwidth and a balanced gate model. It computes expected top-k destinations, read and write work, NVLink traffic, RDMA traffic, and the bounded link. It then chooses an even SM count, applies a margin, and clamps to the device's multiprocessor count. If overlap is preferred, it can stay near the minimum needed; otherwise it may choose at least 64 SMs.

This is a model, not a tuner oracle. The implementation explicitly assumes balanced routing and says group-limited gates need different treatment. DeepSeek-style routing can be asymmetric by design, so inspect the actual token distribution before trusting the estimate. Passing num_sms manually is appropriate when measured topology or an application-level overlap budget beats the balanced approximation.

QP sizing follows a related rule. Direct mode encourages roughly one QP per communication SM, capped at nine in the current helper. Hybrid mode encourages num_sms * 16 + 1 QPs to give channels and notification work independent queues, capped by allocated QPs. More QPs consume resources and can ring database or network state more aggressively. Use the theoretical count as a starting point, then measure queue pressure and tail latency.

JIT compilation is part of the runtime

DeepEP V2 compiles kernels at runtime. The Python or C++ launch code assembles template parameters such as topology, expansion mode, hidden bytes, expert alignment, number of warps, SM count, and QP count. The JIT compiler parses required headers, invokes NVCC or the configured compiler, caches the resulting module, and launches it with a generated configuration.

JIT avoids compiling every possible hardware and shape combination during package installation. It also moves a failure boundary into first use. A missing CUDA compiler, wrong TORCH_CUDA_ARCH_LIST, stale cache, incompatible NCCL headers, or unsupported PTX instruction can surface when the first dispatch runs. Warm-up time must be separated from steady-state communication benchmarks.

Relevant environment switches include:

For a production launch, pre-warm every required shape on every worker or include compilation time in startup readiness. A cache mounted from one host may contain code for a different device capability. Treat cache keys and compiler versions as part of the deployment artifact.

V2 and V1 are different products

DeepEP V1 is archived documentation, not a hidden compatibility mode for the V2 API. V1 exposes a Buffer, uses NVSHMEM for internode and low-latency methods, asks callers to provide separate NVLink and RDMA sizes, and has distinct normal and low-latency kernels. Its low-latency path can use a hook to defer an RDMA receive without occupying SMs. V2 uses ElasticBuffer, NCCL Gin, unified high-throughput and low-latency APIs, and larger scale-up and scale-out domains.^{[2]Reference 2DeepEP V1 Documentationhttps://github.com/deepseek-ai/DeepEP/blob/dd758caf451848bd150e1046af3d0a73e5fff38d/docs/legacy.md}

V1 source remains relevant when reading old DeepSeek-V3 integrations or reproducing a historical benchmark. That history doesn't show that a current V2 deployment supports the same CUDA, PyTorch, NVSHMEM, or low-latency assumptions. V1 performance tables use H800 and CX7 setups, which should not be compared to V2's SM90 and SM100 tables without matching shapes and links.^{[2]Reference 2DeepEP V1 Documentationhttps://github.com/deepseek-ai/DeepEP/blob/dd758caf451848bd150e1046af3d0a73e5fff38d/docs/legacy.md[3]Reference 3DeepSeek-V3 Technical Report.https://arxiv.org/abs/2412.19437}

Migration order matters: port the route contract first, then port the transport. Replace Buffer layout calls with ElasticBuffer handles, verify NCCL Gin availability, resize memory, and rerun deterministic correctness tests before measuring throughput. Don't infer compatibility from a successful import alone.

Experimental PP, CP, and Engram boundaries

The current repository includes experimental primitives beyond EP:

• Pipeline parallel (PP): send and receive tensors through symmetric memory and a small number of SMs. It doesn't replace Megatron's full pipeline scheduler.
• Context parallel (CP): use copy-engine-oriented paths for context slices. It doesn't define attention semantics or sequence partitioning policy.
• Engram: fetch remote key-value entries through RDMA, with optional CPU-backed storage and TMA-aligned scale factors. It doesn't provide a model-level retrieval index.
• AGRS: experimental all-gather and reduce-scatter sessions over NVLink symmetric memory.

These APIs share allocation and event machinery with ElasticBuffer, which makes them relevant systems experiments. They aren't proof that the whole feature set has the same production maturity as V2 EP. The README labels Engram, PP, and CP experimental and calls 0-SM PP and Engram features available with RDMA.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

For a large RL system, keep these boundaries explicit. A PP send can be healthy while EP RDMA is congested. An Engram fetch can consume the same QP or memory budget that dispatch sizing assumed was free. A CP copy can change stream dependencies. Share a buffer only when its maximum concurrent sessions and lifetimes are accounted for.

Requirements and cluster checks

The current README lists Hopper (SM90) GPUs or architectures with SM90 Parallel Thread Execution (PTX) support, CUDA 12.3 or newer for SM90, PyTorch 2.10 or newer, and NCCL 2.30.4 or newer. NVLink is expected for intranode communication and RDMA for internode communication. The repository reports full InfiniBand testing and theoretical compatibility with RDMA over Converged Ethernet (RoCE).^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

Run these checks before starting a multi-node job:

Don't run pip install on a login node and assume the worker image matches. The source's JIT cache and NCCL library path are runtime dependencies. Include their versions, topology dump, and EP_BUFFER_DEBUG=1 initialization output in a launch receipt, while redacting credentials and private host details.

Applications in giant-model post-training

DeepEP is especially relevant when a policy or teacher is a sparse MoE model. During online RL, every sampled trajectory can trigger many MoE layers. The rollout service needs fast expert routing; the trainer needs exact route metadata and reproducible gradients. A slow all-to-all multiplies across generated tokens, while an incorrect or stale handle can corrupt every subsequent update.

Use DeepEP to reason about three placement patterns:

The communication library doesn't solve policy staleness, reward hacking, or checkpoint synchronization. It can, however, become the lowest-level source of latency and numerical mismatch in those systems. Pair route metrics with rollout weight versions, expert-load histograms, and reward receipts. If reward falls after a topology change, first separate transport delay, dtype error, and algorithm behavior.

A runnable routing ledger

The following standard-library example models the metadata transformation without CUDA. It doesn't claim to emulate DeepEP's kernels or bandwidth. It checks the invariant that dispatch creates expert slots and combine reconstructs source rows with gate weights. The leetllm:test marker makes the snippet eligible for the repository's runnable-example checks.

Expected output is:

The dictionary groups rows by expert, standing in for a receive layout. source_index, topk_slot, and weight stand in for handle metadata. A real combine reduces vectors, not strings, and a real dispatch can cross ranks. Keep this tiny invariant visible when reviewing a more complex route.

Strengths and weaknesses

DeepEP's strength is focus. It exposes a compact EP API while specializing kernels for modern GPU and RDMA paths. V2's unified buffer, analytical resource estimates, NCCL communicator reuse, deterministic option, and JIT compilation make the transport boundary inspectable. Its limits are equally concrete.

Weakness isn't the same as a bug. Larger V2 buffers may trade memory for broader topology support, while minimum SM counts can favor overlap over standalone benchmarks. Failed Gin assertions can expose network deployments that don't satisfy the chosen mode. Diagnose against the contract before patching a kernel.

Team, contributors, governance, and licenses

The repository is published under the DeepSeek organization. Its citation lists Chenggang Zhao, Shangyan Zhou, Liyue Zhang, Chengqi Deng, Zhean Xu, Yuxuan Liu, Kuai Yu, Jiashi Li, and Liang Zhao. The README also thanks NCCL contributors and the NCCL team for V2 Gin support. These names establish the source snapshot's credited contributors, not a permanent ranking of current maintainers.^{[1]Reference 1DeepEPhttps://github.com/deepseek-ai/DeepEP/tree/dd758caf451848bd150e1046af3d0a73e5fff38d}

The source's MIT license permits use, modification, and distribution with the required notice. It doesn't grant a license to NCCL or NVSHMEM binaries, model checkpoints, benchmark datasets, or network firmware. A production image should inventory those dependencies separately. The V2 repository says NVSHMEM remains needed for legacy methods, so removing every NVSHMEM package from an image can break a V1 compatibility path even when V2 EP itself uses NCCL Gin.^{[4]Reference 4DeepEP MIT Licensehttps://github.com/deepseek-ai/DeepEP/blob/dd758caf451848bd150e1046af3d0a73e5fff38d/LICENSE[5]Reference 5DeepEP NVSHMEM Installation Guidehttps://github.com/deepseek-ai/DeepEP/blob/dd758caf451848bd150e1046af3d0a73e5fff38d/docs/nvshmem.md}

There is no peer-reviewed DeepEP system paper in this pinned repository. The README contains a project citation and benchmark tables. Use the DeepSeek-V3 report to explain why expert-parallel communication and group-limited routing matter, but don't cite it as proof that every V2 kernel or result came from that paper.^{[3]Reference 3DeepSeek-V3 Technical Report.https://arxiv.org/abs/2412.19437}

Benchmark and failure-mode checklist

Before trusting a number, capture a receipt with:

1. DeepEP commit and JIT/compiler versions.
2. GPU architecture, GPU count, NVLink map, NIC model, and rail topology.
3. CUDA, PyTorch, NCCL, and optional NVSHMEM versions.
4. Tokens per rank, hidden size, top-k, expert count, and expert alignment.
5. Gate histogram, local-bypass rate, and deterministic setting.
6. FP8/BF16 scale layout, SM count, QP count, and hybrid/direct mode.
7. Warm-up policy, JIT time, dispatch time, combine time, and end-to-end layer time.
8. Logical-byte formula and physical-link counters.

Common failures map to specific boundaries:

The most dangerous failure is a plausible tensor with wrong provenance. Add route checksums, per-expert counts, and deterministic small-cluster tests before launching a long RL job. Communication success is not model correctness.

Source-reading map

Read one vertical slice instead of opening every kernel at once:

At each boundary, write down the authoritative state. Router output owns expert choices. EPHandle owns source-to-slot metadata. NCCL owns communicator and window state. The event owns readiness. The expert GEMM owns its local output. Combine should never guess any of those values from decoded text or a second route calculation.

What to remember

• MoE EP turns expert routing into an irregular all-to-all with source metadata and gate weights.
• V2's ElasticBuffer unifies dispatch and combine around a reusable EPHandle and asynchronous EventOverlap.
• Scale-up NVLink and scale-out RDMA have different bandwidth and queue costs. Direct and hybrid modes choose different logical domains.
• NCCL Gin lets V2 reuse communicators and issue device-side communication, but only when the cluster exposes the required capability.
• FP8 dispatch with BF16 combine reduces payload while keeping a higher-precision reduction boundary. Scale metadata remains part of correctness.
• Analytical SM and QP helpers are starting models with balanced-gate assumptions, not universal tuning results.
• Runtime JIT avoids a giant install-time build matrix but makes compiler, cache, and first-use readiness production concerns.
• V1 is an archived NVSHMEM-based API. Experimental PP, CP, Engram, and AGRS paths need separate capacity and maturity checks.
• DeepEP's MIT source license doesn't cover NCCL, NVSHMEM, model weights, datasets, or cluster firmware.