Mixture of Experts Architecture

Suppose a serving team wants one model to handle code, math, and tool use, but each request needs only a small slice of that capability. In a dense Transformer, its Feed-Forward Network (FFN) processes every token with the same weights, so adding capacity raises work for every token.

Mixture of Experts (MoE) breaks that coupling. It keeps many expert FFNs and uses a lightweight router to score them, then sends each token to only a few highest-scoring experts. One token can activate a small amount of expert compute while the model stores a much larger capacity pool.

The tradeoff moves rather than disappears: weights still need a home, routed token states may cross GPUs, and skewed choices can overload one expert. For one token, the router chooses a few experts and blends their outputs. Skipped experts do no expert computation for that token, but their weights still have to be stored or fetched for other tokens.

Switch Transformer^{[1]Reference 1Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.https://arxiv.org/abs/2101.03961} showed that sparse activation could reach trillion-parameter scale. Later open-weight designs such as Mixtral,^{[2]Reference 2Mixtral of Experts.https://arxiv.org/abs/2401.04088} DeepSeek-V2,^{[3]Reference 3DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434} DeepSeek-V3,^{[4]Reference 4DeepSeek-V3 Technical Report.https://arxiv.org/abs/2412.19437} Qwen3.6-35B-A3B,^{[5]Reference 5Qwen3.6-35B-A3Bhttps://huggingface.co/Qwen/Qwen3.6-35B-A3B} GLM-5.2,^{[6]Reference 6GLM-5.2 Model Cardhttps://huggingface.co/zai-org/GLM-5.2} and DeepSeek V4 Flash 0731^{[7]Reference 7DeepSeek-V4-Flash-0731 Model Cardhttps://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731} make different choices about expert pools, routing, attention, and serving. We'll compare those choices after deriving the small case.

What link does MoE break compared with a dense FFN layer?

Answer

A dense FFN uses all of its parameters for every token. MoE separates total capacity from active compute by storing many expert FFNs but routing each token to only a small top-k subset.

From one FFN to many experts

In a standard Transformer layer, the FFN sublayer is one dense network. In an MoE layer, that FFN is replaced by $N$ parallel expert FFNs plus a router (also called a gating network) that picks which $k$ experts handle each token. Those winners are a top-k subset.^{[8]Reference 8Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.https://arxiv.org/abs/1701.06538}

In many modern LLMs, experts replace only the FFN. Attention stays dense and shared across tokens.^{[2]Reference 2Mixtral of Experts.https://arxiv.org/abs/2401.04088[3]Reference 3DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434} A common router is a single linear projection: multiply the token's hidden state by a learned matrix $W_g$ to get $N$ scores. The router chooses a path through the FFN; it doesn't choose which tokens attend to one another.

Before the figure, make two predictions: with scores 0.52, 0.17, 0.26, and 0.05 and top-2 routing, which experts run, and do the skipped experts disappear from memory? The next walkthrough checks both answers. Bar length is the softmax probability, not a decorative meter.

Which parts of a Transformer layer are usually replicated in modern MoE LLMs?

Answer

The FFN sublayer is replaced by expert FFNs. Attention, embeddings, normalization, and most shared block logic usually stay dense and shared.

A concrete routing walkthrough

Before any formula, make a prediction: with top-$k = 2$, only two experts can contribute, but their blend weights won't be the original softmax probabilities. Trace one token through the layer by hand to see where that renormalization comes from.

This is the computation Mixtral-style models run at every token and every MoE layer.

Setup

• One token arrives with a hidden state $x$ of dimension 4.
• There are 4 experts, each a small FFN.
• The router uses top-$k = 2$.

Step 1: Router scores

The router multiplies the token by a learned weight matrix $W_g$ to get a raw score for each expert. Suppose the scores come out as:

Step 2: Softmax probabilities

Apply softmax so the scores become probabilities that sum to 1. Approximate probabilities:

Step 3: Top-k selection

Pick the top 2 experts: Expert 1 (0.52) and Expert 3 (0.26).

Step 4: Renormalize

The selected weights are renormalized so they sum to 1 inside the chosen subset:

• Expert 1: $0.52 / (0.52 + 0.26) \approx 0.67$
• Expert 3: $0.26 / (0.52 + 0.26) \approx 0.33$

Step 5: Expert computation

Only Experts 1 and 3 run. Each processes the same input token $x$ through its own FFN weights.

• $E_1(x)$ produces output vector $o_1$
• $E_3(x)$ produces output vector $o_3$

Step 6: Weighted sum

The layer output blends the two expert outputs with the renormalized gate weights:

$$\text{Output} = 0.67 \cdot E_1(x) + 0.33 \cdot E_3(x)$$

Experts 2 and 4 never process this token, so they contribute no expert-FFN FLOPs for this decision. Their weights still need to be stored or fetched for other tokens that may select them.

The next snippet uses the exact logits, not the rounded table, and should recover the same selected pair and 0.67 / 0.33 split.

What are the six steps for tracing one token through top-k MoE routing?

Answer

Compute router logits, apply softmax, select the top-k experts, renormalize the selected weights, run only the selected expert FFNs, then blend their outputs with the renormalized gate weights.

The general MoE layer formula

The same walkthrough is one compact equation. For an input token $x$:

$$\text{MoE}(x) = \sum_{i \in \text{TopK}(g(x))} \tilde{g}_i(x) \cdot E_i(x)$$

The router computes a gate distribution $g(x)$ over all $N$ experts. Keep the configured top-$k$ entries, zero the rest, and renormalize the survivors into $\tilde{g}(x)$ so those remaining weights sum to 1. Mixtral uses top-2 out of 8; later architectures choose different expert counts and scoring functions.^{[2]Reference 2Mixtral of Experts.https://arxiv.org/abs/2401.04088[3]Reference 3DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434} Each selected expert $E_i$ processes $x$ independently, then the outputs are blended with $\tilde{g}_i$.

Why do selected expert weights need renormalization after top-k selection?

Answer

The original softmax probabilities include experts that are later zeroed out. Renormalization makes the surviving top-k weights sum to 1, so the blended output keeps the intended scale.

The weighted sum explains one token's result. The next question is resource accounting: what grows with the full expert pool, and what stays tied to $k$?

Why sparse activation can pay off

Two quantities now diverge. An MoE layer can keep per-token expert FLOPs tied mostly to $k$ while total weight residency follows all experts. Distributed serving adds token-dispatch traffic, so active parameters alone can't predict latency.

Adding experts while keeping top-$k$ fixed grows the FFN parameter pool, but each token still runs only a small subset. Routing, batching, and communication still add overhead, so "same active parameters" isn't identical latency.

Compute per token is sparse. The serving system still needs access to the full expert pool, so memory footprint and communication often scale with total parameters, not active parameters.

Only the FFN blocks are replicated across experts. Attention layers, embeddings, normalization layers, and the router stay shared. That's why Mixtral 8x7B is about 46.7B total parameters and about 12.9B active parameters per token, not the naive 56B and 14B you would get from multiplying everything by 8. The Mixtral paper rounds those counts to 47B and 13B.^{[2]Reference 2Mixtral of Experts.https://arxiv.org/abs/2401.04088}

Active parameters mostly drive per-token expert FLOPs. Total parameters drive weight memory and checkpoint size. Network traffic depends on where selected experts live.

If an MoE has 46.7B total parameters and 12.9B active parameters per token, which number drives FLOPs and which number drives memory residency?

Answer

Active parameters mostly drive per-token expert FLOPs. Total parameters drive checkpoint size and the amount of expert weight memory the serving system must keep available.

A top-k router you can run

The walkthrough hid the matrix multiply. Here the gate is an explicit $4 \times 4$ matrix. For the unit vector $x = [1, 0, 0, 0]$, predict what $W_g x$ should return: the first column contains the four logits, so the same top-2 pair and weights should come back.

Switch Transformer used top-1 routing. Mixtral and DeepSeek-V2 select multiple routed experts per token.^{[1]Reference 1Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.https://arxiv.org/abs/2101.03961[2]Reference 2Mixtral of Experts.https://arxiv.org/abs/2401.04088[3]Reference 3DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434} Extra selected experts change quality, compute, capacity, and communication. They aren't automatically a better serving configuration.

What additional trade-off does top-2 routing introduce compared with top-1?

Answer

Top-2 lets a token blend two routed outputs, but it also executes and dispatches more expert work per token. Whether that trade improves quality or serving efficiency has to be measured for the architecture and hardware.

Alternative: expert choice routing

The standard top-k router forces each token to choose its top $k$ experts. Expert Choice Routing^{[9]Reference 9Mixture-of-Experts with Expert Choice Routinghttps://arxiv.org/abs/2202.09368} flips the mechanism: each expert selects a fixed bucket of top-scoring tokens from the current batch.

Each expert processes a fixed token budget, so expert-side load is predictable and easier to schedule. Coverage moves to the token side: one token may land in many expert buckets, one bucket, or none unless the design adds coverage rules. Token-choice routing with balancing losses remains the simpler picture for open-weight MoE LLMs. Expert-choice routing still shows up in training-efficiency papers.

What trade-off does expert choice routing make compared with token choice top-k routing?

Answer

Expert choice gives each expert a fixed token budget, so load balancing is easy. The risk moves to the token side: some tokens may get many experts while others may be skipped unless the design enforces coverage.

Why routers collapse without a balancing policy

During training, routers can drift toward expert collapse: many tokens route to a few popular experts while others starve. Load statistics tell you whether this is occurring. They don't prove that a perfectly uniform router is best for model quality.

Suppose four interchangeable experts start with nearly tied scores, but E1 wins slightly more often on common tokens. Predict the next step: E1 receives more assignments, so it gets more gradient updates. It can improve faster on the tokens it already sees, which makes the router even more likely to choose it.

That feedback starves E2 through E4. After enough steps, E1 may take most assignments while the others stay under-trained. The model still reserves memory for all four experts, but that isn't a math expert plus three specialists. It's one overworked expert and three idle parameter blocks.

The histograms use the same 60 / 20 / 15 / 5 split as the numeric loss check later. Compare them with quality, overflow, and communication before you pick a balancing policy.

The router is trained jointly with the rest of the network.^{[8]Reference 8Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.https://arxiv.org/abs/1701.06538} Inspect assignment shares, overflow, task loss, and communication load together.

Why does expert collapse reinforce itself during training?

Answer

The router sends more tokens to an expert that looks slightly better. That expert gets more gradient updates, improves faster, and becomes even more attractive to the router, starving other experts.

An auxiliary load-balancing loss

How can a training run push back against that feedback loop? Switch Transformer uses a top-1 auxiliary loss that penalizes uneven routing.^{[1]Reference 1Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.https://arxiv.org/abs/2101.03961} This form is also a useful teaching calculation for top-$k$ routing when $f_i$ is normalized over routed assignments:

$$\mathcal{L}_{\text{balance}} = \alpha \cdot N \sum_{i=1}^N f_i \cdot p_i$$

Multiply each expert's routed-assignment fraction $f_i$ by its average routing probability $p_i$ and sum them. If the router sends too many assignments to one expert (high $f_i$) and gives it high probability (high $p_i$), the product is large, so the loss penalizes that concentration.

It's a balancing pressure, not proof that equal utilization maximizes task quality.

where:

• $f_i = \frac{\text{assignments routed to expert } i}{\text{total routed assignments}}$; with top-$k$, the denominator is $T \times k$
• $p_i = \frac{1}{T}\sum_{t=1}^T \text{softmax}(W_g \cdot x_t)_i$ (average router probability)
• $\alpha$ is a configurable coefficient (Switch reports experiments including $10^{-2}$)
• $N$ is the number of experts

A balanced reference point is $f_i \approx p_i \approx 1/N$. A trained deployment still needs task-quality and systems measurements before you judge its observed specialization.

Here's a concrete top-1 numeric check. Suppose we have 4 experts and a mini-batch of 100 tokens:

If the load were perfectly uniform, each expert would get $f_i = p_i = 0.25$, and the sum would be $4 \times (0.25 \times 0.25) = 0.25$. With positive $\alpha$, this auxiliary term applies pressure against the imbalanced 0.400 case. Tune its weight with task-quality measurements, because too much balancing pressure can fight the language-modeling objective.

The auxiliary loss says "use the expert pool evenly enough." The language-modeling loss still decides whether the model predicts the next token well.

In the load-balancing loss, what do $f_i$ and $p_i$ measure?

Answer

$f_i$ measures how many tokens were actually routed to expert $i$. $p_i$ measures the router's average probability for expert $i$. A high value for both means the expert is both preferred and overloaded.

Balancing asks how assignments are distributed on average. Capacity asks what happens when one batch is bursty and a hot expert fills its buffer.

Capacity factor and overflow policy

Start with a concrete batch: 4096 tokens choose top-2 from 32 experts. Uniform routing would send 256 token-slots to each expert. If one hot expert receives 400 assignments, 144 need an overflow policy.

In a batched environment, many MoE implementations use fixed-size expert buffers so the layer can run predictable tensor operations. The capacity factor multiplies that uniform slot estimate:

$$\text{Expert Capacity} = \frac{\text{Tokens per Batch} \times k}{N} \times \text{Capacity Factor}$$

That expression is the average number of tokens each expert would receive under perfectly uniform routing, then multiplied by the capacity factor. A factor of $1.0$ sizes the buffer for uniform routing. Setting it to $1.2$ lets an expert take 20% more tokens than the average.

What happens after an expert is assigned more tokens than its buffer allows depends on implementation:

1. Dropping or pass-through: Switch-style designs can skip overflowed expert computations and rely on the residual path for those token-layer assignments.^{[1]Reference 1Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.https://arxiv.org/abs/2101.03961}
2. Rerouting: A system can send overflow to another eligible expert, changing load and communication.
3. Dropless execution: A system can execute all assignments with variable work, keeping quality paths while accepting less predictable workload.

A higher capacity factor reduces overflow risk but reserves more buffer space and may waste compute on empty slots. A lower factor is more compact, so whichever overflow policy you selected matters more.

Why does a capacity factor of 1.2 reduce overflow risk versus 1.0, and what must you know when an expert is full?

Answer

1.2 gives each expert a 20% buffer above uniformly distributed assignments, reducing overflow. The result of overflow is an implementation policy: it may drop, reroute, or execute extra assignments, with different quality and throughput costs.

How modern architectures scale the idea

Routing, balance, and capacity are the knobs. The design question is now whether a larger expert pool adds useful capacity without making per-token compute, memory, or communication unmanageable. Production models turn those knobs in different directions: fewer coarse experts, many fine-grained routed experts, always-on shared experts, and routers that are no longer a plain softmax.

Mixtral 8x7B[2]Reference 2Mixtral of Experts.https://arxiv.org/abs/2401.04088

MoE is applied only to the FFN layers. Attention stays shared. Mixtral uses a SwiGLU (Swish Gated Linear Unit) expert, a Gated Linear Unit (GLU) variant that improved Transformer quality in the GLU-variants paper.

DeepSeek-V2[3]Reference 3DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434

DeepSeek-V2 extends coarse-grained MoE in several ways:

• Fine-grained experts: 160 smaller routed experts instead of a handful of coarse experts, which gives the router many more combinations per token.^{[3]Reference 3DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434}
• Shared experts: 2 experts are always active for all tokens. The design aims to capture common computation and reduce redundancy among routed experts.^{[3]Reference 3DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434}
• Device-limited routing: selected routed experts for each token are constrained to a small number of devices (the paper reports $M \ge 3$ as a practical setting), which bounds cross-device communication.^{[3]Reference 3DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434}
• Multi-level balancing losses: expert-level, device-level, and communication-balance losses for cluster-scale routing.^{[3]Reference 3DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434}
• Multi-head Latent Attention (MLA): compresses the KV cache into a low-rank latent vector.^{[3]Reference 3DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434}

DeepSeek's report pairs MLA with FP8 weights and KV-cache quantization averaging 6 bits. On one node with 8 H800 GPUs, using the prompt and generation-length distribution from its deployed DeepSeek 67B service, it reports more than 50K tokens per second, or 5.76× the DeepSeek 67B maximum generation throughput.^{[3]Reference 3DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434} The paper doesn't give a separate output-correctness tolerance for this comparison, so reproduce the claim only with an equivalence check on your own workload.

Cell counts in the comparison are exact. Tile sizes are schematic.

DeepSeek-V3[4]Reference 4DeepSeek-V3 Technical Report.https://arxiv.org/abs/2412.19437

DeepSeek-V3 keeps the DeepSeekMoE pattern at a larger reported scale. The first three layers stay dense FFNs; later layers are MoE. Compare it with the softmax walkthrough on two choices: how V3 selects routed experts and how it balances them.

Each MoE layer has 1 shared expert and 256 routed experts, with 8 routed experts activated per token.^{[4]Reference 4DeepSeek-V3 Technical Report.https://arxiv.org/abs/2412.19437}

DeepSeek-V3 also changes scoring. It computes token-to-expert affinity with sigmoid scores, normalizes the selected scores, and adds a per-expert bias only when choosing the top-$k$ routed experts. Training adjusts that bias from observed expert load. The weighted expert output still uses the original affinity score, not the routing bias.^{[4]Reference 4DeepSeek-V3 Technical Report.https://arxiv.org/abs/2412.19437}

The paper calls the main batch-wise mechanism auxiliary-loss-free because it doesn't rely on an auxiliary loss to keep expert load balanced across a training batch. It still keeps a small complementary sequence-wise balance loss to prevent extreme imbalance inside one sequence.^{[4]Reference 4DeepSeek-V3 Technical Report.https://arxiv.org/abs/2412.19437} The softmax router you implemented is a teaching sketch, not a universal contract.

GLM-5.2 and DeepSeek V4 Flash 0731

These releases make stored-versus-active capacity impossible to ignore:

This table is not a quality ranking. GLM-5.2 stores much more expert parameter capacity and targets long-horizon coding and tool workflows. Flash 0731 activates a smaller path and targets efficient agentic coding and serving.

Two V4 details change how you read the earlier softmax router. Every Transformer block is an MoE layer, and the first three MoE layers use hash routing from token identity instead of a learned gate.^{[12]Reference 12DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencehttps://arxiv.org/abs/2606.19348} Affinity scores use $\sqrt{\mathrm{softplus}(\cdot)}$ rather than V3's sigmoid, still with auxiliary-loss-free bias plus a small sequence-wise balance loss.^{[12]Reference 12DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencehttps://arxiv.org/abs/2606.19348}

Both checkpoints still need full-weight residency planning and workload-local evals. Inter-device expert communication and sparse kernels may be required for intended performance, depending on runtime and topology.^{[6]Reference 6GLM-5.2 Model Cardhttps://huggingface.co/zai-org/GLM-5.2[13]Reference 13GLM-5.2 Configurationhttps://huggingface.co/zai-org/GLM-5.2/blob/main/config.json[7]Reference 7DeepSeek-V4-Flash-0731 Model Cardhttps://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731[12]Reference 12DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencehttps://arxiv.org/abs/2606.19348}

Qwen3.6 sparse MoE checkpoint[5]Reference 5Qwen3.6-35B-A3Bhttps://huggingface.co/Qwen/Qwen3.6-35B-A3B

Qwen3.6 puts the active-versus-total split in the model name:

"A3B" hints at active compute, not full deployment footprint. You still need memory, sharding, quantization, or offload capacity for the full checkpoint, plus serving support for routing and grouped expert execution.

The backbone also interleaves Gated DeltaNet layers with gated attention. That is a sequence-mixer choice, not an FFN-routing choice, and it's the bridge into the next chapter.

What does DeepSeek-style fine-grained MoE add beyond "more experts"?

Answer

It uses many smaller routed experts, shared always-on experts, device-limited routing, and balancing policies designed for cluster-scale communication. The architecture problem is routing quality plus hardware efficiency, not parameter count alone.

The model comparisons answer how capacity is allocated. Serving adds a second question: can the runtime move, group, and execute those routed token states fast enough on its actual devices?

Serving an MoE on real hardware

Sparse compute per token doesn't remove the systems work. Experts often live on different devices, token states have to move, and decode microbatches re-expose that tax every step.

Communication overhead

In distributed training and inference, experts often reside on different devices to meet memory requirements. Routing a token to an expert on another GPU requires transferring the token's hidden state over a high-bandwidth interconnect such as NVLink within a node or InfiniBand across nodes.

That transfer can stall the layer. The usual pattern is an all-to-all dispatch that scatters tokens to the GPUs hosting their selected experts, then an all-to-all combine that returns expert outputs to the original token order. System designers try to overlap those transfers with useful compute so the network isn't the only thing running.

The next count uses four experts on two GPUs. Three of five assignments cross the link.

Memory requirements

Unlike dense models with one FFN per layer, MoE models carry many expert FFNs. Fast GPU deployments generally keep those expert weights resident somewhere in the serving pool so routing doesn't stall on weight fetches.

That is the core MoE paradox: low active compute per token, high total weight residency. Mixtral 8x7B only activates about 12.9B parameters per token, yet the serving system still needs the full ~46.7B-parameter checkpoint available across the cluster.^{[2]Reference 2Mixtral of Experts.https://arxiv.org/abs/2401.04088} Tensor Parallelism (TP) splits individual layer computations across multiple GPUs, which helps large shared layers fit, but it doesn't solve expert placement. Expert Parallelism (EP) places different full experts on different GPUs while keeping every expert reachable when a token needs it.

Why doesn't tensor parallelism alone solve MoE serving?

Answer

Tensor parallelism shards individual matrix operations. MoE also needs expert placement, dispatch, and combine across whole expert FFNs, so serving systems use expert parallelism to keep different experts reachable across devices.

Training instability

MoE training can be more sensitive than training a comparable dense Transformer. The top-$k$ routing step introduces discontinuities at expert-selection boundaries, and small routing biases can snowball into expert collapse. Gradients still flow through the selected experts and their gate weights, but the router is noisier to optimize than a dense FFN.

Router z-loss^{[17]Reference 17ST-MoE: Designing Stable and Transferable Sparse Expert Modelshttps://arxiv.org/abs/2202.08906} penalizes unnecessarily large routing logits; the ST-MoE paper reports improved training stability from this term. Input jitter is another candidate to test, but ST-MoE's XL-scale ablation improved stability while reducing quality.^{[17]Reference 17ST-MoE: Designing Stable and Transferable Sparse Expert Modelshttps://arxiv.org/abs/2202.08906} Track training loss, routing distribution, overflow, and downstream quality rather than assuming one stabilizer is free.

Expert parallelism layout

Large distributed MoE deployments commonly use Expert Parallelism. TP splits individual matrix multiplications. EP maps entire experts to specific GPUs. Real deployments can combine EP with TP, sequence parallelism, data parallelism, and redundant expert placement. DeepSeek-V3 describes a decoding layout with TP4 for attention and EP320 for the MoE path.^{[4]Reference 4DeepSeek-V3 Technical Report.https://arxiv.org/abs/2412.19437}

The two-GPU sketch below matches the traffic snippet: GPU0 owns E0 and E1, GPU1 owns E2 and E3, and three of five token states cross the link. Expert weights never migrate.

At a larger 96-expert split the same ownership rule still holds:

Sizing EP traffic

A lower bound on dispatch + combine volume for $T$ tokens with top-$k$ routing, hidden size $H$, and element size $b$ bytes is:

$$\text{bytes}_{\text{dispatch+combine}} \approx 2 \cdot T \cdot k \cdot H \cdot b$$

(plus small routing metadata). Compare that wire cost to expert GEMM work on a device that receives $T_{\text{exp}}$ token-slots: roughly $4 \cdot T_{\text{exp}} \cdot H \cdot I$ FLOPs for a two-projection FFN with intermediate size $I$, counting one multiply and one add as two FLOPs in each of the up and down projections. A gated three-projection FFN is roughly $6 \cdot T_{\text{exp}} \cdot H \cdot I$ FLOPs under the same convention. When interconnect is weak or $T$ is small (decode microbatches), the $2TkHb$ term can dominate even if active FLOPs look cheap.

Balanced global expert histograms aren't enough. If many popular experts land on the same EP rank, that rank's per-device token histogram spikes, capacity overflows, and the all-to-all waits on a straggler. Device-limited routing and capacity policy should be checked against per-rank token counts as well as expert-ID frequencies.

Prefill vs decode under EP: large prefill batches can amortize dispatch/combine across many tokens per expert. Decode microbatches re-expose router, grouping, and all-to-all latency every step, which is why dense paths sometimes win p95 at batch size 1 even when MoE wins throughput.

Expert prefetching

For CPU-offloaded serving (for example, running a quantized MoE checkpoint on consumer hardware), one possible design is:

1. Keep attention weights and router weights in GPU memory (always needed).
2. Keep expert weights on CPU or SSD.
3. After a layer's router decides which experts are needed, transfer those expert weights before its expert computation.
4. Optionally use prediction or caching to stage likely later-layer experts, accepting misses or unused transfers.

This can make large MoE checkpoints fit where full GPU residency can't, especially when combined with weight quantization, but it trades memory for transfer-dependent latency. You can't know an exact next-layer route until that layer receives its hidden state, so unconditional "prefetch the next experts" isn't free.

Batch routing efficiency

In batch serving, different tokens in a batch may route to different experts. Batched expert execution is the serving trick:

1. Router assigns all tokens in the batch to their top-k experts.
2. Group tokens by expert assignment.
3. Execute each expert as a batched matrix multiply over its assigned tokens.
4. Scatter results back to original token positions.

That groups many per-token expert calls into a smaller number of larger matrix multiplies, which helps keep GPU utilization high.

Benchmark slices instead of memorizing a crossover:

Exact outcomes depend on kernels, expert placement, interconnect, and request mix. MoE adds router computation, expert selection, token grouping, and sometimes all-to-all communication. Batching can amortize that overhead, but only measurements on the target system establish whether MoE wins latency or throughput.

Treat the next numbers as a fixture you would replace with benchmark output from the serving stack under evaluation.

These values are intentionally synthetic. For a real speed or latency claim, record accelerator and software versions, request shape and batch or concurrency, precision and kernel path, exact dense baseline, and a correctness check beside the measurement. Without that packet, 34.1% is a fixture result, not a portable benchmark.

Why can dense models beat MoE on single-request latency even when MoE has fewer active parameters?

Answer

MoE adds router work, token grouping, expert dispatch, and often all-to-all communication. With batch size 1, that overhead may dominate. Larger batches amortize it across many routed tokens.

The serving measurements expose the real tradeoff. Now turn them into a deployment choice instead of inferring a winner from active-parameter counts.

When MoE is worth the complexity

Use MoE when the problem needs more model capacity at roughly fixed active compute, and the serving stack can absorb routing complexity. Prefer a dense model when the bottleneck is single-request latency, weak interconnect, or full-checkpoint memory.

Don't stop at "this model activates fewer parameters." Ask whether the target workload still gains quality or throughput after full weight residency, routing overhead, interconnect cost, and operational visibility.

What is the fastest practical dense-vs-MoE check?

Answer

Compare dense and MoE on the real serving regime. Require quality to meet the bar, p95 latency to fit the product budget, throughput to justify the extra routing machinery, and memory residency to fit the deployment plan.

Use each failure below as a hypothesis to test. Start with the symptom, trace the resource or routing signal behind it, then change one policy and rerun the same workload.

When MoE serving breaks down

"Active FLOPs are small, so deployment memory will be small too"

• Symptom: FLOP estimates look cheap, but deployment still runs out of VRAM.
• Cause: Team budgeted only for active parameters per token and forgot that the full expert pool must still be resident somewhere in the serving system.
• Fix: Separate active expert FLOPs from total weight residency. Size memory for the full expert pool as well as the routed path.

"Router imbalance will fix itself later"

• Symptom: One or two experts receive most tokens by early training.
• Cause: Router found a slightly better path, that expert got more gradients, and collapse reinforced itself.
• Fix: Track routing histograms from the start and tune the load-balancing policy before unused experts go stale. Evaluate router z-loss separately for numerical stability, and inspect capacity plus overflow policy separately for dropped or delayed work.

"Sparse compute guarantees lower latency"

• Symptom: MoE benchmark wins disappear at batch size 1.
• Cause: Sparse expert FLOPs aren't the whole latency story. Router work, grouping, dispatch, and all-to-all traffic can dominate single-request latency.
• Fix: Evaluate MoE on the real serving regime. Use dense models for low-latency single requests when routing overhead outweighs sparse compute gains.

"Expert placement is a later infra detail"

• Symptom: Throughput drops sharply after experts are split across GPUs.
• Cause: Expert placement created communication hotspots. Tokens now bounce across devices faster than grouped GEMMs can amortize the transfers.
• Fix: Treat routing and placement as one systems problem. Use expert parallelism, device-aware routing, and batching to keep all-to-all traffic under control.

"Balanced histograms mean routing is healthy"

• Symptom: Quality drops even though router histograms look balanced.
• Cause: Experts may be hitting capacity limits, and the configured overflow policy is dropping, rerouting, or delaying assignments.
• Fix: Inspect overflow counts and policy directly. Evaluate capacity factor, routing balance, burstiness, and throughput before assuming the issue is elsewhere.

"Expert-choice routing covers every token automatically"

• Symptom: Expert-choice routing leaves some tokens underprocessed.
• Cause: Experts select their favorite tokens first, so token coverage is no longer guaranteed automatically.
• Fix: Add explicit coverage rules or use token-choice top-k routing when every token must reliably receive expert computation.

"Experts are neat human-labeled topic buckets"

• Symptom: Teams describe experts as clean topic buckets like "math expert" or "biology expert."
• Cause: Human-friendly labels are being projected onto behavior that often follows token patterns, formatting, or structural clusters instead.
• Fix: Validate specialization with routing traces, activation analysis, and targeted prompts that test a claimed specialty before telling a story about it.

What's the simplest way to tell whether someone understands MoE serving cost?

Answer

They separate three quantities: active FLOPs per token, total expert weight residency, and communication cost from routing tokens to experts across devices.

Replace selected dense FFNs with routed expert sets and activate only the configured paths per token. A real serving claim still has to pass the router trace, total-parameter versus active-compute distinction, balance, overflow policy, memory residency, and communication checks.