Mixture of Experts Architecture

Serving pressure shows up as KV-cache memory, long contexts, batching, and GPU capacity. Mixture of Experts (MoE) attacks a different bottleneck inside the Transformer layer itself.

You've seen how a Transformer layer works: self-attention mixes information across tokens, then a Feed-Forward Network (FFN) processes each token independently. That FFN is dense. Every single weight in it participates in every single forward pass, no matter what the input says. This is simple, but it's also expensive. If you want the model to know more facts, handle more syntax, or reason about more domains, the standard recipe is to make the FFN wider or deeper. Every extra neuron adds parameters and adds compute for every token.

Mixture of Experts (MoE) breaks that link. Instead of one giant FFN, an MoE layer keeps many smaller FFNs in parallel and uses a lightweight router to decide which ones to run for each token. This means the model can have hundreds of billions of total parameters while only spending compute on a small subset of them for any given input.

MoE is one widely used way to separate model capacity from per-token compute. Switch Transformer^{[1]Reference 1Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.https://arxiv.org/abs/2101.03961} popularized sparse activation in trillion-parameter models. Open-weight systems now span Mixtral^{[2]Reference 2Mixtral of Experts.https://arxiv.org/abs/2401.04088}, DeepSeek-V2^{[3]Reference 3DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434}, DeepSeek-V3^{[4]Reference 4DeepSeek-V3 Technical Report.https://arxiv.org/abs/2412.19437}, Qwen3.6-35B-A3B^{[5]Reference 5Qwen3.6-35B-A3Bhttps://huggingface.co/Qwen/Qwen3.6-35B-A3B}, GLM-5.2^{[6]Reference 6GLM-5.2 Model Cardhttps://huggingface.co/zai-org/GLM-5.2}, and DeepSeek V4 Flash 0731.^{[7]Reference 7DeepSeek-V4-Flash-0731 Model Cardhttps://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731} They share sparse activation, but expert counts, attention designs, active paths, and serving costs differ sharply.

What link does MoE break compared with a dense FFN layer?

Answer

A dense FFN uses all of its parameters for every token. MoE separates total capacity from active compute by storing many expert FFNs but routing each token to only a small top-k subset.

From one FFN to many experts

In a standard Transformer layer, the FFN sublayer is a single dense network. In an MoE layer, that one FFN is replaced by $N$ parallel "expert" FFNs, plus a small router (also called a gating network) that decides which $k$ experts handle each token. Those winners form a small top-k subset.^{[8]Reference 8Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.https://arxiv.org/abs/1701.06538}

In many modern LLMs, the experts replace the FFN sublayer while attention stays dense and shared across all tokens.^{[2]Reference 2Mixtral of Experts.https://arxiv.org/abs/2401.04088[3]Reference 3DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434} A common router design is a single linear projection from the token's hidden state down to $N$ scores.

Which parts of a Transformer layer are usually replicated in modern MoE LLMs?

Answer

The FFN sublayer is replaced by expert FFNs. Attention, embeddings, normalization, and most shared block logic usually stay dense and shared.

A concrete routing walkthrough

Before any formula or code, trace one token through an MoE layer by hand. This is the shape of the computation that happens inside models like Mixtral on every single token.

Setup

• One token arrives with a hidden state $x$ of dimension 4.
• There are 4 experts, each a small FFN.
• The router uses top-$k = 2$ (it picks the 2 best experts).

Step 1: Router scores

The router multiplies the token by a learned weight matrix $W_g$ to get a raw score for each expert. Suppose the scores come out as:

Step 2: Softmax probabilities

Apply softmax so the scores become probabilities that sum to 1. Approximate probabilities:

Step 3: Top-k selection

Pick the top 2 experts. That's Expert 1 (0.52) and Expert 3 (0.26).

Step 4: Renormalize

The selected weights are renormalized so they sum to 1 again within the chosen subset:

• Expert 1: $0.52 / (0.52 + 0.26) \approx 0.67$
• Expert 3: $0.26 / (0.52 + 0.26) \approx 0.33$

Step 5: Expert computation

Only Experts 1 and 3 run. Each processes the same input token $x$ through its own FFN weights.

• $E_1(x)$ produces output vector $o_1$
• $E_3(x)$ produces output vector $o_3$

Step 6: Weighted sum

The final layer output blends the two expert outputs using the renormalized gate weights:

$$\text{Output} = 0.67 \cdot E_1(x) + 0.33 \cdot E_3(x)$$

That's it. The other two experts never process this token, so they contribute no expert-FFN FLOPs for this routing decision. Their weights still need to be stored or fetched by the serving system for other tokens that may select them.

What are the six steps for tracing one token through top-k MoE routing?

Answer

Compute router logits, apply softmax, select the top-k experts, renormalize the selected weights, run only the selected expert FFNs, then blend their outputs with the renormalized gate weights.

The general MoE layer formula

We can write the same walkthrough as a compact equation. For an input token $x$, the MoE layer output is:

$$\text{MoE}(x) = \sum_{i \in \text{TopK}(g(x))} \tilde{g}_i(x) \cdot E_i(x)$$

Reading the formula: the router computes a gate distribution $g(x)$ over all $N$ experts. We keep only the configured top-$k$ entries, zero out the rest, and renormalize the survivors into $\tilde{g}(x)$ so those remaining weights sum to 1. Mixtral uses top-2 out of 8, while other architectures choose different expert counts and routing policies.^{[2]Reference 2Mixtral of Experts.https://arxiv.org/abs/2401.04088[3]Reference 3DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434} Each selected expert $E_i$ processes $x$ independently, then their outputs are blended using the renormalized gates $\tilde{g}_i$.

The architecture figure above is the visual form of this equation: the router scores every expert, keeps the top-$k$, runs only those FFNs, then blends the selected outputs.

Why do selected expert weights need renormalization after top-k selection?

Answer

The original softmax probabilities include experts that are later zeroed out. Renormalization makes the surviving top-k weights sum to 1, so the blended output keeps the intended scale.

Why sparse activation can pay off

A dense feed-forward layer applies the same parameter matrix to every token. An MoE layer stores several expert matrices but routes each token through only top-$k$ experts. Per-token expert FLOPs therefore track active experts, while memory tracks all resident expert weights and distributed serving adds token-dispatch traffic.

The main benefit is that MoE can scale total parameters faster than per-token compute. By adding more experts while keeping top-$k$ fixed, the model gets a larger pool of FFN parameters, but each token still runs only a small subset. The expert FLOPs stay roughly tied to the active experts, not the full expert pool. Routing, batching, and communication still add overhead, so "same active parameters" doesn't mean identical latency.

That doesn't mean serving cost scales only with active parameters. Compute per token is sparse, but the system still needs access to the full expert pool, so memory footprint and communication often scale with total parameters, not active parameters.

Only the FFN blocks are replicated across experts. Attention layers, embeddings, normalization layers, and the router are still shared. That's why Mixtral 8x7B exposes 46.7B total parameters and about 12.9B active parameters per token, not the naive 56B and 14B you would get from multiplying everything by 8.^{[2]Reference 2Mixtral of Experts.https://arxiv.org/abs/2401.04088}

The split matters: active parameters mostly drive per-token expert FLOPs, while total parameters drive weight memory and checkpoint size. Network traffic depends on where selected experts live.

If an MoE has 46.7B total parameters and 12.9B active parameters per token, which number drives FLOPs and which number drives memory residency?

Answer

Active parameters mostly drive per-token expert FLOPs. Total parameters drive checkpoint size and the amount of expert weight memory the serving system must keep available.

The router in code

Here's a basic PyTorch implementation of the standard top-k softmax router. It takes token embeddings, computes scores for each expert, and returns the indices and normalized weights for the top-$k$ choices.

Switch Transformer used top-1 routing, while Mixtral and DeepSeek-V2 use multiple selected routed experts per token.^{[1]Reference 1Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.https://arxiv.org/abs/2101.03961[2]Reference 2Mixtral of Experts.https://arxiv.org/abs/2401.04088[3]Reference 3DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434} Selecting more experts changes quality, compute, capacity, and communication trade-offs; it isn't automatically a better serving configuration.

What additional trade-off does top-2 routing introduce compared with top-1?

Answer

Top-2 lets a token blend two routed outputs, but it also executes and dispatches more expert work per token. Whether that trade improves quality or serving efficiency has to be measured for the architecture and hardware.

Alternative: expert choice routing

While the standard Top-K router forces each token to choose its top $k$ experts, Expert Choice Routing^{[9]Reference 9Mixture-of-Experts with Expert Choice Routinghttps://arxiv.org/abs/2202.09368} flips the mechanism: each expert selects a fixed bucket of top-scoring tokens from the current batch.

Each expert processes a fixed token budget, so expert-side load is predictable and hardware utilization is easier to schedule. The trade-off moves to token coverage: one token may land in many expert buckets, one bucket, or none unless the design adds coverage rules. Token-choice routing with balancing losses remains the simpler mental model for open-weight MoE LLMs; expert-choice routing is still worth knowing when you read training-efficiency papers.

What trade-off does expert choice routing make compared with token choice top-k routing?

Answer

Expert choice gives each expert a fixed token budget, so load balancing is easy. The risk moves to the token side: some tokens may get many experts while others may be skipped unless the design enforces coverage.

Why routers collapse without a balancing policy

During training, routers can drift toward expert collapse: many tokens route to a few "popular" experts while others are underutilized. Load statistics tell you whether this is occurring; they don't prove that a perfectly uniform router is best for model quality.

Start with four interchangeable experts labeled E1 through E4. Early in training, E1 happens to win slightly higher gate scores on common tokens. The router therefore assigns E1 more often. Because those tokens flow through E1, E1 receives more gradient updates and becomes even better at the tokens it already sees. The preference reinforces itself. After enough steps, E1 may take most assignments while E2 through E4 stay under-trained. The model still reserves memory for all four experts, but it's effectively operating with one overworked expert. That is load collapse, not proof that E1 became a "math expert" or that the idle ones own other domains.

The router is trained jointly alongside the rest of the network.^{[8]Reference 8Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.https://arxiv.org/abs/1701.06538} If a particular expert looks better for common tokens early in training, the router can send it more assignments. That expert then receives more gradient updates, which can reinforce the preference and leave other experts under-trained. The operational test isn't a story about specialization: inspect assignment shares, overflow, task loss, and communication load together.

Why does expert collapse reinforce itself during training?

Answer

The router sends more tokens to an expert that looks slightly better. That expert gets more gradient updates, improves faster, and becomes even more attractive to the router, starving other experts.

An auxiliary load-balancing loss

Switch Transformer uses a top-1 auxiliary loss that penalizes uneven routing.^{[1]Reference 1Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.https://arxiv.org/abs/2101.03961} This form is also a useful teaching calculation for top-$k$ routing when $f_i$ is normalized over routed assignments:

$$\mathcal{L}_{\text{balance}} = \alpha \cdot N \sum_{i=1}^N f_i \cdot p_i$$

Reading the formula: multiply each expert's routed-assignment fraction $f_i$ by its average routing probability $p_i$ and sum them up. If the router sends too many assignments to one expert (high $f_i$) and gives it high probability (high $p_i$), this product is large, so the loss penalizes that concentration. It's a balancing pressure, not proof that equal utilization maximizes task quality.

where:

• $f_i = \frac{\text{assignments routed to expert } i}{\text{total routed assignments}}$; with top-$k$, the denominator is $T \times k$
• $p_i = \frac{1}{T}\sum_{t=1}^T \text{softmax}(W_g \cdot x_t)_i$ (average router probability)
• $\alpha$ is a configurable coefficient (Switch reports experiments including $10^{-2}$)
• $N$ is the number of experts

A balanced reference point is $f_i \approx p_i \approx 1/N$ (uniform distribution). A trained deployment still needs task-quality and systems measurements before you judge its observed specialization.

Here's a concrete top-1 numeric check. Suppose we have 4 experts and a mini-batch of 100 tokens:

If the load were perfectly uniform, each expert would get $f_i = p_i = 0.25$, and the sum would be $4 \times (0.25 \times 0.25) = 0.25$. With positive $\alpha$, this auxiliary term applies pressure against the imbalanced 0.400 case. Tune its weight with task-quality measurements because too much balancing pressure can conflict with the primary objective.

The auxiliary loss is a routing pressure, not the main learning objective. It says, "use the expert pool evenly enough," while the language-modeling loss still decides whether the model predicts the next token well.

In the load-balancing loss, what do $f_i$ and $p_i$ measure?

Answer

$f_i$ measures how many tokens were actually routed to expert $i$. $p_i$ measures the router's average probability for expert $i$. A high value for both means the expert is both preferred and overloaded.

Capacity factor and overflow policy

In a batched environment, many MoE implementations use fixed-size expert buffers so the layer can run predictable tensor operations. This size is determined by the capacity factor:

$$\text{Expert Capacity} = \frac{\text{Tokens per Batch} \times k}{N} \times \text{Capacity Factor}$$

This computes the average number of tokens each expert would receive under perfectly uniform routing, then multiplies by the capacity factor. A factor of $1.0$ means the buffer is exactly sized for uniform routing. Setting it to $1.2$ allows an expert to receive 20% more tokens than the average, providing a safety margin.

What happens if an expert is assigned more tokens than its capacity buffer allows depends on implementation:

1. Dropping or pass-through: Switch-style designs can skip overflowed expert computations and rely on the residual path for those token-layer assignments.^{[1]Reference 1Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.https://arxiv.org/abs/2101.03961}
2. Rerouting: A system can send overflow to another eligible expert, changing load and communication.
3. Dropless execution: A system can execute all assignments with variable work, retaining quality paths while accepting less predictable workload.

This shows the trade-off: a higher capacity factor reduces overflow risk but reserves more buffer space and may waste compute on empty slots. A lower factor is more compact but makes whichever overflow policy you selected more important.

Why does a capacity factor of 1.2 reduce overflow risk versus 1.0, and what must you know when an expert is full?

Answer

1.2 gives each expert a 20% buffer above uniformly distributed assignments, reducing overflow. The result of overflow is an implementation policy: it may drop, reroute, or execute extra assignments, with different quality and throughput costs.

How modern architectures scale the idea

The basic MoE pattern is consistent across modern models, but architects have refined it in interesting ways.

Mixtral 8x7B^{[2]Reference 2Mixtral of Experts.https://arxiv.org/abs/2401.04088}

MoE is applied only to the Feed-Forward Network (FFN) layers. Attention layers are shared across all tokens, which is the standard approach. It uses a SwiGLU (Swish Gated Linear Unit) activation function, a variant of the Gated Linear Unit (GLU) that's been shown to improve Transformer performance.

DeepSeek-V2^{[3]Reference 3DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434}

DeepSeek-V2 uses several extensions to coarse-grained MoE:

• Fine-grained experts: 160 smaller routed experts instead of a handful of coarse experts, which gives the router many more combinations to compose per token.^{[3]Reference 3DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434}
• Shared experts: 2 experts are always active for all tokens. The design aims to capture common computation and reduce redundancy among routed experts; every token gets 2 shared expert outputs in addition to the 6 routed ones.^{[3]Reference 3DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434}
• Device-limited routing: The selected routed experts for each token are constrained to a small number of devices, reducing cross-device communication overhead.^{[3]Reference 3DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434}
• Multi-level balancing losses: DeepSeek-V2 adds separate expert-level, device-level, and communication-balance losses to keep routing efficient at cluster scale.^{[3]Reference 3DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434}
• Multi-head Latent Attention (MLA): Compresses the KV cache into a low-rank latent vector, which the paper reports as a large KV-memory and throughput win relative to DeepSeek 67B.^{[3]Reference 3DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelhttps://arxiv.org/abs/2405.04434}

The diagram contrasts Mixtral's coarse-grained top-2 routing with DeepSeek-V2's fine-grained routed-plus-shared design. It shows the full expert counts, while tile sizes and the example selections are schematic.

DeepSeek-V3^{[4]Reference 4DeepSeek-V3 Technical Report.https://arxiv.org/abs/2412.19437}

DeepSeek-V3 retains the DeepSeekMoE pattern at a larger reported scale.

Each MoE layer has 1 shared expert and 256 routed experts, with 8 routed experts activated per token.^{[4]Reference 4DeepSeek-V3 Technical Report.https://arxiv.org/abs/2412.19437}

DeepSeek-V3 changes the routing details too. It computes token-to-expert affinity with sigmoid scores, normalizes the selected scores, and adds a per-expert bias only when choosing the top-$k$ routed experts. Training adjusts that bias from observed expert load. The weighted expert output still uses the original affinity score, not the routing bias.^{[4]Reference 4DeepSeek-V3 Technical Report.https://arxiv.org/abs/2412.19437}

The paper calls this main batch-wise mechanism auxiliary-loss-free because it doesn't rely on an auxiliary loss to keep expert load balanced across a training batch. It still keeps a small complementary sequence-wise balance loss to prevent extreme imbalance inside one sequence.^{[4]Reference 4DeepSeek-V3 Technical Report.https://arxiv.org/abs/2412.19437} The distinction matters: the standard softmax router above is a useful mental model, not a universal implementation contract.

GLM-5.2 and DeepSeek V4 Flash 0731

These releases make the stored-versus-active distinction impossible to ignore:

This table is not a quality ranking. GLM-5.2 stores much more expert parameter capacity and targets long-horizon coding and tool workflows. Flash 0731 activates a smaller path and targets efficient agentic coding and serving. Both need full-checkpoint residency planning and workload-local evals. Inter-device expert communication and specialized sparse kernels may be needed for intended performance, depending on runtime and topology.^{[6]Reference 6GLM-5.2 Model Cardhttps://huggingface.co/zai-org/GLM-5.2[13]Reference 13GLM-5.2 Configurationhttps://huggingface.co/zai-org/GLM-5.2/blob/main/config.json[7]Reference 7DeepSeek-V4-Flash-0731 Model Cardhttps://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731[12]Reference 12DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencehttps://arxiv.org/abs/2606.19348}

Qwen3.6 sparse MoE checkpoint^{[5]Reference 5Qwen3.6-35B-A3Bhttps://huggingface.co/Qwen/Qwen3.6-35B-A3B}

Qwen3.6 makes the active-versus-total split visible in the model name:

This naming is a useful production reminder. "A3B" hints at active compute, not full deployment footprint. You still need enough memory, sharding, quantization, or offload capacity for the full checkpoint, and you still need serving support for routing, grouped expert execution, and any long-context KV-cache pressure.

What does DeepSeek-style fine-grained MoE add beyond "more experts"?

Answer

It uses many smaller routed experts, shared always-on experts, device-limited routing, and balancing policies designed for cluster-scale communication. The architecture problem is routing quality plus hardware efficiency, not parameter count alone.

When MoE is worth the complexity

Use MoE when the problem needs more model capacity at roughly fixed active compute, and the serving stack can absorb routing complexity. Prefer a dense model when the bottleneck is single-request latency, weak interconnect, or full-checkpoint memory.

The wrong question is "does this model activate fewer parameters?" The useful question is whether the target workload gains enough quality or throughput after you account for full weight residency, routing overhead, interconnect cost, and operational visibility.

What is the fastest practical dense-vs-MoE gate?

Answer

Compare dense and MoE on the real serving regime. Require quality to meet the bar, p95 latency to fit the product budget, throughput to justify the extra routing machinery, and memory residency to fit the deployment plan.

The hardware reality of serving an MoE

Implementing MoE introduces complexities not present in dense models. The compute per token is sparse, but the system still faces several hard serving constraints.

Communication overhead

In distributed training and inference, experts often reside on different devices to meet memory requirements. Routing a token to an expert on another GPU requires transferring the token's hidden state over a high-bandwidth interconnect such as NVLink within a node or InfiniBand across nodes.

This communication can quickly become a bottleneck if not managed carefully. The process usually has an all-to-all dispatch phase that scatters tokens to the GPUs hosting their selected experts, followed by an all-to-all combine phase that returns the expert outputs to the original token order. To maintain high throughput, system designers try to overlap those transfers with useful compute so the network isn't stalling the whole layer.

Memory requirements

Unlike dense models where there's only one FFN per layer, MoE models carry many expert FFNs. Fast GPU deployments generally keep those expert weights resident somewhere in the serving pool so routing doesn't stall on weight fetches.

This creates the core MoE paradox: low active compute per token, but very high total weight residency. A model like Mixtral 8x7B only activates about 12.9B parameters per token, yet the serving system still needs the full 46.7B parameter checkpoint available across the cluster.^{[2]Reference 2Mixtral of Experts.https://arxiv.org/abs/2401.04088} Tensor Parallelism (TP) splits individual layer computations across multiple GPUs, allowing larger shared layers to fit into memory, but it doesn't solve the unique structure of MoE. Expert Parallelism (EP) is needed to place different full experts on different GPUs while keeping every expert reachable when a token needs it.

Why doesn't tensor parallelism alone solve MoE serving?

Answer

Tensor parallelism shards individual matrix operations. MoE also needs expert placement, dispatch, and combine across whole expert FFNs, so serving systems use expert parallelism to keep different experts reachable across devices.

Training instability

MoE training can be more sensitive than training a comparable dense Transformer. The top-$k$ routing step introduces discontinuities at expert-selection boundaries, and small routing biases can snowball into expert collapse. Gradients still flow through the selected experts and their gate weights, but the router is noisier to optimize than a dense FFN.

To combat this, implementations may add stabilizers. Router z-loss^{[17]Reference 17ST-MoE: Designing Stable and Transferable Sparse Expert Modelshttps://arxiv.org/abs/2202.08906} penalizes unnecessarily large routing logits; the ST-MoE paper reports improved training stability from this term. Input jitter is another candidate to test, but ST-MoE's XL-scale ablation improved stability while reducing quality.^{[17]Reference 17ST-MoE: Designing Stable and Transferable Sparse Expert Modelshttps://arxiv.org/abs/2202.08906} The relevant evidence is training loss, routing distribution, overflow, and downstream quality rather than assuming one stabilizer is free or mandatory for every MoE.

Expert parallelism layout

Large distributed MoE deployments commonly use Expert Parallelism (EP). While Tensor Parallelism (TP) splits individual matrix multiplications, EP maps entire experts to specific GPUs. The layout below is deliberately schematic: it illustrates expert ownership and a shared attention path, not one universal sharding plan. Real deployments can combine EP with TP, sequence parallelism, data parallelism, and redundant expert placement. For example, DeepSeek-V3 describes a decoding layout with TP4 for attention and EP320 for the MoE path.^{[4]Reference 4DeepSeek-V3 Technical Report.https://arxiv.org/abs/2412.19437}

The smaller 96-expert sketch looks like this:

Each GPU stores a subset of experts. When a token routes to an expert on another GPU, an all-to-all communication sends the token there and retrieves the result.

Sizing EP traffic

A lower bound on dispatch + combine volume for $T$ tokens with top-$k$ routing, hidden size $H$, and element size $b$ bytes is:

$$\text{bytes}_{\text{dispatch+combine}} \approx 2 \cdot T \cdot k \cdot H \cdot b$$

(plus small routing metadata). Compare that wire cost to expert GEMM work on a device that receives $T_{\text{exp}}$ token-slots: roughly $4 \cdot T_{\text{exp}} \cdot H \cdot I$ FLOPs for a two-projection FFN with intermediate size $I$, counting one multiply and one add as two FLOPs in each of the up and down projections. A gated three-projection FFN is roughly $6 \cdot T_{\text{exp}} \cdot H \cdot I$ FLOPs under the same convention. When interconnect is weak or $T$ is small (decode microbatches), the $2TkHb$ term can dominate even if active FLOPs look cheap.

Balanced global expert histograms are not enough. If many popular experts land on the same EP rank, that rank's per-device token histogram spikes, capacity overflows, and the all-to-all waits on a straggler. Device-limited routing and capacity policy should be checked against per-rank token counts as well as expert-ID frequencies.

Prefill vs decode under EP: large prefill batches can amortize dispatch/combine across many tokens per expert. Decode microbatches re-expose router, grouping, and all-to-all latency every step, which is why dense paths sometimes win p95 at batch size 1 even when MoE wins throughput.

Expert prefetching

For CPU-offloaded serving (for example, running a quantized MoE checkpoint on consumer hardware), one possible design is:

1. Keep attention weights and router weights in GPU memory (always needed).
2. Keep expert weights on CPU or SSD.
3. After a layer's router decides which experts are needed, transfer those expert weights before its expert computation.
4. Optionally use prediction or caching to stage likely later-layer experts, accepting misses or unused transfers.

This can make large MoE checkpoints fit where full GPU residency can't, especially when combined with weight quantization, but it trades memory for transfer-dependent latency. You can't know an exact next-layer route until that layer receives its hidden state, so unconditional "prefetch the next experts" isn't free.

Batch routing efficiency

In batch serving, different tokens in a batch may route to different experts. Batched expert execution is the serving trick:

1. Router assigns all tokens in the batch to their top-k experts.
2. Group tokens by expert assignment.
3. Execute each expert as a batched matrix multiply over its assigned tokens.
4. Scatter results back to original token positions.

This groups many per-token expert calls into a smaller number of larger matrix multiplies, which helps keep GPU utilization high with MoE models.

Benchmark slices instead of memorizing a crossover:

Exact outcomes depend on kernels, expert placement, interconnect, and request mix. MoE adds router computation, expert selection, token grouping, and sometimes all-to-all communication; batching can amortize that overhead, but only measurements on the target system establish whether MoE wins latency or throughput.

The next code block uses an illustrative measurement fixture. Replace these numbers with benchmark output from the serving stack under evaluation.

Why can dense models beat MoE on single-request latency even when MoE has fewer active parameters?

Answer

MoE adds router work, token grouping, expert dispatch, and often all-to-all communication. With batch size 1, that overhead may dominate. Larger batches amortize it across many routed tokens.

When MoE serving breaks down

"Active FLOPs are small, so deployment memory will be small too"

• Symptom: FLOP estimates look cheap, but deployment still runs out of VRAM.
• Cause: Team budgeted only for active parameters per token and forgot that the full expert pool must still be resident somewhere in the serving system.
• Fix: Separate active expert FLOPs from total weight residency. Size memory for the full expert pool as well as the routed path.

"Router imbalance will fix itself later"

• Symptom: One or two experts receive most tokens by early training.
• Cause: Router found a slightly better path, that expert got more gradients, and collapse reinforced itself.
• Fix: Track routing histograms from the start and tune the load-balancing policy before unused experts go stale. Evaluate router z-loss separately for numerical stability, and inspect capacity plus overflow policy separately for dropped or delayed work.

"Sparse compute guarantees lower latency"

• Symptom: MoE benchmark wins disappear at batch size 1.
• Cause: Sparse expert FLOPs aren't the whole latency story. Router work, grouping, dispatch, and all-to-all traffic can dominate single-request latency.
• Fix: Evaluate MoE on the real serving regime. Use dense models for low-latency single requests when routing overhead outweighs sparse compute gains.

"Expert placement is a later infra detail"

• Symptom: Throughput drops sharply after experts are split across GPUs.
• Cause: Expert placement created communication hotspots. Tokens now bounce across devices faster than grouped GEMMs can amortize the transfers.
• Fix: Treat routing and placement as one systems problem. Use expert parallelism, device-aware routing, and batching to keep all-to-all traffic under control.

"Balanced histograms mean routing is healthy"

• Symptom: Quality drops even though router histograms look balanced.
• Cause: Experts may be hitting capacity limits, and the configured overflow policy is dropping, rerouting, or delaying assignments.
• Fix: Inspect overflow counts and policy directly. Evaluate capacity factor, routing balance, burstiness, and throughput before assuming the issue is elsewhere.

"Expert-choice routing covers every token automatically"

• Symptom: Expert-choice routing leaves some tokens underprocessed.
• Cause: Experts select their favorite tokens first, so token coverage is no longer guaranteed automatically.
• Fix: Add explicit coverage rules or use token-choice top-k routing when every token must reliably receive expert computation.

"Experts are neat human-labeled topic buckets"

• Symptom: Teams describe experts as clean topic buckets like "math expert" or "biology expert."
• Cause: Human-friendly labels are being projected onto behavior that often follows token patterns, formatting, or structural clusters instead.
• Fix: Validate specialization with routing traces, activation analysis, and targeted prompts that test a claimed specialty before telling a story about it.

What's the simplest way to tell whether someone understands MoE serving cost?

Answer

They separate three quantities: active FLOPs per token, total expert weight residency, and communication cost from routing tokens to experts across devices.

Sparse routing checklist

The central trick behind large sparse MoE models should be concrete: replace selected dense FFNs with routed expert sets and activate only configured paths per token. A real serving claim still has to pass the router trace, total-parameter versus active-compute distinction, balance, overflow policy, memory residency, and communication checks.