Model Parallelism for LLM Inference

A large language model (LLM) has an estimated 70 GB weight payload. Give each GPU an assumed 80 decimal GB budget and set aside 16 GB for other allocations and headroom. The weights exceed the remaining 64 GB before any request arrives. A runtime may reject this configuration or run out of memory. A second GPU adds capacity, but which tensors belong on each device, and what must cross the wires to recover the model's mathematical result?

Earlier chapters treated a single GPU as a standalone serving box: weights load into VRAM, and then active requests compete for whatever bytes remain for their KV cache. Data-parallel replicas help when a full model copy already fits on one device. They can't shrink an oversized model that doesn't fit in the first place.

Model parallelism divides one model across accelerators. Communication, replicated tensors, stage imbalance, and idle slots affect its benefit. In a coding assistant, weight placement helps a replica load; per-rank KV state and execution work affect which sessions can be admitted within latency targets. Matching the unsharded mathematical computation doesn't guarantee bit-identical floating-point results or sampled text.

Use Qwen3.6-35B-A3B's approximately 35B total and 3B active parameters as the starting estimate.^{[1]Reference 1Qwen3.6-35B-A3Bhttps://huggingface.co/Qwen/Qwen3.6-35B-A3B} At two BF16 bytes per parameter, that gives about 70 decimal GB. MoE routing selects only some experts per token, but a GPU-resident deployment still stores the full expert set across its devices. Offloading changes that placement and adds transfers. Active parameters are a compute proxy, not a full FLOP count or weight-memory estimate.

We'll use 70 GB as our rounded weight checkpoint, with an assumed 16 GB runtime reserve per device. Real production setups replicate some layers, balance shards unevenly, and load multimodal encoders. The matrix examples that follow teach the core sharding mechanics from first principles.

Why inference sharding differs from training

Distributed training focuses on gradients, optimizer states, backward passes, and aggregate throughput across thousands of sample steps. Serving runs on a very different clock: time to first token (TTFT) exposes prompt processing latency, tokens per second (TPS) governs the autoregressive decode loop, and per-token KV cache memory controls concurrency.

The names of parallel techniques carry over from training literature, but their operational costs change completely. An optimization that saves activation memory during a backward pass might do nothing for inference latency, while an engine layout that fits a single prompt might leave pipeline hardware starved for work. Here's how their trade-offs separate:

Serving engines like vLLM expose tensor_parallel_size and pipeline_parallel_size. Checked September 22, 2026: its guide suggests one GPU when sufficient, intra-node TP for a larger model, and TP plus PP across nodes. It also describes intra-node PP for uneven splits or GPUs without NVLink, and allows TP across nodes.^{[2]Reference 2Distributed Inference and Serving.https://docs.vllm.ai/en/latest/serving/parallelism_scaling.html} These are candidates to measure, not transport requirements or speed guarantees.

A logical plan doesn't specify physical transport. TP=4 assigns work to four ranks; it doesn't reveal their topology, shared paths, achieved collective latency, or whether communication is exposed on the request's critical path.

The NVIDIA Collective Communication Library (NCCL) detects hardware topology and dispatches optimized collective primitives like all-reduce, all-gather, and all-to-all across available paths.^{[3]Reference 3NVIDIA Collective Communication Library User Guidehttps://docs.nvidia.com/deeplearning/nccl/user-guide/index.html} Always track the physical interconnect alongside your logical topology: a TP=4 layout over NVLink behaves very differently from TP=4 over PCIe.

Before picking a strategy, write down your per-device memory ledger:

Use two bytes per BF16 parameter for raw payload. A preallocated KV pool may keep its device allocation constant while occupied blocks change. The following even-shard calculation gives a lower bound under its assumed reserve, before KV, replication, and placement constraints:

The two-device bound belongs to this budget, not every deployment. Legal sharding depends on the runtime and architecture. INT4's 17.5 GB raw estimate excludes scales, packing, and unquantized tensors. The official Qwen card gives a TP=8 launch example at 262,144 context; it doesn't prove eight GPUs are required or identify KV as the only reason for that choice.^{[1]Reference 1Qwen3.6-35B-A3Bhttps://huggingface.co/Qwen/Qwen3.6-35B-A3B}

Calculate the same fixture's per-rank remainder:

Tensor parallelism

Tensor parallelism shards operations inside a layer, whether or not that individual layer is too large. Each rank computes with local weight slices and communicates the required results. The participating ranks jointly advance the batch; that doesn't mean all devices perform useful work at every instant.

A rank is an individual execution worker, typically one GPU process. A collective coordinates communication across all ranks in the group. The Megatron-LM formulation makes this partitioning elegant by pairing complementary matrix cuts.^{[4]Reference 4Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.https://arxiv.org/abs/1909.08053}

Consider a standard linear projection:

If we slice $W$ across ranks, each chip computes a partial result. Communication synchronizes those fragments so downstream layers receive the correct mathematical output. Tensor parallelism pays off when GPUs connect over ultra-fast links like NVLink.

The Megatron pattern: column then row

The original Megatron-LM pairing avoids an intermediate communication by matching column and row cuts.^{[4]Reference 4Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.https://arxiv.org/abs/1909.08053} It reduces communication in the block; it doesn't guarantee communication is no longer a bottleneck.

In the MLP block $Z = \text{GeLU}(x A) B$:

1. Column-parallel expansion: Split weight matrix $A$ along its columns into $[A_0 \mid A_1]$. The input vector $x$ is replicated on all ranks. Rank 0 calculates $x A_0$ while Rank 1 calculates $x A_1$. Because GeLU is an element-wise function, each rank applies $\text{GeLU}(\cdot)$ to its local activation slice independently. No cross-device communication happens here. If you had split $A$ by rows, you'd need an immediate all-reduce before the activation, because $\text{GeLU}(u + v) \neq \text{GeLU}(u) + \text{GeLU}(v)$.
2. Row-parallel projection: Split weight matrix $B$ along its rows into $\begin{bmatrix} B_0 \\ B_1 \end{bmatrix}$. The local activation from Rank 0 ($h_0 = \text{GeLU}(x A_0)$) has the exact inner dimension required to multiply directly against local row shard $B_0$. Rank 0 produces partial product $h_0 B_0$, and Rank 1 produces $h_1 B_1$.
3. Single all-reduce reduction: Notice that $h B = h_0 B_0 + h_1 B_1$. Slicing $B$ by rows turns the final output into a sum of partial products. One sum all-reduce collective combines these partial outputs across ranks, and the bias is added once to restore the exact logical output.

The multi-head attention block mirrors this exact column-then-row symmetry. The query ($W_Q$), key ($W_K$), and value ($W_V$) projections shard column-wise, which maps cleanly onto independent attention heads. Each rank computes scaled dot-product attention for its local subset of heads without talking to its neighbors. The attention output projection ($W_O$) then shards row-wise, taking local head outputs and producing partial products. A single sum all-reduce restores the full multi-head output.

For this classic dense forward pattern, count two sum all-reduces per transformer layer. This excludes embeddings/output handling and assumes TP>1. Sequence/context parallelism, expert routing, fusion, and other runtime layouts can change the collectives; it isn't a universal count for the hybrid Qwen model.

Verify the split with actual tensors

Let's test this sharding math directly in PyTorch. We'll take an input with two features, expand it to four hidden features through matrix $A$, apply GeLU, and project back to two features through matrix $B$. GPU 0 owns the first two hidden features; GPU 1 owns the last two. Their local outputs must be summed, not averaged or concatenated:

With two ranks, each local weight slice $A_i$ has shape [2, 2], each hidden activation slice is [1, 2], and each local projection slice $B_i$ has shape [2, 2]. Both ranks compute their local contribution independently. An all-reduce sum combines the slices so every participating rank holds the identical final activation vector.^{[3]Reference 3NVIDIA Collective Communication Library User Guidehttps://docs.nvidia.com/deeplearning/nccl/user-guide/index.html}

KV head counts and sharding constraints

For clean head sharding, your tensor parallel size must divide the attention head counts. In standard multi-head attention (MHA), that means $\text{num\_attention\_heads} \pmod{\text{TP}} = 0$.

Under Grouped-Query Attention (GQA), multiple query heads share a single key-value head. Slicing KV heads across ranks without replicating them imposes a much stricter condition:

$n_{\text{kv\_heads}} \pmod{\text{TP}} = 0$

For example, Llama 3 70B has 64 query heads and 8 KV heads. On TP=8, each GPU owns 8 query heads and exactly 1 KV head ($8 / 8 = 1$). On TP=4, each GPU owns 16 query heads and 2 KV heads ($8 / 4 = 2$). Both divide evenly.

What happens when TP exceeds the KV-head count? Qwen's ten full-attention layers have 16 query heads, 2 KV heads, and head dimension 256. Its other thirty layers use Gated DeltaNet state; a full-history KV formula doesn't describe all forty layers.^{[5]Reference 5Qwen3.6-35B-A3B Configurationhttps://huggingface.co/Qwen/Qwen3.6-35B-A3B/blob/main/config.json}

Pure nonduplicating head sharding fails that divisibility test at TP=4 or 8. Current vLLM's QKV linear layer instead supports KV replication when TP is a multiple of the smaller KV-head count.^{[6]Reference 6vLLM QKV Parallel Linear Layerhttps://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/linear.py} At TP=8 with two KV heads, four ranks carry each head's projection. Verify the model/backend's cache layout too.

For plain replicated-head full-history caching, TP=2 to TP=8 still leaves one KV head per rank. That history term stops shrinking; the freed weight bytes can nevertheless enlarge the available pool. Other hybrid-layer state has separate rules. Decode context parallelism can shard history within those replicated groups when supported, so zero further KV relief is not a universal limit.

Count communication, then measure its exposed cost

Under the classic 80-layer dense pattern above, one batch decode iteration has 160 in-layer all-reduces. Embedding and sampling collectives can add to the count.

Across 128 decode iterations, that assumed pattern exceeds 20,000 in-layer all-reduces. A 128-token completion normally needs only 127 decode iterations after prefill produces its first token:

Dependencies between blocks can expose communication on the critical path. A sum of collective durations is not automatically the exposed delay: overlap, fusion, algorithms, and launch behavior matter. First examine a deliberately serial startup model:

Compare two assumed per-call startup costs:

The 5 and 20 microseconds are authored assumptions, not NVLink/PCIe/network measurements. In a fully serial comparison, adding 3.2 ms to a 6 ms baseline raises duration by 53.3%. It doesn't prove pure idle time or a particular fabric's latency. Measure the operation at the relevant rank count, payload, dtype, and topology; NCCL tests distinguish small-message latency, algorithm bandwidth, and corrected bus bandwidth.^{[7]Reference 7NCCL Tests Performance Reportinghttps://github.com/NVIDIA/nccl-tests/blob/master/doc/PERFORMANCE.md}

Now check payload transfer time for a batch of 8 requests in BF16:

The 0.05 ms value uses an assumed 300 GB/s sustained per-rank send rate and a ring traffic model. It isn't an observed NVLink duration. Comparing it with the separate startup assumptions suggests what to profile; summing the two models still doesn't predict total exposed latency. NCCL may choose a tree or another algorithm.

TP can cross nodes, and a fast scale-up domain needn't stop at eight GPUs: NVIDIA GB200 NVL72 connects 72 GPUs by NVLink.^{[8]Reference 8GB200 NVL72https://www.nvidia.com/en-us/data-center/gb200-nvl72/} Domain size doesn't guarantee that a particular TP degree divides a model or improves its performance.

Pipeline parallelism

Pipeline parallelism partitions the model along its depth. GPU 0 hosts layers 0 to 39, while GPU 1 hosts layers 40 to 79. Activations travel forward across stage boundaries: GPU 0 computes its assigned layers and forwards the intermediate hidden state to GPU 1.

Classic PP sends hidden activation tensors $[B,s,h]$ at stage boundaries, plus any required metadata. This reduces communication frequency compared with the earlier dense TP pattern. TP reductions also communicate activations, rather than whole weight matrices. Fewer boundary transfers can make PP useful over slower links; measure their size, scheduling, and imbalance costs.

For ordinary autoregressive decode, the next token depends on the previous token's final logits and sampling. One request must cross all stages. Other independent microbatches can occupy earlier stages; a shallow queue leaves those slots idle. Speculative/multi-token methods need a separate schedule.

Microbatch scheduling and pipeline bubbles

When multiple independent requests arrive, the serving engine groups them into microbatches and interleaves their execution across stages. Stage $s$ processes microbatch $m$ during time slot $s + m$.

Assume equal-duration stages, independent equal-size microbatches, one forward traversal, and no transfer/queue cost. Then $p$ stages and $m$ microbatches take $m+p-1$ slots. Occupied slots total $pm$ out of $p(m+p-1)$. This is a finite fill/drain fixture, not a complete steady-state autoregressive serving schedule.

The pipeline bubble fraction $F_{\text{bubble}}$ is the proportion of idle hardware time during the fill (warm-up) and drain (cool-down) phases:

$F_{\text{bubble}} = \frac{p - 1}{m + p - 1}$

Relative to the ideal throughput interval $m t_{\text{stage}}$, fill/drain overhead has this dimensionless ratio:

$R_{\text{bubble}} = \frac{p-1}{m}$

The extra elapsed time is $(p-1)t_{\text{stage}}$. Neither ratio is a multiplier on the original full-model single-request latency.

Let's compute the occupancy schedule explicitly:

Counts must be discrete. A fractional stage count must not quietly produce a plausible utilization:

When pipeline parallelism is justified

For the three-stage, one-microbatch fixture, two-thirds of stage slots are idle. Compare its critical path with a full-stack baseline before turning that occupancy into a latency claim:

These are chosen times, not a guarantee that partitioned kernels preserve the baseline compute time.

Pipeline parallelism in inference is justified primarily when:

1. Depth partitioning solves a placement constraint. Use PP when a validated layout needs more capacity, uneven layer splits, or fewer frequent collectives. It can be useful inside a node without NVLink, as vLLM's guide notes.^{[2]Reference 2Distributed Inference and Serving.https://docs.vllm.ai/en/latest/serving/parallelism_scaling.html}
2. The topology favors a hybrid. TP groups can stay within a fast domain while PP crosses slower links. TP=8 per node is one candidate, not a universal optimum or domain limit.
3. The workload and scheduler sustain useful overlap. More independent microbatches can amortize bubbles; no fixed m≥8 threshold keeps arbitrary stages continuously busy.

The names Llama 3.1 405B and DeepSeek-V3 671B don't by themselves prove a multi-node requirement.^{[9]Reference 9The Llama 3 Herd of Models.https://arxiv.org/abs/2407.21783[10]Reference 10DeepSeek-V3 Technical Report.https://arxiv.org/abs/2412.19437} Check the raw INT4 arithmetic:

Context parallelism for long-sequence serving

Long inputs can make attention work or retained history a constraint alongside weights. Required storage depends on the architecture; sequence sharding doesn't extend a checkpoint's trained/supported context.

In Megatron Core, sequence parallelism shards the sequence dimension across LayerNorm and Dropout operations alongside tensor parallelism to save activation memory during training.^{[11]Reference 11Parallelism Strategies Guide.https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/parallelism-guide.html} It doesn't distribute prompt attention or KV cache state across independent serving nodes.

Context parallelism (CP) partitions the input sequence across multiple GPUs, allowing chips to share the KV cache and attention computation of a single massive prompt.

For a hypothetical 1M-token full-attention GQA history (80 layers, 8 KV heads, head dimension 128), count raw FP16 payload. This isn't a supported Llama context or Qwen's hybrid state:

At 305.2 GiB, history alone exceeds one small-device budget. Four even shards still need 76.3 GiB each, before weights/workspace. The previous 80,000,000,000-byte budget is only 74.5 GiB, so four shards don't make that fixture fit. Inspect both prefill and decode mechanisms.

Ring Attention

Ring Attention shards the sequence dimension into $C$ blocks across a logical ring of GPUs.^{[12]Reference 12Ring Attention with Blockwise Transformers for Near-Infinite Context.https://arxiv.org/abs/2310.01889} Each GPU holds its local Query block throughout the layer.

During each step of the ring:

1. GPU $i$ computes blockwise self-attention using its local Query block and its current Key-Value block.
2. Simultaneously, GPU $i$ sends its current Key-Value block to GPU $i+1$ and receives a new Key-Value block from GPU $i-1$ over peer-to-peer links.
3. Online softmax merges statistics and weighted outputs across visited blocks. For the simple noncausal scheme, each query block visits C KV blocks, requiring C−1 transfers after its local block; causal masks can skip work.

Transfers can overlap block attention. Fully hiding them requires enough compute and suitable buffering/topology; the paper's overlap condition is not a guarantee for short decode queries.^{[12]Reference 12Ring Attention with Blockwise Transformers for Near-Infinite Context.https://arxiv.org/abs/2310.01889}

DeepSpeed Ulysses

DeepSpeed Ulysses was introduced for long-sequence training using all-to-all collectives.^{[13]Reference 13DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelshttps://github.com/deepspeedai/DeepSpeed/blob/master/blogs/deepspeed-ulysses/README.md} Its layout mechanism helps explain prefill attention, but isn't itself a promise that a serving runtime retains a sequence-sharded decode cache.

Starting with sequence-partitioned inputs:

1. Each rank first projects its local tokens into Q/K/V. An all-to-all then redistributes those projected tensors from sequence slices (all heads, S/C tokens) to head slices (assigned heads, full sequence).
2. Each GPU computes standard local FlashAttention across the full sequence for its assigned subset of heads.
3. A second All-to-All collective transposes the output back to sequence-parallel before the linear output projection.

The simple nonduplicating layout needs compatible head divisibility, including the smaller KV-head count under GQA. Supported replication or hybrid layouts change that limit. Ring sequence partitioning doesn't require splitting heads by C, but brings ring transfers, buffer requirements, and masking/load-balance concerns.

Decode context sharding addresses another axis

Current vLLM documentation separates prefill CP from decode context parallelism (DCP).^{[14]Reference 14vLLM Context Parallel Deploymenthttps://github.com/vllm-project/vllm/blob/main/docs/serving/context_parallel_deployment.md} DCP shards retained history inside existing TP groups to reduce KV duplication, without adding another factor to the GPU count. The guide describes GQA/MLA support; validate the particular model and attention backend, especially hybrid state.

Try the ideal layout for two KV heads, TP=8, DCP=4, and 32 history positions. Ignore page rounding and metadata:

These ranks must combine attention statistics and weighted contributions. Averaging independently normalized shard outputs is generally wrong. The CPU exercise checks position ownership, not attention numerics, backend support, or a measured speedup.

Expert parallelism for MoE models

Mixture-of-experts architectures present another dimension of model partitioning: where does each expert live? Instead of routing every token through identical dense feed-forward blocks, a router directs each token to top-$k$ experts.^{[15]Reference 15Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.https://arxiv.org/abs/1701.06538}

Expert parallelism (EP) places different experts on different GPUs, scaling total parameter capacity without replicating every expert onto every chip.

Tensor parallelism and expert parallelism slice parameters along different axes:

• TP slices expert matrices within a tensor-parallel group.
• EP places different experts on different ranks. A runtime can combine expert placement with tensor sharding or keep redundant expert copies.

Routing requires dispatching token activations to expert owners and combining weighted results back at their origins. This is logically an all-to-all exchange; the transport may use specialized dispatch/combine kernels or all-gather/reduce-scatter rather than two literal NCCL all-to-all calls.^{[16]Reference 16Expert Parallel Deploymenthttps://docs.vllm.ai/en/latest/serving/expert_parallel_deployment/}

Router decisions depend on hidden representations, not an assumed keyword rule. Skewed assignments can create stragglers; measure expert work and transfers rather than infer latency from counts alone. Checked September 22, 2026: vLLM's --enable-expert-parallel forms an expert group across TP×DP ranks, while attention uses TP inside each DP group. Its load-balancing options can add redundant expert copies.^{[16]Reference 16Expert Parallel Deploymenthttps://docs.vllm.ai/en/latest/serving/expert_parallel_deployment/}

Here's an explicit simulation showing how token assignments distribute across two devices:

Multi-GPU serving Pareto trade-offs

Let's synthesize these parallel dimensions into an engineering decision framework. On a cluster of 8x 80 GB GPUs, how should you partition your hardware to serve a 70 GB model under production traffic?

Three distinct operational topologies compete:

1. Pure Tensor Parallelism (TP=8)

• Profile: A single replica spanning all 8 GPUs.
• Potential benefit: More ranks share eligible matrix work and weight storage; measure whether that reduces prompt or decode duration.
• Costs: Collectives, smaller local kernels, replicated state, and a shared replica's admission policy. Neither fastest latency nor maximum communication cost follows from TP=8 alone; the classic 160 count belongs to the earlier dense example.

2. Sharded Data-Parallel Replicas (TP=2, DP=4)

• Profile: Four independent 2-GPU replicas, each running TP=2.
• Potential benefit: Four separately resident pairs can process independent requests if each validated layout fits. Fourfold scaling relative to one such pair requires enough demand, good balance, and no shared bottleneck; it isn't fourfold versus one eight-GPU TP replica.
• Costs: Replicated weight storage, separate caches, load balancing, and skew. Single-request latency can be higher or lower than TP=8. These pairs are independent only if their expert layers are also independent; enabling cross-group EP couples them.

3. Prefill-Decode Disaggregation

• Profile: Physically separated worker pools for the prefill and decode phases.^{[17]Reference 17DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving.https://arxiv.org/abs/2401.09670[18]Reference 18Splitwise: Efficient Generative LLM Inference Using Phase Splitting.https://arxiv.org/abs/2311.18677}
• Potential benefit: Tune each phase's placement and scheduling separately and reduce mixed-phase interference. Suitable parallelism depends on the measured model, prompt/history mix, backend, and hardware.
• Costs: Both queues, KV-state movement/re-layout, compatibility, coordination, and additional workers. Transfers may overlap some work but still need critical-path accounting. No TP degree or RDMA label guarantees the benefit.

Let's verify the KV cache headroom for a TP=2 deployment assuming each concurrent request requires 0.75 GB of KV cache across the pair:

What to measure

Compare configurations on the same hardware accounting, model/precision, arrival mix, prompt/output distribution, quality checks, and measurement boundary. Record per-rank allocations and achieved communication behavior alongside user-visible delivery.

Structure your benchmarking decisions through a three-gate evaluation process:

1. Gate 1: Physical Fit: Does the checkpoint, runtime reserve, and required KV cache headroom fit in physical VRAM without triggering out-of-memory crashes?
2. Gate 2: Latency SLOs: Does the candidate satisfy both p95 TTFT (e.g. $\le 400$ ms) and p95 TPOT (e.g. $\le 55$ ms) under target concurrency?
3. Gate 3: Cost: Among eligible candidates, compare sustained useful output per billed interval. Raw TPS alone doesn't establish request goodput or cost efficiency when GPU counts differ.

First exercise fit and latency filtering with assigned values. This sketch ranks remaining candidates by raw TPS; it has no prices or deployment action:

In this synthetic fixture, TP=4's assigned 700 TPS doesn't override its failing latencies. TP=2 is the only eligible candidate under these gates. No trace identifies the cause, and eligibility here isn't production validation. A p95 of per-request average TPOT also doesn't bound each token gap or prove that 95% of requests meet both limits jointly.

Now suppose both candidates pass the other gates. Predict which wins on GPU-only cost when the larger group costs twice as much per hour:

The prices and rates are arithmetic assumptions, not provider quotes or a benchmark. Whole-service cost also includes idle intervals, other workers, retries, and CPU/network/storage charges.

When multi-GPU serving breaks down

These symptoms guide investigation without identifying a cause by themselves:

• OOM despite on-paper fit: Inspect each rank's checkpoint/replication, KV layout, workspace, graph capture, other allocations, and reported free bytes. A group-total sum can hide a local deficit.
• Latency worsens with more ranks: Profile collectives, local kernel efficiency, scheduling, and topology. Test whether exposed communication exceeds saved work; a PCIe or cross-node label alone doesn't prove dominance.
• KV history stops shrinking with TP: Check head replication and hybrid-layer state. Weight savings can still enlarge the pool. Test supported DCP if history duplication is the measured problem.
• Concurrent traffic creates token gaps: Separate queued admission, mixed-step work, transfers, and delivery buffering in a trace. Chunking or phase separation may help; measure their added work and queues too.

Evaluation rubric

• Reconstruct the dense MLP result from column-parallel and row-parallel shards, adding output bias once.
• Count collectives under the classic dense layout and measure which communication is exposed.
• Derive the pipeline bubble fraction $F_{\text{bubble}} = \frac{p-1}{m+p-1}$ from first principles.
• Distinguish Ring Attention from DeepSpeed Ulysses sequence sharding for long-context workloads.
• Filter by per-rank fit and the actual latency contract, then compare sustained useful rates and billed costs.

Follow-up questions