Mamba & State Space Models

The previous chapter on Mixture of Experts cut feed-forward work by routing each token through a few experts. Its sequence mixer was still self-attention, though. During decode, every prompt token leaves a key and value behind, and every generated token adds another entry to the key-value (KV) cache.

A service receives a 32K-token API-documentation prompt. A Transformer can revisit an earlier identifier exactly, but it must retain keys and values for all 32K tokens. A state-space model (SSM) takes the opposite bet: carry a fixed-size running summary forward. Predict the tradeoff before naming a model: memory becomes easier to budget, but exact lookup can get harder because a summary replaces addressable history.^{[1]Reference 1Mamba: Linear-Time Sequence Modeling with Selective State Spaceshttps://arxiv.org/abs/2312.00752}

Mamba makes that bet selective. It updates one recurrent state for each new token instead of caching every token. Mamba-2^{[2]Reference 2Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualityhttps://arxiv.org/abs/2405.21060} and Mamba-3^{[3]Reference 3Mamba-3: Improved Sequence Modeling using State Space Principleshttps://arxiv.org/abs/2603.15569} refine the update and its hardware path; hybrids such as Jamba^{[4]Reference 4Jamba: A Hybrid Transformer-Mamba Language Modelhttps://arxiv.org/abs/2403.19887} and Nemotron-H^{[5]Reference 5Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelshttps://arxiv.org/abs/2504.03624} add occasional attention to recover explicit lookup. The practical question running through this chapter is simple: when does bounded memory beat exact history, and what evidence would convince you?

What is the core serving tradeoff between attention and Mamba-style SSMs?

Answer

Attention keeps token-level access to prior context through a KV cache that grows with sequence length. Mamba-style SSMs keep a fixed-size recurrent state, which removes context-length-dependent decode-state growth but compresses history; evaluate exact retrieval and throughput on the target workload.

The visual gives us the serving contract, not yet the mechanism. To see what a fixed state can and can't carry, start with a recurrence small enough to calculate by hand.

A discrete recurrence you can compute by hand

The fixed state isn't a hidden database. It's an activation updated once per token. At each step, old state is carried forward, the current input writes into it, and the output reads the result:

$h_t = \bar{A}\, h_{t-1} + \bar{B}\, x_t$ $y_t = \bar{C}\, h_t + \bar{D}\, x_t$

Read the first term as memory and the second as a write. The readout turns the updated state into the layer output. During decode, state size is fixed by the model rather than by preceding context length.

Walkthrough with numbers

A one-dimensional state can track a running activation as four tokens arrive. Set:

• $\bar{A} = 0.9$ (the state keeps 90% of its previous value)
• $\bar{B} = 0.2$ (new input contributes 20%)
• $\bar{C} = 1.0$ (read the full state)
• $\bar{D} = 0$ (no skip)

Now process four token activations $x = [3, 1, 4, 2]$:

Before calculating, predict the direction. Because $\bar{A}=0.9$, old state should fade slowly. Because $\bar{B}=0.2$, input 4 should pull the state up more than input 1. The result should smooth the sequence, not copy it.

The state smooths the input sequence. Input 4 pulls it up, but the earlier 0.74 still contributes through the 0.9 carryover.

That compression is useful and lossy at the same time. The layer stores one summary instead of four addressable token values, so its memory stays bounded while exact recovery becomes a learned capability rather than a direct lookup.

If you change A-bar to 0.5 and B-bar to 0.5, what is the final state after processing [3, 1, 4, 2] from zero state?

Answer

Step 1 is 1.5. Step 2 is 0.5 * 1.5 + 0.5 * 1 = 1.25. Step 3 is 0.5 * 1.25 + 0.5 * 4 = 2.625. Step 4 is 0.5 * 2.625 + 0.5 * 2 = 2.3125.

Now change only $\bar{A}$. If it were 0.1 instead of 0.9, the final state would be about $0.483$ with the same inputs and $\bar{B}=0.2$, rather than $1.719$.

Earlier events now contribute far less. Whether that helps or hurts depends on what the task needs to retain, which is why state dynamics matter as much as state size.

In the discrete recurrence, what role does the hidden state play?

Answer

The hidden state is the compressed summary of past inputs. It's updated from the previous state and the current token, then read out to produce the layer output.

Where $\bar{A}$ and $\bar{B}$ come from

The hand calculation hid one design choice: where do $\bar{A}$ and $\bar{B}$ come from? State Space Models borrow a control-theory view of dynamical systems, where a continuous signal changes over time. The continuous variables are:

• $x(t) \in \mathbb{R}$: the input signal
• $h(t) \in \mathbb{R}^s$: the hidden state, a vector of dimension $s$ that acts as a compressed summary of all past inputs
• $y(t) \in \mathbb{R}$: the output
• $\mathbf{A} \in \mathbb{R}^{s \times s}$: the state transition matrix, shaping how much past information to retain
• $\mathbf{B} \in \mathbb{R}^{s \times 1}$: the input matrix, shaping how much new information to absorb
• $\mathbf{C} \in \mathbb{R}^{1 \times s}$: the output matrix, which reads out the relevant part of the state

The scalar input and output keep notation readable. Real neural layers run many channels in parallel, but each channel follows the same kind of state update.

From these, the continuous dynamics are:

$h'(t) = \mathbf{A}h(t) + \mathbf{B}x(t)$ $y(t) = \mathbf{C}h(t) + \mathbf{D}x(t)$

The derivative equation says how state changes between observations. $\mathbf{A}$ controls carryover, while $\mathbf{B}$ determines how the input enters. The output $y(t)$ reads the state through $\mathbf{C}$, and $\mathbf{D}$ provides a direct input-to-output path.

This is the continuous version of the table above. We still need one step that turns smooth time into token steps.

Discretization

Tokens arrive as samples, but the equations above describe smooth change. To connect them, sample every $\Delta$ time units and assume the current input stays constant across each interval. This assumption is zero-order hold (ZOH). It converts the continuous matrices into the discrete $\bar{\mathbf{A}}$ and $\bar{\mathbf{B}}$ you used by hand:^{[6]Reference 6Efficiently Modeling Long Sequences with Structured State Spaceshttps://arxiv.org/abs/2111.00396}

$$\begin{aligned} \bar{\mathbf{A}} &= e^{\Delta \mathbf{A}} \\ \bar{\mathbf{B}} &= (\Delta \mathbf{A})^{-1} \left(e^{\Delta \mathbf{A}} - \mathbf{I}\right) \cdot \Delta \mathbf{B} \end{aligned}$$

where $e^{\Delta \mathbf{A}}$ is the matrix exponential. These discrete matrices encode the same dynamics as the continuous equations, but in a form suitable for step-by-step updates:

$h_t = \bar{\mathbf{A}} h_{t-1} + \bar{\mathbf{B}} x_t$ $y_t = \mathbf{C} h_t + \mathbf{D} x_t$

After discretization, each token step does exactly what the hand calculation did: carry state through $\bar{\mathbf{A}}$, absorb input through $\bar{\mathbf{B}}$, and read out through $\mathbf{C}$. The $\mathbf{D}$ term still skips input directly to output. Continuous notation explains where the coefficients come from; it isn't a second model.

With that bridge in place, we can ask how to choose dynamics that retain useful long-range signals and still run efficiently.

S4: structured state spaces

The Structured State Space for Sequence Modeling (S4)^{[6]Reference 6Efficiently Modeling Long Sequences with Structured State Spaceshttps://arxiv.org/abs/2111.00396} makes that bounded-state idea useful over long horizons. Its structured dynamics retain signals and expose two ways to execute the same recurrence. The step-by-step form you calculated is the generation view; training can use a parallel view.

HiPPO initialization

In S4, the continuous-time $\mathbf{A}$ matrix starts from the HiPPO (High-order Polynomial Projection Operators) framework, which compresses input history into coefficients of orthogonal polynomials.^{[7]Reference 7HiPPO: Recurrent Memory with Optimal Polynomial Projections.https://arxiv.org/abs/2008.07669[6]Reference 6Efficiently Modeling Long Sequences with Structured State Spaceshttps://arxiv.org/abs/2111.00396}

HiPPO turns history into coefficients over orthogonal-polynomial bases, giving state different coordinates for different temporal patterns. Without useful structure, a long recurrence can wash out old signals. S4 starts from those dynamics instead of asking an arbitrary recurrent matrix to discover long-horizon retention from scratch.

Dual computation modes

S4 can be computed in two equivalent ways. At training time, the whole sequence is known; at generation time, only the next token is available. For notational simplicity, the costs below are written for one channel; a full layer runs many channels in parallel.

Recurrent mode (generation)

Efficient for generation: $h_t = \bar{\mathbf{A}} h_{t-1} + \bar{\mathbf{B}} x_t, \quad y_t = \mathbf{C} h_t + \mathbf{D} x_t$

Time: $O(L \cdot s)$ in Big-O notation, Memory: $O(s)$ per step.

Convolutional mode (training)

Efficient for training: $y = x * \bar{K}, \quad \text{where } \bar{K}_t = \mathbf{C} \bar{\mathbf{A}}^t \bar{\mathbf{B}}$

In plain terms

The recurrence above can be unrolled into a single convolution with a pre-computed kernel $\bar{K}$. Because the full sequence is available during training, an FFT (Fast Fourier Transform) can apply that kernel in parallel. The sequence-mixing work then runs in $O(L \log L)$ time instead of a token-by-token loop. Same math, different execution strategy.

Why does S4 need two computation views?

Answer

Recurrent mode is natural for generation because it updates one fixed-size state per new token. Convolutional mode is useful for training because it processes a full known sequence in parallel instead of looping token by token.

S4 solves long-range dynamics and execution choice. It still has one policy for every token, which creates the next failure mode.

The LTI problem

S4 has one limitation that matters for selective memory: every token uses the same state-transition rule. A filler token such as the and a decisive token such as max_retries can carry very different information, but the update can't change how strongly it writes either token based on content.

That property is called Linear Time-Invariant (LTI). Matrices $\mathbf{A}$, $\mathbf{B}$, and $\mathbf{C}$ stay fixed regardless of the input. The recurrence can still decay with position, but it can't change its write or read policy when a particular token matters.^{[1]Reference 1Mamba: Linear-Time Sequence Modeling with Selective State Spaceshttps://arxiv.org/abs/2312.00752}

Attention takes a different route: query-key interactions provide content-dependent access. Fixed LTI SSMs therefore struggle on selective-copy tests, while Mamba makes part of the update input-dependent.

Why is a fixed LTI SSM weaker than attention for content-dependent retrieval?

Answer

Its update matrices are the same for every token, so it can't choose a different read/write behavior when a special word appears. Attention can route based on query-key content, which makes exact copying and lookup easier.

Mamba: selective state spaces

Mamba^{[1]Reference 1Mamba: Linear-Time Sequence Modeling with Selective State Spaceshttps://arxiv.org/abs/2312.00752} solves the LTI limitation with one core idea: make part of the SSM input-dependent.

The selective mechanism

A documentation-QA decoder moving through a long API spec makes the limitation concrete. A filler token such as the may need a small state update, while a rare identifier such as max_retries may need a stronger write. Predict what a useful memory system should do: preserve the identifier, then spend less state capacity on filler.

Mamba makes that prediction part of the update. Instead of fixed $\mathbf{B}$, $\mathbf{C}$, and $\Delta$, it computes them as functions of the input:

$\mathbf{B}_t = \text{Linear}_B(x_t), \quad \mathbf{C}_t = \text{Linear}_C(x_t), \quad \Delta_t = \text{softplus}(\text{Linear}_\Delta(x_t))$

Unlike S4, which uses the same $\mathbf{B}$, $\mathbf{C}$, and $\Delta$ for every token, Mamba computes these values fresh from each token's content. The learned transition $\mathbf{A}$ and skip term $\mathbf{D}$ remain input-independent in Mamba-1. Selectivity enters through $\mathbf{B}_t$, $\mathbf{C}_t$, and $\Delta_t$.^{[1]Reference 1Mamba: Linear-Time Sequence Modeling with Selective State Spaceshttps://arxiv.org/abs/2312.00752}

The step size $\Delta_t$ controls how much of that change is applied. With negative $\mathbf{A}$, it determines how quickly old state decays and how strongly the current input is integrated:

• Large $\Delta_t$: Decay older state faster and absorb more of the current input
• Small $\Delta_t$: Keep more of the previous state and make a smaller update

If $\Delta_t$ is very large, does the model remember more or less of the previous state?

Answer

Less. Because $\mathbf{A}$ is constrained negative, a large $\Delta_t$ makes $\bar{\mathbf{A}} = \exp(\Delta_t \mathbf{A})$ smaller. The old state receives less weight, and the current token has a stronger update.

This gives Mamba a learned, content-dependent recurrent update while retaining linear-length recurrent computation. It still doesn't give every output explicit access to every prior token as attention does.

Avoid a common inference: no KV cache doesn't mean Mamba remembers everything. A large $\Delta_t$ can make old state decay quickly, and a poorly learned gate can wash out max_retries when the arrives. Mamba learns how much to forget per token, not whether to open a full history for lookup.

What does "selective" mean in Mamba?

Answer

The model computes B_t, C_t, and Delta_t from the current token, so different tokens can write to state, read from state, and forget history differently. It's content-aware recurrence, not fixed recurrence.

Architecture

Selective recurrence is the core, not the entire production block. A Mamba block replaces attention with projection, causal convolution, a selective scan over input-dependent state updates, and gating.^{[1]Reference 1Mamba: Linear-Time Sequence Modeling with Selective State Spaceshttps://arxiv.org/abs/2312.00752} It uses SiLU (Sigmoid Linear Unit) activations and softplus (a smooth approximation of ReLU defined as log(1 + e^x), which is always positive and differentiable everywhere) to keep step sizes positive:

The example below focuses on the selective recurrence rather than the entire fused block. It leaves out causal convolution and output gate on purpose, so the state update stays easy to inspect.

Continue the four-token sequence with a negative scalar $\mathbf{A}$. Here $B_t$ and $\Delta_t$ are simple functions of the current input, so a larger token writes harder and forgets faster. Before reading the output, expect input 4 to create the largest step and the smallest retained fraction.

Why is a Python loop over tokens useful for learning but wrong for production training?

Answer

It exposes the recurrence clearly, but a Python loop over sequence positions destroys GPU parallelism and spills work through slow paths. Production Mamba uses fused scan kernels that keep state in fast memory and parallelize the sequence update.

The recurrence code explains behavior, not serving capacity. We can sanity-check the memory claim separately by counting stored state elements in a small illustrative fixture.

What goes wrong when you build this from scratch

The loop above has the right math, but it hides three implementation boundaries. Each symptom below points to a different one: numerical stability, selectivity, or GPU execution.

The hardware-aware parallel scan

Mamba's hardware-aware scan is why this recurrence can train at all. On paper, $h_t = \bar{\mathbf{A}}_t h_{t-1} + \bar{\mathbf{B}}_t x_t$ looks strictly sequential. If $h_{t-1}$ must be known before computing $h_t$, a Python loop over tokens would stall training on GPUs next to heavily parallelized attention.

The bottleneck isn't only arithmetic. It's memory traffic too. If every token update spills intermediate state back to high-bandwidth memory (HBM), the recurrence becomes bandwidth-bound. Mamba fuses discretization and state updates so recurrent state stays in fast on-chip static random-access memory (SRAM) as much as possible, then uses an associative scan to recover parallelism across positions.^{[1]Reference 1Mamba: Linear-Time Sequence Modeling with Selective State Spaceshttps://arxiv.org/abs/2312.00752}

The key reformulation treats each update as an affine transform, $h \mapsto a h + b$. Two such transforms can be composed without evaluating every intermediate state, and composition is associative. Like a parallel prefix sum, that lets chunks be combined in a tree instead of waiting on one long chain.

Before the visual, predict the split: training can compose many updates in parallel, while serving still advances one state per new token.

The scan has three phases:

1. First, it computes all the input-dependent parameters and pairs of $(\bar{\mathbf{A}}_t, \bar{\mathbf{B}}_t x_t)$ in parallel across the entire sequence.
2. Next, it applies the parallel prefix sum using a highly optimized associative operator to combine these terms.
3. Finally, it reads off all the hidden state values $h_t$ simultaneously.

What makes the selective scan "hardware-aware"?

Answer

It fuses parameter computation, discretization, and recurrence updates so intermediate state stays in fast on-chip memory as much as possible. The associative scan recovers parallelism that a naive recurrent loop would lose.

By reformulating the recurrence for an associative scan, this hardware-aware approach achieves $O(L)$ total work with much smaller critical-path depth than a naive recurrent loop. That makes Mamba trainable on GPUs, but it still depends on custom scan kernels and doesn't map as cleanly to Tensor Core matmuls as transformers do.

Transformers can reuse FlashAttention and a mature PyTorch stack. SSMs need their own fused kernels, serving paths, and state management, so linear asymptotics alone don't settle adoption. That remaining kernel gap is what Mamba-2 addresses.

Mamba-2: state space duality

Mamba showed that selective SSMs could work well, but its parallel scan still followed a custom hardware path. That path underused the matrix-multiplication engines (Tensor Cores) that make GPUs fast.

Mamba-2^{[2]Reference 2Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualityhttps://arxiv.org/abs/2405.21060} attacks that gap through Structured State Space Duality (SSD). The core observation is that selective SSMs and variants of attention can both be written as operations over structured semiseparable matrices. If chunks can do more work as matrix operations, the GPU has more useful parallel work. Mamba-2 therefore uses a chunked algorithm where:

• intra-chunk outputs are computed in parallel
• chunk final states are computed with batched matrix multiplications
• only the much shorter inter-chunk state passing remains a scan
• the initial-state contribution for each chunk is added back with more batched matrix multiplications

In SSD, more work becomes matrix multiplication, while a reduced inter-chunk recurrent core remains. The model keeps the bounded recurrent state, but changes the route used to compute it.

Why does Mamba-2 matter if Mamba-1 was already linear time?

Answer

Linear asymptotic complexity isn't the same as fast GPU execution. Mamba-2 moves more of the work onto batched matrix multiplications and Tensor Cores, reducing reliance on custom scan-heavy kernels.

Mamba-1 vs. Mamba-2 comparison

In its dedicated implementation, the Mamba-2 paper reports 2-8x faster core-layer execution than the optimized selective-scan implementation of Mamba-1 while retaining competitive language-model results in its experiments.^{[2]Reference 2Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualityhttps://arxiv.org/abs/2405.21060} This is a measured core-layer comparison, not a guarantee for every end-to-end server. Check hardware, kernels, batch shape, sequence lengths, and quality on your own stack.

Beyond Mamba-2 toward Mamba-3

SSD changes how the recurrence runs. Mamba-3^{[3]Reference 3Mamba-3: Improved Sequence Modeling using State Space Principleshttps://arxiv.org/abs/2603.15569} asks a more specific serving question: once state must be read every token, how can the update carry more useful arithmetic without moving more state bytes? It takes an explicitly inference-first view and changes the recurrent update along three axes:

• Exponential-trapezoidal discretization, which generalizes the exponential-Euler rule used in Mamba-1 and Mamba-2. It can be second-order accurate when the trapezoidal parameter stays near $1/2$, though the paper's ablations prefer not enforcing that constraint.
• A complex-valued state update, implemented as a data-dependent rotary embedding, evaluated on state-tracking tasks such as parity and modular arithmetic that real-valued linear recurrences miss.
• A multi-input, multi-output (MIMO) formulation that raises arithmetic intensity during memory-bound decode by doing more work in the state update without growing the state.

At 1.5B scale, Mamba-3 (SISO) improved average downstream accuracy by 0.6 points over Gated DeltaNet. The MIMO variant improved by 1.8 points over GDN and 1.9 points over Mamba-2. MIMO with state size 64 matched Mamba-2 perplexity at state size 128.

The paper also reports up to 4× more decode FLOPs for MIMO at similar wall-clock decode latency. That is a hardware-utilization result, not a 4× tokens/s claim.^{[3]Reference 3Mamba-3: Improved Sequence Modeling using State Space Principleshttps://arxiv.org/abs/2603.15569} The quality comparisons are at the 1.5B scale, while the latency analysis uses released fused kernels and fixed state-size settings. Treat both as experiment-scoped evidence, then re-measure hardware, precision, batch, context, and correctness in the target serving stack.

What kind of improvement does Mamba-3 target compared with earlier Mamba work?

Answer

It targets inference-first quality and state efficiency: stronger recurrence, complex-valued state tracking, and MIMO updates that the paper evaluates for improved quality or reduced state size at matched decode latency.

Complexity comparison

Separate training from decode to see the serving payoff. The chart uses the same 32-layer, $d=4096$, BF16 fixture as the byte-count script. The table isolates sequence mixing in one layer, not MLP or projection costs.

The generation advantage is fixed state size. SSMs maintain roughly $s \cdot d$ elements per layer regardless of context length. A transformer serving 128K context must keep on the order of $128\text{K} \times d$ KV cache activations per layer. For a long documentation prompt, an SSM-based model keeps the same recurrent state at token 1 and token 100,000, while a transformer keeps adding KV entries.

That is the core contrast: explicit history versus fixed summary. Sometimes a specific token is more useful than the summary, so memory savings need a retrieval test beside them.

The original Mamba paper reported 4-5× higher inference throughput than similar-size Transformers on an A100, with a 2048-token prompt and 128 generated tokens. Skipping the KV cache allowed much larger batch sizes in that comparison. The baseline was a HuggingFace GPT-3-style Transformer, not a FlashAttention-tuned production server.^{[1]Reference 1Mamba: Linear-Time Sequence Modeling with Selective State Spaceshttps://arxiv.org/abs/2312.00752}

Treat that as a reported result, not a guarantee. Batch size, context length, precision, quality, and kernel maturity can change the outcome. The fixture below keeps the paper multiplier separate from a hypothetical local measurement; replace its values with a target-workload benchmark and correctness check.

Why is constant-memory decoding both an advantage and a limitation?

Answer

It keeps serving memory predictable for long generations, which helps throughput and edge budgets. The limitation is that history is compressed into a fixed state, so exact token-level retrieval can be worse than attention over a full KV cache.

Hybrid architectures: combining trade-offs

A documentation-QA decoder can update a compact running state for most tokens, then use an attention layer when it needs to copy an exact function signature. That is the hybrid bet: SSM mode for the bulk of the prompt, attention mode when lookup has to be exact.

Fixed-state SSM designs can regress on precise retrieval, while attention-only stacks incur KV-cache growth as contexts expand. Hybrid architectures are candidates when a workload needs both properties.

Hybrid models: Jamba and Nemotron-H

Jamba^{[4]Reference 4Jamba: A Hybrid Transformer-Mamba Language Modelhttps://arxiv.org/abs/2403.19887} interleaves Transformer and Mamba layers in a structured hybrid stack and adds MoE to some multilayer perceptron (MLP) layers. Its released configuration uses one attention layer for every seven Mamba layers in each 8-layer block, with MoE applied every other layer.

Nemotron-H^{[5]Reference 5Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelshttps://arxiv.org/abs/2504.03624} uses attention in roughly 8% of layers: 4 self-attention layers out of 52 in its 8B model. The remainder is an even split of Mamba-2 and FFN layers. Its 8B stack starts with Mamba-2, ends with FFN, and places an FFN immediately after each attention layer.

Across both examples, most depth stays on the fixed-state path, while attention appears where evaluation shows that explicit token access matters. Tune how often those checkpoints appear and how much depth remains on Mamba-family layers.

• Architecture: Mamba-family layers handle the bulk of depth; attention layers act as retrieval checkpoints
• MoE integration: Jamba uses routing on some MLP layers to increase total capacity without activating every parameter

If attention is a recall checkpoint, expect its placement and frequency to affect both cache and retrieval. Reported hybrid-model evaluations motivate three measurements:

• long-context throughput versus an attention-only baseline
• KV-cache memory reduction as the number of attention layers falls
• retrieval and in-context-learning quality versus both pure SSM and attention-only baselines

Why do hybrids keep occasional attention layers instead of using only SSM layers?

Answer

Attention layers reintroduce explicit token-to-token access. A hybrid can preserve more recall-heavy behavior than a pure SSM while reducing cache growth relative to an all-attention stack, but both outcomes require evaluation.

Why hybrids work

The attention layers act as explicit-access checkpoints: tokens periodically get attention over the context, while Mamba layers process the remaining positions through recurrent state:

When to use SSMs vs. transformers vs. hybrids

There's no universal winner. Start with the failure you need to avoid: exact copying, growing decode memory, or both. Then benchmark the candidate that addresses it.

The fixture below uses hypothetical measurements to show how a deployment gate combines recall, memory, and latency. Use it as a decision shape, not evidence about any model. Replace every number with measurements from the target workload and hardware.

The gate makes one point visible: asymptotic complexity doesn't choose a model for you. Profile actual workloads. SSM candidates are motivated when long-context decoding makes KV-cache growth a bottleneck. Prefill still scales with input length, and short-context batch inference may favor optimized transformer implementations with FlashAttention because their GPU kernels are mature.

Exact-retrieval release gate (NIAH / multi-needle)

Don't ship a pure SSM long-context path on recall SLOs without the same style of needle-in-a-haystack (NIAH) and multi-needle / variable-tracking suite taught in Long Context Window Management. Attention keeps explicit token access. A fixed recurrent state compresses history, so middle-depth and multi-hop retrieval can regress even when average language-modeling loss looks fine.

Use a minimal architecture release gate for SSM vs hybrid vs full attention:

1. Single-needle NIAH at claimed context lengths and depths $\{0, 0.5, 1.0\}$ (start / middle / end).
2. Multi-needle and variable tracking (RULER-style) on the same length grid.
3. Compare pure SSM, hybrid (occasional attention layers), and attention baseline on the same harness, same decode settings, and the same product SLO for recall.
4. Only promote a candidate that clears the recall floor and the memory / p95 budget from the architecture-selection gate above.

Illustrative (hypothetical) middle-depth single-needle recall at a fixed long length; replace with measured heatmaps before any launch decision:

Hybrids often recover much of the recall that pure SSMs lose at middle depth while still cutting KV bytes. That recovery is measured, not guaranteed by the layer fraction alone.

This final fixture is intentionally illustrative. It shows a crossover: a candidate with more expensive prefill can lose on a short generation, then win once lower per-token decode cost amortizes that startup penalty. Predict which path wins at 16 and 256 generated tokens before reading the output.

When should a production team prefer a transformer over an SSM-heavy model despite worse long-context asymptotics?

Answer

Use the transformer when context is short or moderate, exact retrieval and few-shot copying matter, or the serving stack benefits more from mature attention kernels than from SSM-specific scans.

Where fixed-state models lead

You can now trace one token through a fixed state, explain why selective parameters help, and separate a kernel result from a product result. SSMs compress sequence history, Mamba changes what that state keeps, and hybrids trade some of attention's recall for recurrent efficiency. Extra generated tokens still cost decode time either way. Reasoning & Test-Time Compute spends those tokens on purpose.