RNNs, LSTMs, GRUs, and Sequence Modeling

An action probability can choose ignore, rollback, or escalate for an incident assistant. One event saying "rollback requested" is useful, but it means something different after "error logged" and "reproduction confirmed" than it does by itself.

A sequence model reads ordered evidence. Recurrent neural networks (RNNs) do this by applying one learned update repeatedly and carrying a hidden state from one event to the next. LSTMs and GRUs keep the recurrent idea, then add learned gates that control what state should survive.^{[1]Reference 1Long Short-Term Memory.https://doi.org/10.1162/neco.1997.9.8.1735[2]Reference 2Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation.https://arxiv.org/abs/1406.1078}

An ordered history changes a decision

The order of events is part of the input. The small accumulator below isn't a trained RNN, but it exposes the requirement: a late event has more immediate effect because it's applied after earlier state has decayed.

The events are identical as a set, but the final state differs. A useful model must therefore accept variable-length sequences without discarding order.

A recurrent state carries earlier evidence

A basic RNN cell updates one state vector:

$$h_t = \tanh(W_{xh}x_t + W_{hh}h_{t-1} + b_h)$$

Here $x_t$ is the current event vector and $h_{t-1}$ is memory from earlier events. The same matrices $W_{xh}$ and $W_{hh}$ are reused at every timestep. A longer history causes more applications of the cell, not more parameters.

In one dimension, the update is easy to inspect:

The cell turns three inputs into one final value. Real cells use vectors so different dimensions can carry different kinds of evidence.

Trace a vector RNN

Represent each incident event with three binary features:

Use a two-dimensional hidden state, small enough to print at every step:

Attach a classifier to the last hidden state exactly as you attached one to a feature vector in the previous chapter:

Weight sharing is the core economy of recurrence. This count stays fixed as history length grows:

The compression is useful but severe: every past event must influence the decision through two final hidden numbers.

When early learning signals fade

Training unfolds the recurrence across time and backpropagates through every copy. This is backpropagation through time (BPTT). For a loss attached to the final hidden state, its gradient path back to the initial state contains a product of recurrent Jacobians:

$$\frac{\partial L}{\partial h_0} = \frac{\partial L}{\partial h_T} \prod_{t=1}^{T} \frac{\partial h_t}{\partial h_{t-1}}$$

If a model receives losses at several timesteps, their backward paths add together. Each long route still contains repeated factors.

In a scalar simplification, a local derivative below 1 shrinks the signal exponentially:

The reverse failure also occurs. Repeated factors above 1 can make gradients too large:

Gradient clipping can stop an exploding update from destabilizing training. It doesn't give a plain RNN a reliable long memory; a vanished signal is already gone.

LSTM adds a controlled memory track

Long Short-Term Memory (LSTM) introduced gated memory cells designed to preserve useful learning signals over long delays.^{[1]Reference 1Long Short-Term Memory.https://doi.org/10.1162/neco.1997.9.8.1735} The version below uses the later, common forget-gate form. Gers et al. added the adaptive forget gate so a cell could learn when to reset state in continual streams. With a cell state $c_t$, the model has learned controls for keeping, writing, and exposing memory:

$$\begin{aligned} f_t &= \sigma(W_f[h_{t-1},x_t] + b_f) && \text{how much old memory to keep} \\ i_t &= \sigma(W_i[h_{t-1},x_t] + b_i) && \text{how much candidate to write} \\ \tilde{c}_t &= \tanh(W_c[h_{t-1},x_t] + b_c) && \text{candidate content} \\ c_t &= f_t \odot c_{t-1} + i_t \odot \tilde{c}_t \\ o_t &= \sigma(W_o[h_{t-1},x_t] + b_o) && \text{how much memory to expose} \\ h_t &= o_t \odot \tanh(c_t) \end{aligned}$$

The key structural difference is the additive cell update. If the model learns $f_t$ near 1 and $i_t$ near 0 for an important memory dimension, that value can persist instead of being replaced at every step.

One cell dimension is enough to show the mechanism:

routine scan changes little because its write gate is small and its forget gate is high. These values explain the update; during training the network learns its gates from data.

GRU combines the keep and write choice

The Gated Recurrent Unit (GRU) uses one hidden state instead of separate cell and exposed states.^{[2]Reference 2Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation.https://arxiv.org/abs/1406.1078} Using the convention in which $z_t$ is the fraction kept from old state:

$$\begin{aligned} z_t &= \sigma(W_z[h_{t-1},x_t]) \\ r_t &= \sigma(W_r[h_{t-1},x_t]) \\ \tilde{h}_t &= \tanh(W_h[r_t \odot h_{t-1},x_t]) \\ h_t &= z_t \odot h_{t-1} + (1-z_t) \odot \tilde{h}_t \end{aligned}$$

With this notation, $z_t$ close to 1 keeps the prior value. The reset gate $r_t$ controls how much earlier state influences the candidate $\tilde{h}_t$. Some libraries and explanations name or arrange GRU terms differently, so check the equation before interpreting a printed gate.

Pack padding before a library GRU

A production batch often contains claim histories with different lengths. Framework code pads the shorter histories to one rectangular tensor, but those padding rows aren't real events. If a final-state classifier reads a GRU after it processes padding, the filler values can silently change the decision.

PyTorch's nn.GRU accepts a packed sequence. pack_padded_sequence records each history's real length so the recurrent cell stops updating that history before its padding rows. Setting batch_first=True makes the input shape (batch, timesteps, features), which matches the tensor printed below:

The second history has only two real events. Packing makes its final state independent of the third row, while the unpacked GRU treats that row as another event. Masking can solve the same class of problem when an architecture exposes the right state control, but padding must never become evidence.

An encoder can compress a whole sequence

Cho et al. trained an RNN encoder-decoder in which an encoder turns a source sequence into a fixed-size representation and a decoder generates a target sequence from it.^{[2]Reference 2Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation.https://arxiv.org/abs/1406.1078} Sutskever et al. demonstrated the approach with multilayer LSTMs for sequence-to-sequence translation.^{[3]Reference 3Sequence to Sequence Learning with Neural Networks.https://arxiv.org/abs/1409.3215}

The fixed-size interface is easy to see: both a short claim history and a longer one leave the encoder as the same two-number state.

That fixed-size context is both an interface and a bottleneck. The next chapter studies bottlenecks directly: encoders that learn compact representations and decoders that reconstruct from them.

Why attention shortens the route

An RNN has a serial dependency: $h_3$ can't be computed before $h_2$. In the Transformer's comparison of layer types, a recurrent layer has $O(n)$ sequential operations and an $O(n)$ maximum path length between positions. A self-attention layer has $O(1)$ sequential operations and an $O(1)$ maximum path length.^{[4]Reference 4Attention Is All You Need.https://arxiv.org/abs/1706.03762} Full self-attention compares every allowed pair of positions, so its computation and attention-matrix size grow quadratically with sequence length.

This doesn't make recurrence useless. It establishes the design pressure that leads into attention: long dependency routes and a fixed running-state bottleneck.

The comparison is about processing positions already present in a sequence. Autoregressive generation still emits one new token at a time because each next token depends on tokens generated earlier.

Debugging checks

• Print tensor shapes at the cell boundary: input feature size, hidden size, and output size should agree with the weight matrices.
• Confirm that a timestep loop reuses weights. Creating new weights inside the loop changes the model.
• Check which GRU convention your implementation uses before reading gate values.
• For padded batches, prevent padding from updating state through packing or masking. The packed-GRU example above shows the failure directly.
• Clip exploding gradients when needed, then diagnose vanishing memory separately.

---

Mastery check

Key concepts

• Ordered events require a state that's updated in order.
• An RNN reuses one cell's weights across all timesteps.
• The final hidden state can feed a standard logits and softmax action head.
• BPTT multiplies recurrent derivatives across time, producing vanishing or exploding gradients.
• LSTM uses a separate cell state plus forget, input, and output gates.
• GRU uses update and reset gates around a single hidden state.
• Packed recurrent inputs prevent padding rows from changing a shorter history's final state.
• Encoder-decoder recurrence exposes the fixed-size sequence bottleneck.
• Self-attention reduces the path length between distant positions within a layer.

Evaluation rubric

• Foundational: Writes the RNN recurrence and explains weight sharing across timesteps.
• Foundational: Traces the three-event forward pass and maps the final state to action logits.
• Intermediate: Explains vanishing and exploding gradients as repeated derivative products.
• Intermediate: Calculates one LSTM cell update and interprets its gates.
• Intermediate: Reads a declared GRU update-gate convention without reversing keep and write behavior.
• Intermediate: Demonstrates why a padded batch needs packing or masking before a final-state classifier.
• Advanced: Compares a recurrent encoder bottleneck with a self-attention layer's direct paths.

Common pitfalls

• Symptom: Reordering events doesn't affect the result. Cause: History was pooled as an unordered set. Fix: Process events in timestamp order and test a reversed sequence.
• Symptom: Parameter count is claimed to grow with sequence length. Cause: Unrolled computation was confused with separate parameters. Fix: Mark every timestep as sharing W_xh and W_hh.
• Symptom: Training produces unstable updates. Cause: Recurrent gradients explode. Fix: Inspect gradient norms and apply clipping while checking learning-rate and recurrence choices.
• Symptom: Early evidence is never learned. Cause: Gradients vanish through many recurrent updates. Fix: Test gated recurrence or an attention-based architecture for the required dependency length.
• Symptom: A GRU gate is explained backward. Cause: A different update-gate convention was assumed. Fix: State the exact equation before interpreting z_t.
• Symptom: Short histories change prediction when batch padding changes. Cause: Padding rows updated the recurrent state as if they were events. Fix: Pack real sequence lengths or mask state updates before classification.

Follow-up questions