RNNs, LSTMs, GRUs, and Sequence Modeling

An incident assistant ranks ignore, rollback, and escalate from action probabilities. One event saying "rollback requested" is useful, but it means something different after "error logged" and "reproduction confirmed" than it does by itself.

A sequence model reads ordered evidence. Recurrent neural networks (RNNs) do this by applying one learned update repeatedly and carrying a hidden state from one event to the next. LSTMs and GRUs keep the recurrent idea, then add learned gates that control what state should survive.^{[1]Reference 1Long Short-Term Memory.https://doi.org/10.1162/neco.1997.9.8.1735[2]Reference 2Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation.https://arxiv.org/abs/1406.1078}

An ordered history changes a decision

The order of events is part of the input. The small accumulator below isn't a trained RNN, but it exposes the requirement: a late event has more immediate effect because it's applied after earlier state has decayed.

The events are identical as a set, but the final state differs. A useful model must therefore accept variable-length sequences without discarding order.

A recurrent state carries earlier evidence

A basic RNN cell updates one state vector:

$$h_t = \tanh(W_{xh}x_t + W_{hh}h_{t-1} + b_h)$$

Here $x_t$ is the current event vector and $h_{t-1}$ is memory from earlier events. The same matrices $W_{xh}$ and $W_{hh}$ are reused at every timestep. A longer history causes more applications of the cell, not more parameters.

In one dimension, the update is easy to inspect:

The cell turns three inputs into one final value. Real cells use vectors so different dimensions can carry different kinds of evidence.

Why can an RNN read either 3 events or 30 events without changing its parameter count?

Answer

It reuses the same input and recurrent weights at each timestep. Sequence length changes the number of cell applications, not the number of learned weights.

Trace a vector RNN

Represent each incident event with three binary features:

Use a two-dimensional hidden state, small enough to print at every step:

Attach a classifier to the last hidden state exactly as you attached one to a feature vector in the previous chapter:

Weight sharing is the core economy of recurrence. This count stays fixed as history length grows:

The compression is useful but severe: every past event must influence the decision through two final hidden numbers.

When early learning signals fade

Training unfolds the recurrence across time and backpropagates through every copy. This is backpropagation through time (BPTT). For a loss attached to the final hidden state, its gradient path back to the initial state contains a product of recurrent Jacobians:

$$\frac{\partial L}{\partial h_0} = \frac{\partial L}{\partial h_T} \prod_{t=1}^{T} \frac{\partial h_t}{\partial h_{t-1}}$$

If a model receives losses at several timesteps, their backward paths add together. Each long route still contains repeated factors.

In a scalar simplification, a local derivative below 1 shrinks the signal exponentially:

The reverse failure also occurs. Repeated factors above 1 can make gradients too large:

Gradient clipping can stop an exploding update from destabilizing training. It doesn't give a plain RNN a reliable long memory; a vanished signal is already gone.

LSTM adds a controlled memory track

Long Short-Term Memory (LSTM) introduced gated memory cells designed to preserve useful learning signals over long delays.^{[1]Reference 1Long Short-Term Memory.https://doi.org/10.1162/neco.1997.9.8.1735} The version below uses the later, common forget-gate form. Gers et al. added the adaptive forget gate so a cell could learn when to reset state in continual streams.^{[3]Reference 3Learning to Forget: Continual Prediction with LSTMhttps://doi.org/10.1162/089976600300015015} With a cell state $c_t$, the model has learned controls for keeping, writing, and exposing memory:

$$\begin{aligned} f_t &= \sigma(W_f[h_{t-1},x_t] + b_f) && \text{how much old memory to keep} \\ i_t &= \sigma(W_i[h_{t-1},x_t] + b_i) && \text{how much candidate to write} \\ \tilde{c}_t &= \tanh(W_c[h_{t-1},x_t] + b_c) && \text{candidate content} \\ c_t &= f_t \odot c_{t-1} + i_t \odot \tilde{c}_t \\ o_t &= \sigma(W_o[h_{t-1},x_t] + b_o) && \text{how much memory to expose} \\ h_t &= o_t \odot \tanh(c_t) \end{aligned}$$

The key structural difference is the additive cell update. If the model learns $f_t$ near 1 and $i_t$ near 0 for an important memory dimension, that value can persist instead of being replaced at every step.

That same product also governs the backward path. Along the cell-state track, treating the gates as fixed for the derivative with respect to earlier cell values,

$$\frac{\partial c_t}{\partial c_{t-1}} = \mathrm{diag}(f_t)$$

(up to additional terms when candidate writes also depend on $c_{t-1}$ through the gates). If $f_t \approx 1$ on a dimension, the local factor is near 1, so error can travel many steps without the repeated nonlinear Jacobian product that vanishes in a plain RNN $h_t = \tanh(W_{hh}h_{t-1}+\ldots)$. Hochreiter and Schmidhuber called this the constant error carousel: the forget gate chooses when that path stays open.^{[1]Reference 1Long Short-Term Memory.https://doi.org/10.1162/neco.1997.9.8.1735}

One cell dimension is enough to show the mechanism:

routine check changes little because its write gate is small and its forget gate is high. These values explain the update; during training the network learns its gates from data.

What does an LSTM forget gate value near 1 do?

Answer

It passes most of the corresponding old cell-state dimension into the next cell state. Combined with a small input gate, it preserves that stored feature across the current step.

GRU combines the keep and write choice

The Gated Recurrent Unit (GRU) uses one hidden state instead of separate cell and exposed states.^{[2]Reference 2Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation.https://arxiv.org/abs/1406.1078} Using the convention in which $z_t$ is the fraction kept from old state:

$$\begin{aligned} z_t &= \sigma(W_z[h_{t-1},x_t]) \\ r_t &= \sigma(W_r[h_{t-1},x_t]) \\ \tilde{h}_t &= \tanh(W_h[r_t \odot h_{t-1},x_t]) \\ h_t &= z_t \odot h_{t-1} + (1-z_t) \odot \tilde{h}_t \end{aligned}$$

With this notation, $z_t$ close to 1 keeps the prior value. The reset gate $r_t$ controls how much earlier state influences the candidate $\tilde{h}_t$. Some libraries and explanations name or arrange GRU terms differently, so check the equation before interpreting a printed gate.

Pack padding before a library GRU

A production batch often contains event histories with different lengths. Framework code pads the shorter histories to one rectangular tensor, but those padding rows aren't real events. If a final-state classifier reads a GRU after it processes padding, the filler values can silently change the decision.

PyTorch's nn.GRU accepts a packed sequence.^{[4]Reference 4GRUhttps://docs.pytorch.org/docs/2.13/generated/torch.nn.GRU.html} pack_padded_sequence records each history's real length so the recurrent cell stops updating that history before its padding rows.^{[5]Reference 5torch.nn.utils.rnn.pack_padded_sequencehttps://docs.pytorch.org/docs/2.13/generated/torch.nn.utils.rnn.pack_padded_sequence.html} Setting batch_first=True makes the input shape (batch, timesteps, features), which matches the tensor printed below:

The second history has only two real events. Packing makes its final state independent of the third row, while the unpacked GRU treats that row as another event. Masking can solve the same class of problem when an architecture exposes the right state control, but padding must never become evidence.

Why can padding corrupt a final-state classifier even if every padded row uses the same placeholder value?

Answer

Without packing or masking, the recurrent cell still applies another update for each padded timestep. The classifier then receives the state after filler events instead of the state after the last real event.

An encoder can compress a whole sequence

Cho et al. trained an RNN encoder-decoder in which an encoder turns a source sequence into a fixed-size representation and a decoder generates a target sequence from it.^{[2]Reference 2Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation.https://arxiv.org/abs/1406.1078} Sutskever et al. demonstrated the approach with multilayer LSTMs for sequence-to-sequence translation.^{[6]Reference 6Sequence to Sequence Learning with Neural Networks.https://arxiv.org/abs/1409.3215}

The fixed-size interface is easy to see: both a short claim history and a longer one leave the encoder as the same two-number state.

That fixed-size context is both an interface and a bottleneck. The next chapter studies bottlenecks directly: encoders that learn compact representations and decoders that reconstruct from them.

Why attention shortens the route

An RNN has a serial dependency: $h_3$ can't be computed before $h_2$. In the Transformer's comparison of layer types, a recurrent layer has $O(n)$ sequential operations and an $O(n)$ maximum path length between positions. A self-attention layer has $O(1)$ sequential operations and an $O(1)$ maximum path length.^{[7]Reference 7Attention Is All You Need.https://arxiv.org/abs/1706.03762} Full self-attention compares every allowed pair of positions, so its computation and attention-matrix size grow quadratically with sequence length.

This doesn't make recurrence useless. It separates two design pressures: the fixed running-state bottleneck and the long dependency route. The next lesson isolates the bottleneck with autoencoders; the Transformer lesson after it replaces the recurrent route with attention.

The comparison is about processing positions already present in a sequence. Autoregressive generation still emits one new token at a time because each next token depends on tokens generated earlier.

Debugging checks

• Print tensor shapes at the cell boundary: input feature size, hidden size, and output size should agree with the weight matrices.
• Confirm that a timestep loop reuses weights. Creating new weights inside the loop changes the model.
• Check which GRU convention your implementation uses before reading gate values.
• For padded batches, prevent padding from updating state through packing or masking. The packed-GRU example above shows the failure directly.
• Clip exploding gradients when needed, then diagnose vanishing memory separately.

---

Mastery check

Key concepts

• Ordered events require a state that's updated in order.
• An RNN reuses one cell's weights across all timesteps.
• The final hidden state can feed a standard logits and softmax action head.
• BPTT multiplies recurrent derivatives across time, producing vanishing or exploding gradients.
• LSTM uses a separate cell state plus forget, input, and output gates.
• GRU uses update and reset gates around a single hidden state.
• Packed recurrent inputs prevent padding rows from changing a shorter history's final state.
• Encoder-decoder recurrence exposes the fixed-size sequence bottleneck.
• Self-attention reduces the path length between distant positions within a layer.

Evaluation rubric

• Foundational: Writes the RNN recurrence and explains weight sharing across timesteps.
• Foundational: Traces the three-event forward pass and maps the final state to action logits.
• Intermediate: Explains vanishing and exploding gradients as repeated derivative products.
• Intermediate: Calculates one LSTM cell update and interprets its gates.
• Intermediate: Reads a declared GRU update-gate convention without reversing keep and write behavior.
• Intermediate: Demonstrates why a padded batch needs packing or masking before a final-state classifier.
• Advanced: Compares a recurrent encoder bottleneck with a self-attention layer's direct paths.

Common pitfalls

• Symptom: Reordering events doesn't affect the result. Cause: History was pooled as an unordered set. Fix: Process events in timestamp order and test a reversed sequence.
• Symptom: Parameter count is claimed to grow with sequence length. Cause: Unrolled computation was confused with separate parameters. Fix: Mark every timestep as sharing W_xh and W_hh.
• Symptom: Training produces unstable updates. Cause: Recurrent gradients explode. Fix: Inspect gradient norms and apply clipping while checking learning-rate and recurrence choices.
• Symptom: Early evidence is never learned. Cause: Gradients vanish through many recurrent updates. Fix: Test gated recurrence or an attention-based architecture for the required dependency length.
• Symptom: A GRU gate is explained backward. Cause: A different update-gate convention was assumed. Fix: State the exact equation before interpreting z_t.
• Symptom: Short histories change prediction when batch padding changes. Cause: Padding rows updated the recurrent state as if they were events. Fix: Pack real sequence lengths or mask state updates before classification.

Follow-up questions

Why does the final RNN state contain information from all three incident events?

Answer

Each update consumes the current event and the previous state. The second state depends on the first event through the first state, and the third state depends on both earlier updates through the second state.

Why can clipping an exploding gradient not repair a vanished gradient?

Answer

Clipping limits an update whose norm is too large. A vanished gradient has already shrunk toward zero, so limiting its norm can't restore the missing learning signal.

What extra path does an LSTM add beyond a plain RNN hidden state?

Answer

It adds a cell state updated by a gated additive rule. A forget gate near 1 and an input gate near 0 can carry a stored value across a step with little change.

Why can a forget gate near 1 keep gradients from vanishing along the cell path?

Answer

Along the cell track, ∂c_t/∂c_{t-1} equals the forget gate (on the diagonal). A gate near 1 multiplies the backward signal by about 1 at that step, so the error isn't forced through a full nonlinear Jacobian product the way a plain RNN hidden-state path is.

What bottleneck does a classic recurrent encoder-decoder create?

Answer

The encoder must place the information needed by the decoder into a fixed-size final context state, regardless of input sequence length.