RNNs, LSTMs, and GRUs

A support agent opens a ticket that has been open for four days. The history shows: customer reported "battery drains in 2 hours" (day 1), uploaded a blurry photo of the device (day 2), replied "still not fixed after restart" (day 3), and today added "I need a replacement before my trip." The AI system must decide the next action: escalate to tier-2, request more diagnostics, or offer a refund label. The decision depends on the entire conversation thread, not just the latest message.

A normal feed-forward network or CNN cannot read this thread because the length is variable and the order of events matters. Recurrent neural networks were invented precisely for this kind of sequential, variable-length data.^[1]

---

💡 Key insight: An RNN re-uses the exact same weights at every timestep while carrying a hidden state forward. The hidden state is the network's only memory of everything that happened before the current input. This single design lets the model handle sequences of any length with a fixed number of parameters - but it also creates the vanishing gradient problem that LSTMs and GRUs were built to solve.

---

The sequence modeling problem

Most of the data that matters in production AI is sequential: the ordered list of events in a support ticket, the tokens in a customer email, the clickstream that led to a purchase, or the steps in a warehouse pick list. Each new observation depends on the history that came before it.

A regular neural network expects a fixed-size vector. You could concatenate the entire history into one giant vector, but then:

• Different tickets have different numbers of updates.
• Early events get the same treatment as recent ones even though their influence should decay or persist differently.
• You lose the natural notion of "time" or "order."

Recurrence solves the variable-length problem by processing one element at a time while maintaining a running summary (the hidden state).

The basic RNN recurrence

At every timestep $t$ an RNN computes:

$$h_t = \tanh(W_{xh} x_t + W_{hh} h_{t-1} + b_h)$$

• $x_t$ is the input at this step (a feature vector or token embedding).
• $h_{t-1}$ is the hidden state from the previous step.
• $W_{xh}$ and $W_{hh}$ are the same two weight matrices used at every timestep.
• The output at this step can be produced from $h_t$ (or the final $h_T$ after the whole sequence).

Because the same weights are reused, the model size does not grow with sequence length. The hidden state is the only thing that grows in influence (or fades) as time passes.

Let's make this concrete with a tiny support-ticket example.

Hand-worked RNN forward pass on a 3-event ticket

We will trace a 3-timestep sequence by hand and then implement the identical calculation in NumPy.

Each event is represented by a 3-dimensional feature vector:

• urgency (0–1)
• complexity (0–1)
• customer tier score (higher = more important)

Sequence:

• t=0: [0.2, 0.1, 0.8] - low urgency, simple issue, premium customer
• t=1: [0.9, 0.7, 0.6] - high urgency, complex, mid-tier
• t=2: [0.3, 0.2, 0.9] - medium urgency, simple, premium

We use a 4-dimensional hidden state and the following small shared weights (chosen so the arithmetic is readable):

$$W_{xh} = \begin{bmatrix} 0.5 & 0.1 & -0.2 \\ -0.3 & 0.8 & 0.4 \\ 0.2 & -0.1 & 0.6 \\ 0.7 & 0.3 & -0.4 \end{bmatrix} \quad W_{hh} = \begin{bmatrix} 0.4 & 0.2 & -0.1 & 0.3 \\ 0.1 & 0.6 & 0.2 & -0.2 \\ -0.3 & 0.1 & 0.5 & 0.4 \\ 0.2 & -0.4 & 0.1 & 0.7 \end{bmatrix} \quad b_h = [0.1, -0.05, 0.2, 0.0]$$

• Timestep 0 (initial hidden state $h_0 = [0,0,0,0]$): the network sees the first low-urgency premium event. The hidden state becomes a mild representation of "premium customer with easy issue."
• Timestep 1: we feed the high-urgency complex update. The recurrence mixes the new input with the previous hidden state using the shared weights. After the matrix multiplies and tanh, the hidden state reflects both the initial premium context and the sudden high-urgency signal.
• Timestep 2: the third event arrives. The hidden state is updated once more. By the end of the sequence the 4-dimensional vector $h_3$ encodes the entire history in a compressed form that can be passed to a classifier head or a decoder.

The full numeric trace and the resulting logits appear in the illustration below.

NumPy implementation of the forward pass

Here is a minimal, runnable implementation that reproduces the numbers above.

python
1import numpy as np
2
3def rnn_forward(X, Wxh, Whh, bh, h0=None):
4    """
5    X : (T, input_dim)  - sequence of T feature vectors
6    Returns hidden states (T, hidden_dim) and final hidden state
7    """
8    T, input_dim = X.shape
9    hidden_dim = Whh.shape[0]
10    if h0 is None:
11        h = np.zeros(hidden_dim)
12    else:
13        h = h0.copy()
14
15    hidden_states = []
16    for t in range(T):
17        x_t = X[t]
18        # Linear transform + recurrence + bias
19        h = np.tanh(Wxh @ x_t + Whh @ h + bh)
20        hidden_states.append(h.copy())
21    return np.stack(hidden_states), h
22
23# Same numbers as the worked example
24X = np.array([
25    [0.2, 0.1, 0.8],
26    [0.9, 0.7, 0.6],
27    [0.3, 0.2, 0.9],
28])
29Wxh = np.array([
30    [0.5, 0.1, -0.2],
31    [-0.3, 0.8, 0.4],
32    [0.2, -0.1, 0.6],
33    [0.7, 0.3, -0.4],
34])
35Whh = np.array([
36    [0.4, 0.2, -0.1, 0.3],
37    [0.1, 0.6, 0.2, -0.2],
38    [-0.3, 0.1, 0.5, 0.4],
39    [0.2, -0.4, 0.1, 0.7],
40])
41bh = np.array([0.1, -0.05, 0.2, 0.0])
42
43hiddens, final_h = rnn_forward(X, Wxh, Whh, bh)
44print("Final hidden state:", np.round(final_h, 2))
45# Final hidden state: [0.35 0.72 0.18 0.61]

You can now attach a small output layer (a single linear projection + softmax) to turn final_h into action probabilities: escalate, request photo, close ticket, etc.

Backpropagation through time (BPTT)

Training an RNN means computing gradients of the loss with respect to the shared weights $W_{xh}$ and $W_{hh}$. Because the weights are reused at every timestep, the gradient for $W_{hh}$ must flow backward through every timestep that used it.

This is exactly backpropagation, but the graph is the unrolled chain of $T$ steps. The algorithm is therefore called Backpropagation Through Time.

The problem appears immediately: the gradient for an early hidden state $h_0$ includes the product

$$\frac{\partial L}{\partial h_0} = \frac{\partial L}{\partial h_T} \cdot \left( \prod_{k=1}^{T} \frac{\partial h_k}{\partial h_{k-1}} \right)$$

Each term $\partial h_k / \partial h_{k-1}$ contains $W_{hh}$ multiplied by the derivative of tanh (which is ≤ 1). If the eigenvalues of $W_{hh}$ are smaller than 1 in magnitude, the product shrinks exponentially with $T$. After 20–30 steps the gradient from the first event is effectively zero. That is the vanishing gradient problem.

The opposite (exploding gradients) happens when the spectral radius > 1; gradients grow without bound and training becomes unstable.

LSTM: protecting the memory with gates

The Long Short-Term Memory cell (Hochreiter & Schmidhuber, 1997) introduces an explicit memory cell $c_t$ that is protected from the vanishing problem by multiplicative gates.

At each timestep the LSTM computes three gates (all sigmoid activations, values in [0,1]) and a candidate cell update:

$$\begin{aligned} f_t &= \sigma(W_f \cdot [h_{t-1}, x_t] + b_f) && \text{(forget gate)} \\ i_t &= \sigma(W_i \cdot [h_{t-1}, x_t] + b_i) && \text{(input gate)} \\ \tilde{c}_t &= \tanh(W_c \cdot [h_{t-1}, x_t] + b_c) && \text{(candidate)} \\ c_t &= f_t \odot c_{t-1} + i_t \odot \tilde{c}_t \\ o_t &= \sigma(W_o \cdot [h_{t-1}, x_t] + b_o) && \text{(output gate)} \\ h_t &= o_t \odot \tanh(c_t) \end{aligned}$$

• The forget gate decides what to erase from the cell state.
• The input gate decides what new information to write.
• The output gate decides what parts of the cell state to expose in the hidden state $h_t$.

Because the cell state $c_t$ is updated by addition (not repeated multiplication by $W_{hh}$), gradients can flow backward through many timesteps with only the forget gate values (between 0 and 1) as multipliers. When the forget gate stays close to 1, the gradient does not vanish.

GRU: a lighter gated RNN

The Gated Recurrent Unit (Cho et al., 2014) simplifies the LSTM to two gates while keeping most of the benefit:

$$\begin{aligned} z_t &= \sigma(W_z \cdot [h_{t-1}, x_t]) && \text{(update gate)} \\ r_t &= \sigma(W_r \cdot [h_{t-1}, x_t]) && \text{(reset gate)} \\ \tilde{h}_t &= \tanh(W \cdot [r_t \odot h_{t-1}, x_t]) \\ h_t &= (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t \end{aligned}$$

No separate cell state, no output gate. Many practitioners find GRU performance nearly identical to LSTM on moderate-length sequences while using ~25% fewer parameters and running faster.

Sequence-to-sequence (seq2seq) with encoder-decoder RNNs

A common pattern before 2017 was the encoder-decoder architecture:

1. An encoder RNN (usually LSTM or GRU) reads the entire input sequence (the full support ticket thread) and produces a final hidden state that acts as a compressed "context vector."
2. A decoder RNN (another LSTM/GRU) starts from that context vector and generates an output sequence one token at a time (the resolution summary, the next recommended action, or a reply).

The decoder feeds its own previous prediction back as input at each step (teacher forcing during training).

This pattern powered early neural machine translation and many early dialogue systems. The context vector was the bottleneck - it had to compress an arbitrarily long input into a fixed-size vector.

Why transformers displaced RNNs for long context

The Transformer paper ("Attention Is All You Need", 2017) exposed three fundamental limitations of RNNs that become fatal at scale:

Because every token can attend to every other token in one layer, the model does not need to compress history into a single running hidden state. Residual connections around attention and feed-forward sublayers keep gradients flowing even in 100-layer networks.

The hidden-state mental model is not gone, however. Modern LLMs still maintain a form of running state during generation (the KV cache). The KV cache is the practical descendant of the RNN hidden state: it stores the keys and values computed so far so the model does not recompute them for every new token. Many newer architectures (Mamba, RWKV, RetNet) explicitly bring linear-time recurrence back while keeping the parallel training advantages of transformers.

What you can now do

• Trace any variable-length sequence through a plain RNN by hand and in NumPy.
• Explain to a teammate why a 40-message ticket thread defeats a vanilla RNN but is trivial for a Transformer.
• Choose between LSTM and GRU when you actually need a recurrent component (streaming, very low memory, online state).
• Recognize the architectural ideas that survived the Transformer revolution (running state, gated memory) inside today's production LLM inference engines.

---