Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

RNNs, LSTMs, and GRUs: Sequence Models Before Attention

7 minute read

Published:

TL;DR: A recurrent network carries one hidden state forward and applies the same weights at every step. Training it sends gradients backwards through a product of Jacobians, which decays or explodes geometrically in the sequence length — so plain RNNs cannot learn long-range dependencies. The LSTM's fix is a cell state updated additively and gated, so the gradient path along it is multiplication by a forget gate rather than by a weight matrix. What neither fixes is that recurrence is inherently sequential.

Weight sharing, but across time

The convolution chapter framed a conv layer as a dense layer with locality and weight sharing imposed. A recurrent network is the same idea rotated: share the weights across time instead of across space.

\[ h_t = \sigma\bigl(W_h h_{t-1} + W_x x_t + b\bigr), \qquad t = 1, \dots, n . \]

One hidden state \(h_t\) summarises everything seen so far. The same \(W_h\), \(W_x\), \(b\) apply at every step, so the model handles sequences of any length with a fixed parameter count — the same bargain a CNN strikes for images of any size.

Backpropagation through time, and the product that kills it

Unroll the recurrence and it is an ordinary feedforward network that happens to reuse its weights. Differentiating the loss at step \(t\) with respect to a hidden state \(k\) steps earlier gives a chain of Jacobians:

\[ \frac{\partial h_t}{\partial h_{t-k}} \;=\; \prod_{j=t-k+1}^{t} \frac{\partial h_j}{\partial h_{j-1}} \;=\; \prod_{j} \operatorname{diag}\bigl(\sigma'(\cdot)\bigr) W_h^{\top} . \]

A product of \(k\) matrices. In the scalar case it is literally \(w^k\), and that is enough to see the problem:

\(w\)\(w^{10}\)\(w^{50}\)\(w^{100}\)
0.900.3490.0052\(2.66\times10^{-5}\)
0.950.5990.0769\(5.92\times10^{-3}\)
1.102.59117\(1.38\times10^{4}\)

A weight of \(0.9\) — not small, not pathological — reduces the gradient by a factor of about 38,000 over a hundred steps. A weight of \(1.1\) multiplies it by 14,000. The knife edge at exactly \(1\) has measure zero.

The nonlinearity makes it worse rather than better. For \(\tanh\), \(\sigma' = 1 - \tanh^2 \le 1\), with equality only at zero: \(\sigma'(1) = 0.42\) and \(\sigma'(2) = 0.071\). Once units saturate, each step contributes a factor well below one. Twenty steps at an effective factor of \(0.42\) gives \(2.9\times10^{-8}\).

The asymmetry is the important part. Exploding gradients are easy to fix: clip the gradient norm and carry on. The direction survives, only the magnitude is capped. Vanishing gradients cannot be fixed that way — there is no information left to rescale. Multiplying a number that has underflowed to \(10^{-8}\) by a large constant recovers nothing. That is why the architectures below attack vanishing specifically.

The general phenomenon — gradients travelling through a long product of Jacobians — is the same one covered in the gradient descent chapter, and it is the reason residual connections exist in deep feedforward networks too.

The LSTM: an additive path through time

The LSTM adds a second state, the cell state \(c_t\), and three gates controlling what enters, leaves, and is read from it. Each gate is a sigmoid, so each produces a value in \((0,1)\) per coordinate.

\[ \begin{aligned} f_t &= \sigma\bigl(W_f [h_{t-1}, x_t] + b_f\bigr) &&\text{forget} \\[2pt] i_t &= \sigma\bigl(W_i [h_{t-1}, x_t] + b_i\bigr) &&\text{input} \\[2pt] \tilde{c}_t &= \tanh\bigl(W_c [h_{t-1}, x_t] + b_c\bigr) &&\text{candidate} \\[2pt] o_t &= \sigma\bigl(W_o [h_{t-1}, x_t] + b_o\bigr) &&\text{output} \\[6pt] c_t &= f_t \odot c_{t-1} + i_t \odot \tilde{c}_t &&\text{cell update} \\[2pt] h_t &= o_t \odot \tanh(c_t) &&\text{hidden state} \end{aligned} \]

The cell update line is the whole design. Look at what it does to the gradient:

\[ \frac{\partial c_t}{\partial c_{t-1}} = f_t . \]

Not a weight matrix. Not a saturating nonlinearity’s derivative. Just the forget gate, elementwise. The path from \(c_{t-k}\) to \(c_t\) is a product of forget gates, and if the network learns to keep \(f \approx 1\) on some coordinate, that coordinate’s gradient passes through essentially undamped. With \(f = 0.99\), a hundred steps still retains \(0.99^{100} = 0.37\) of the signal — against \(2.7\times10^{-5}\) for the vanilla recurrence at \(w = 0.9\).

The multiplicative path in an RNN against the additive cell-state path in an LSTMIn the vanilla recurrent network the state passes through a weight matrix and a nonlinearity at every step. In the LSTM the cell state runs straight through, modified only by an elementwise forget gate and an additive update. Vanilla RNN — every step multiplies by Wₕ and a saturating σ′ Wₕ, σ Wₕ, σ Wₕ, σ Wₕ, σ gradient carries a product of k Jacobians → 0.9¹⁰⁰ ≈ 2.7 × 10⁻⁵ LSTM — the cell state runs straight through ×f ×f ×f ×f gradient carries a product of forget gates → 0.99¹⁰⁰ ≈ 0.37 Same number of steps. The difference is what sits on the path.
The structural difference in one picture. The RNN's state is transformed at every step; the LSTM's cell state is only scaled elementwise and added to, which is what keeps the gradient path open.
The forget-gate bias trick. Initialising \(b_f\) to a positive value (commonly 1) starts every forget gate near \(\sigma(1) \approx 0.73\) rather than near \(0.5\), so the cell state defaults to remembering. It is a one-line change and it matters, because a gate initialised near zero erases the state before the model has had a chance to learn otherwise.

The GRU

The GRU merges the cell and hidden state and uses two gates instead of three — an update gate interpolating between keeping the old state and taking the new candidate, and a reset gate controlling how much history enters the candidate. Fewer parameters, same additive-path idea.

Whether GRU or LSTM is better is task-dependent and usually a small effect. Anyone claiming a universal winner is overstating.

What actually ended them

The gating fixes the gradient problem well enough to be useful. Two limits remain, and neither is about gradients.

Recurrence is sequential. Computing \(h_t\) requires \(h_{t-1}\). That dependency cannot be parallelised across time, so training time scales with sequence length no matter how many GPUs you own. This is a hardware argument, not a modelling one — and it turned out to be the decisive one, because the models that won were the ones that could absorb more compute.

Everything passes through one vector. In an encoder–decoder, the entire input is compressed into a single fixed-size state before decoding begins. Long inputs must lose information; the vector does not grow.

The path between distant positions is long. Information from position 1 reaching position 100 traverses 99 recurrent steps, each an opportunity to be overwritten. Path length is \(O(n)\).

Where this leads. Attention removes both limits at once: every position attends to every other in a single step, so the path length between any two becomes \(O(1)\) instead of \(O(n)\), and all positions are computed in parallel rather than in sequence. The cost is that attention compares every pair, which is \(O(n^2)\) work and memory in the sequence length — a trade the Transformers book takes up from here. None of which makes recurrence obsolete. For short sequences, streaming inputs where the future is genuinely unavailable, or tight memory budgets, a GRU remains a sensible and much smaller choice.

✅ Key Takeaways

  • An RNN shares weights across time as a CNN shares them across space — a fixed parameter count for any sequence length.
  • Backpropagation through time carries a product of \(k\) Jacobians, so gradients move geometrically in \(k\). At \(w = 0.9\), a hundred steps leaves \(2.7\times10^{-5}\); at \(w = 1.1\) it reaches \(1.4\times10^{4}\).
  • Exploding gradients are fixable by clipping. Vanishing gradients are not — there is nothing left to rescale. That asymmetry is what the gating architectures target.
  • The LSTM's cell state is updated additively, so \(\partial c_t / \partial c_{t-1} = f_t\) — a gate, not a weight matrix. At \(f = 0.99\), 0.37 of the gradient survives a hundred steps.
  • Initialise the forget-gate bias positive so the cell defaults to remembering.
  • What ended RNNs was not gradients but parallelism: recurrence is inherently sequential, and the path between distant positions is \(O(n)\). Attention makes both \(O(1)\), at a quadratic cost in sequence length.

You May Also Enjoy

The Standard Library Modules Worth Knowing Before You pip install

8 minute read

Published:

Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed — most of what people reach for a package to solve is one import away.

Files and Context Managers: Why with Is Not Optional

9 minute read

Published:

A file object that falls out of scope without with does get closed eventually — but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.

Errors and Exceptions: EAFP, the Hierarchy, and Reading a Traceback

9 minute read

Published:

Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first — which is faster, shorter, and free of race conditions.