Gradient Descent and Backpropagation: How a Model Learns

The loop, before any mathematics

A model has parameters. Right now they are wrong. You want them less wrong.

So you need two things: a number that says how wrong (the loss), and a direction that says which way to nudge every parameter to make that number smaller. Everything in this post is those two things, done carefully.

The loop:

1. Push a batch of data forward through the model. Get predictions. Compare to targets. That is one number, \(L\).
2. Push the error backward. Get \(\partial L / \partial \theta_i\) for every single parameter \(\theta_i\). That is backpropagation.
3. Step every parameter a little way against its own gradient. That is gradient descent.
4. Repeat a few hundred thousand times.

Step 3 is three lines of code and the source of most training failures. Step 2 is the algorithmic miracle that makes step 3 affordable. Let us take them in that order, because you need to understand what the step does before it matters how you compute it.

Why the negative gradient?

The gradient \(\nabla_\theta L\) is the vector of all the partial derivatives, stacked. Its geometric content is this: for a small step \(\epsilon u\) in a unit direction \(u\), the first-order change in the loss is

You want this as negative as possible. By Cauchy–Schwarz, \(\nabla_\theta L \cdot u \ge -\lVert \nabla_\theta L \rVert\) for any unit \(u\), with equality precisely when

So the negative gradient is not merely a downhill direction — among all directions of a fixed length it is the one that buys the most loss reduction per unit of movement. That is the whole justification, and it takes two lines.

Note also the word approximately. The formula above is a first-order Taylor expansion, valid for small \(\epsilon\). The gradient tells you the direction. It tells you nothing reliable about how far you can walk before the linear approximation stops being true. That is the learning rate’s job, and it is where things break.

The learning rate, derived rather than tuned

Take the simplest loss with curvature: a one-dimensional quadratic

Three lines and you have the complete answer.

Line one. The gradient is \(\nabla L = a\theta\), so the update \(\theta \leftarrow \theta - \eta \nabla L\) becomes

Line two. That is a plain geometric sequence, so by induction

Line three. A geometric sequence converges to zero if and only if the common ratio is smaller than one in magnitude, so we need \(\lvert 1 - \eta a \rvert < 1\), that is \(-1 < 1 - \eta a < 1\), that is \(0 < \eta a < 2\), that is

Done. And it is not an approximation or a sufficient condition — it is exact, and it is sharp. Read off the four regimes:

The multivariable version is the same statement wearing a hat. For \(L(\theta) = \tfrac12 \theta^\top H \theta\) with \(H\) symmetric positive definite, diagonalise: in the eigenbasis of \(H\) the problem splits into independent one-dimensional problems, one per eigenvalue \(\lambda_i\), each contracting by \(1 - \eta\lambda_i\). Stability therefore requires every one of them to be well behaved:

The sharpest direction sets the speed limit for all the others. Hold that thought — it is the entire content of the next section.

Batch, stochastic, and mini-batch

The loss you actually care about is an average over the dataset:

Gradients are linear, so \(\nabla_\theta L\) is the average of the per-example gradients. Three choices of how many examples to average:

For independent samples the standard error of the mini-batch gradient falls as \(1/\sqrt{B}\). Quadrupling the batch size halves the noise — a poor return, which is exactly why nobody pushes batch sizes to infinity even when memory allows.

What the noise actually does

The honest answer is: it depends, and the folklore overstates it.

What is genuinely true is that the mini-batch gradient is an unbiased estimator of the full gradient, so on average you are still walking downhill. The variance means you are walking downhill drunk. A few consequences follow with reasonable confidence:

• Exact minima of the mini-batch loss are not exact minima of the full loss. Noise stops you from settling into any single batch’s idiosyncratic dip.
• Noise lets you leave shallow basins. A perturbation of a given size escapes a shallow minimum and does not escape a deep one, so there is a real filtering effect.
• Noise scale is coupled to \(\eta/B\), not to \(\eta\) or \(B\) alone. This is why doubling the batch size and doubling the learning rate together often reproduces the original training curve almost exactly. It is a rule of thumb that holds well in a useful regime and breaks at large batch sizes.

Conditioning: why plain gradient descent zig-zags

Here is the failure mode that motivates every optimiser after 1964.

Consider \(L(\theta) = \tfrac12(\theta_1^2 + 10\,\theta_2^2)\). The curvatures are \(\lambda_{\min} = 1\) and \(\lambda_{\max} = 10\), so the condition number is

The contours are ellipses ten times more curved across the valley than along it. The gradient at any off-axis point is dominated by the steep direction, so it points mostly across the valley rather than along it — and the single scalar \(\eta\) has to serve both directions at once. Stability caps it at \(2/\lambda_{\max} = 0.2\), which is far too small for the \(\lambda_{\min}=1\) direction, where you would happily have used \(\eta = 1\).

The classically optimal choice balances the two:

At this \(\eta\), the two contraction factors are \(1 - \eta^\star \lambda_{\min} = +9/11\) and \(1 - \eta^\star \lambda_{\max} = -9/11\). Equal in magnitude, opposite in sign: the second coordinate flips sign every step while both shrink by \(9/11 \approx 0.818\). That sign flip is the zig-zag, and note that it happens at the best possible learning rate. It is not a tuning mistake. It is what the geometry forces.

The general result: with the optimal \(\eta\), plain gradient descent on a quadratic contracts the error by a factor of

For \(\kappa = 10\) that is \(9/11 = 0.818\), so reducing the error by a factor of \(10^{6}\) needs about \(\ln(10^{-6})/\ln(0.818) \approx 69\) iterations — and running the iteration confirms 69. For \(\kappa = 1000\), \((\kappa-1)/(\kappa+1) = 0.998\) and you need about 6900. Iteration count scales roughly linearly in \(\kappa\), which is the reason real training problems — where \(\kappa\) can be enormous — are slow.

Momentum

Momentum treats the parameter vector as a ball with mass rather than a point that teleports. Maintain a running velocity and step along it:

with \(\mu\) typically \(0.9\). The mechanism is exactly the one the zig-zag figure suggests: the across-valley component of the gradient reverses sign every step and therefore cancels in the running sum, while the along-valley component keeps the same sign and accumulates. For a constant gradient \(g\) the velocity converges to \(g/(1-\mu)\), so \(\mu = 0.9\) means a persistent direction eventually moves you ten times faster than it would without momentum.

The payoff on a quadratic is a genuine improvement in the exponent. With the optimal heavy-ball settings

the contraction factor becomes \((\sqrt{\kappa}-1)/(\sqrt{\kappa}+1)\) rather than \((\kappa-1)/(\kappa+1)\) — dependence on \(\sqrt{\kappa}\) instead of \(\kappa\). On our \(\kappa = 10\) problem that is \(0.519\) against \(0.818\), and running both to a \(10^{-6}\) relative error gives 26 iterations with momentum against 69 without. For \(\kappa = 1000\) the same formulas predict roughly 6900 against roughly 220.

Adam

Adam keeps two exponential moving averages per parameter — a first moment (the mean gradient) and a second moment (the mean squared gradient) — and uses the second to rescale the first.

Defaults are \(\beta_1 = 0.9\), \(\beta_2 = 0.999\), \(\epsilon = 10^{-8}\).

What the second moment does. The update is a signal-to-noise ratio: a parameter whose gradient is consistently \(0.001\) gets \(\hat m / \sqrt{\hat v} \approx 1\), and so does a parameter whose gradient is consistently \(1000\). The step size becomes roughly \(\eta\) for every parameter, regardless of gradient scale. That is a per-parameter learning rate obtained for free, and it is why Adam tolerates a network whose layers have wildly different gradient magnitudes. It also, as promised earlier, means Adam is doing steepest descent in a diagonal metric of its own devising rather than the Euclidean one.

Why bias correction is not cosmetic. Both averages start at zero, which biases them towards zero early on. Unrolling with a constant gradient \(g\):

which is exactly the factor \(\hat m_t\) divides out. Concretely at \(t=1\): \(m_1 = 0.1\,g\) — a tenth of the true gradient — and \(v_1 = 0.001\,g^2\). Without correction the first update direction would be \(0.1g / \sqrt{0.001 g^2} = \sqrt{10} \approx 3.16\) times its intended size, in the wrong direction of error (the first moment is under-estimated by ten, but the square root of the second moment is under-estimated by 31.6, and the smaller denominator wins). Bias correction makes the ratio exactly \(1\) from the first step. This is the reason Adam without correction needs a warm-up and Adam with it usually does not.

Backpropagation is the chain rule, ordered correctly

Now the other half. Gradient descent needs \(\nabla_\theta L\). A modern model has \(10^{8}\) to \(10^{12}\) parameters. How do you get \(10^{12}\) partial derivatives without doing \(10^{12}\) units of work?

The naive method is finite differences: perturb each parameter, re-run the model, see how the loss moves. That costs one forward pass per parameter. For a billion parameters, one gradient would take a billion forward passes. Training would be over before the heat death of the universe.

Backpropagation costs about two forward passes, total, regardless of the parameter count. Here is why.

Forward mode versus reverse mode

Write your model as a composition of \(k\) elementary steps,

so by the chain rule the Jacobian is a product of the per-step Jacobians:

Matrix multiplication is associative, so you may bracket that product however you like and the answer is identical. The cost is not identical. That single observation is all of automatic differentiation.

Now put in the numbers for training. The inputs are the parameters, so \(n\) is in the billions. The output is the scalar loss, so \(m = 1\).

Reverse mode needs one sweep. Forward mode needs \(n\). That factor of \(n\) is the entire reason deep learning is computationally possible, and it comes from nothing more than choosing which end of an associative product to start at.

Backpropagation is reverse-mode automatic differentiation, specialised to a neural network. Setting \(u = 1\) (the loss is scalar), the row vector propagating backwards is exactly the familiar \(\delta\) of the textbook derivation.

Backprop by hand on a network small enough to check

Two layers, scalar input, scalar output, one nonlinearity. Forward:

Concrete values: \(x = 1.5\), target \(y = 0.5\), and parameters \(w_1 = 0.8\), \(b_1 = -0.2\), \(w_2 = 1.4\), \(b_2 = 0.3\). The forward pass gives

Now backwards, one link of the chain at a time. Each line uses only the line above it and one local derivative — that is the whole algorithm.

Start at the loss.

Layer 2 parameters. Since \(z_2 = w_2 h + b_2\), the local derivatives are \(\partial z_2/\partial w_2 = h\) and \(\partial z_2/\partial b_2 = 1\):

Through layer 2 into the hidden unit. Here \(\partial z_2/\partial h = w_2\):

Through the nonlinearity. This is the step that matters for everything later. Since \(\tanh'(z) = 1 - \tanh^{2}(z) = 1 - h^{2}\):

Layer 1 parameters. With \(\partial z_1/\partial w_1 = x\) and \(\partial z_1/\partial b_1 = 1\):

Notice the shape of the computation. Everything flowed through two scalars, \(\delta_2\) and \(\delta_1\). Each parameter’s gradient is then just “the delta arriving at my layer, times my local input”. In a real network the deltas are vectors and the local products become outer products, but the structure is identical, and the total work is one multiply-add per weight — the same order as the forward pass.

The numerical check

None of the above is worth anything if it is wrong, so verify it against the definition of a derivative. Central differences, \(\left(L(\theta+\epsilon) - L(\theta-\epsilon)\right)/(2\epsilon)\) with \(\epsilon = 10^{-6}\):

import numpy as np

x, y = 1.5, 0.5
p0 = np.array([0.8, -0.2, 1.4, 0.3])          # w1, b1, w2, b2

def loss(p):
    w1, b1, w2, b2 = p
    h = np.tanh(w1 * x + b1)
    return 0.5 * (w2 * h + b2 - y) ** 2

# backprop, exactly as derived above
w1, b1, w2, b2 = p0
z1 = w1 * x + b1
h  = np.tanh(z1)
d2 = (w2 * h + b2) - y
d1 = d2 * w2 * (1 - h * h)
analytic = np.array([d1 * x, d1, d2 * h, d2])

eps = 1e-6
fd = np.array([(loss(p0 + eps * e) - loss(p0 - eps * e)) / (2 * eps)
               for e in np.eye(4)])

print(analytic)
print(fd)
print("max abs diff:", np.max(np.abs(analytic - fd)))

Running it:

Largest absolute discrepancy \(9.27 \times 10^{-11}\); largest relative discrepancy \(1.41 \times 10^{-10}\).

That agreement is not merely “good” — it is as good as double precision permits. Central differences carry two error terms: a truncation error of order \(\epsilon^{2} L'''\) (about \(10^{-12}\) here) and a round-off error of order \(\varepsilon_{\text{machine}}/\epsilon \approx 10^{-16}/10^{-6} = 10^{-10}\). The observed \(10^{-10}\) is the round-off floor. The backprop derivation is exact; the finite difference is the approximate one.

Vanishing and exploding gradients

Now stack the worked example many times and watch what the chain rule does.

For a deep network with hidden states \(h_0, h_1, \ldots, h_T\), the chain rule gives the gradient at the input end as a product of every intermediate Jacobian:

Submultiplicativity of the operator norm gives

The behaviour is exponential in depth, and the base is roughly the typical Jacobian norm:

Saturating nonlinearities make this the default rather than the exception. We computed \(\tanh'(z_1) = 1 - h^{2} = 0.4199743416\) in the worked example above, at the perfectly reasonable pre-activation \(z_1 = 1\). And \(\tanh'\) never exceeds \(1\) anywhere, attaining it only at \(z = 0\). Twenty such layers, ignoring the weights entirely, contribute \(0.42^{20} \approx 2.9 \times 10^{-8}\). The gradient does not so much vanish as evaporate.

This single product is the reason for three things you will meet later:

What this means when you are actually training