Jacobians, Hessians, and Why Newton’s Method Loses at Scale
Published:
The Jacobian
For $f : \mathbb{R}^n \to \mathbb{R}^m$, the derivative from the previous post is represented by the $m\times n$ matrix
Row $i$ is the gradient of output $i$; column $j$ is the sensitivity of all outputs to input $j$. That is the whole object. When $m = n$, $\det J_f(x)$ has a clean geometric meaning: it is the factor by which $f$ scales infinitesimal volume at $x$, with the sign recording whether orientation is preserved.
Change of variables, and the price flows pay
Push a random variable $X$ with density $p_X$ through an invertible, differentiable $f$ to get $Y = f(X)$. Probability mass is conserved but the volume element is not, so
or, in the form that actually appears in code,
Read it as an accounting identity: if $f$ expands a neighbourhood, the same probability mass is spread over more volume, so the density there must drop — hence the subtraction.
This single term dictates the design of normalising flows. A general $\det J$ costs $O(n^3)$ to evaluate and is recomputed every training step, which is fatal. The fix is architectural: make $J$ triangular. An affine coupling layer splits the input and writes
where $s$ and $t$ are arbitrary networks. Because $y_1$ ignores $x_2$, the Jacobian is block lower-triangular with diagonal blocks $I$ and $\operatorname{diag}(\exp s(x_1))$. Its determinant is the product of the diagonal, so
an $O(n)$ sum. The networks $s$ and $t$ can be as expressive as you like and never appear in the determinant — the expensive term was engineered away, not approximated.
The Hessian and curvature
For scalar $f$, the Hessian collects second partials, \(H_{ij} = \partial^2 f/\partial x_i \partial x_j\). For twice continuously differentiable $f$ the mixed partials commute, so $H$ is symmetric and the spectral theorem applies: real eigenvalues, orthonormal eigenvectors. The second-order Taylor expansion is
Each eigenvalue of $H$ is the curvature along its eigenvector: positive means the surface bends upward there, negative means downward. Hence the standard conditions at a stationary point ($\nabla f = 0$):
| Hessian at the stationary point | Conclusion |
|---|---|
| positive definite, all $\lambda_i > 0$ | strict local minimum |
| negative definite, all $\lambda_i < 0$ | strict local maximum |
| indefinite, mixed signs | saddle point |
| singular positive semi-definite | inconclusive from second order alone |
A worked case. Let $f(x,y) = x^2 + 4xy + y^2$. Then $\nabla f = (2x + 4y,\ 4x + 2y)$, zero only at the origin, and
Both diagonal entries are positive, yet the matrix is indefinite and the origin is a saddle. Check it directly: along the eigenvector $(1,1)$, $f(t,t) = 6t^2$ rises; along $(1,-1)$, $f(t,-t) = -2t^2$ falls.
Newton’s method, and why it loses
Minimise the quadratic model exactly instead of taking a fixed step along the gradient. Setting the derivative of the model to zero gives the Newton step
Two genuine advantages. It converges quadratically near a nondegenerate minimum — the number of correct digits roughly doubles per iteration. And it is affine invariant: reparameterise $x \mapsto Px$ and the iterates are unchanged, which repairs precisely the scale-sensitivity that makes plain gradient descent suffer on ill-conditioned problems.
Three reasons it is nonetheless rare in deep learning:
- Storage. $H$ has $n^2$ entries. At $n = 10^9$ that is $10^{18}$ numbers.
- Solve cost. A direct solve is $O(n^3)$, per iteration.
- Indefiniteness. Away from a minimum $H$ has negative eigenvalues, and $-H^{-1}\nabla f$ then points uphill along those directions. Newton’s method is attracted to saddle points, not repelled by them, so it needs damping or a trust region to be safe at all.
What quasi-Newton buys
BFGS never forms $H$. It maintains an approximation $B_k$ satisfying the secant condition
which is a finite-difference statement about curvature along the direction just travelled: gradients that changed a lot mean high curvature. L-BFGS goes further and stores only the last $m$ pairs $(s_k, y_k)$, typically $m$ between 5 and 20, reconstructing the step in $O(mn)$ time and memory. You get superlinear convergence in practice at roughly the cost of a gradient.
The honest caveat: the secant condition assumes the two gradients being differenced come from the same function. With minibatch gradients they do not, and the curvature pairs are corrupted by sampling noise. That is why L-BFGS is standard for deterministic, full-batch problems — logistic regression, CRFs, physics-informed fitting — and largely absent from stochastic deep learning, where cheap diagonal preconditioners such as Adam occupy the same niche.
Recap
- \([J_f]_{ij} = \partial f_i/\partial x_j\); \(\det J\) is the local volume scaling factor.
- Change of variables costs \(-\log\lvert\det J\rvert\); coupling layers make \(J\) triangular so the term is an \(O(n)\) sum.
- The Hessian is symmetric, its eigenvalues are curvatures, and a stationary point is a minimum only if all of them are positive — the diagonal alone tells you nothing.
- Newton's method is affine invariant and locally quadratic, but costs \(O(n^2)\) memory and \(O(n^3)\) per solve, and steps towards saddles when \(H\) is indefinite.
- L-BFGS reconstructs curvature from \(m\) recent gradient differences in \(O(mn)\); it degrades under minibatch noise, which is why deep learning uses diagonal preconditioners instead.
References
- Nocedal, J., & Wright, S. J. Numerical Optimization, 2nd ed., ch. 3, 6 and 7. Springer, 2006.
- Liu, D. C., & Nocedal, J. On the limited memory BFGS method for large scale optimization. Mathematical Programming 45, 503–528, 1989.
- Pearlmutter, B. A. Fast Exact Multiplication by the Hessian. Neural Computation 6(1), 147–160, 1994.
- Dinh, L., Sohl-Dickstein, J., & Bengio, S. Density Estimation using Real NVP. ICLR 2017.
- Rezende, D. J., & Mohamed, S. Variational Inference with Normalizing Flows. ICML 2015.
- Dauphin, Y., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., & Bengio, Y. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. NeurIPS 2014.