Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

Euclidean Space: Inner Products, Projections and the Margin

6 minute read

Published:

TL;DR: Everything in Euclidean geometry descends from the inner product. The norm is \(\lVert x\rVert=\sqrt{\langle x,x\rangle}\), the angle is defined by \(\cos\theta = \langle x,y\rangle/(\lVert x\rVert\lVert y\rVert)\), and orthogonality is \(\langle x,y\rangle = 0\). Projecting \(x\) onto a direction \(u\) means choosing the coefficient that minimises the residual, which gives \(\langle x,u\rangle u/\lVert u\rVert^2\) and leaves a residual orthogonal to \(u\). Apply that to a hyperplane \(\mathbf{w}^\top x + b = 0\) and the distance from any point is \(\lvert \mathbf{w}^\top x_0 + b\rvert / \lVert \mathbf{w}\rVert\) — which is the quantity an SVM maximises.

The one structure everything else comes from

A vector space on its own has no lengths and no angles; it only knows about addition and scaling. Add an inner product $\langle \cdot,\cdot\rangle$ — bilinear, symmetric, and positive definite ($\langle x,x\rangle > 0$ for $x \neq 0$) — and geometry appears. In $\mathbb{R}^n$ the standard choice is $\langle x,y\rangle = x^\top y$, and the norm it induces is

\[ \lVert x \rVert = \sqrt{\langle x, x\rangle} = \sqrt{\textstyle\sum_i x_i^2}. \]

The Cauchy–Schwarz inequality $\lvert\langle x,y\rangle\rvert \le \lVert x\rVert\,\lVert y\rVert$ guarantees that the ratio $\langle x,y\rangle/(\lVert x\rVert \lVert y\rVert)$ always lies in $[-1,1]$, so it is legitimate to define the angle $\theta$ between two vectors by

\[ \cos\theta = \frac{\langle x, y\rangle}{\lVert x\rVert\,\lVert y\rVert}. \]

That definition is cosine similarity. It is not an analogy to angles; it is the angle. Orthogonality, $\langle x,y\rangle = 0$, is the case $\theta = 90^\circ$.

Projection, derived rather than quoted

Given $x$ and a nonzero direction $u$, which multiple of $u$ is closest to $x$? Minimise the squared residual over the scalar $\alpha$:

\[ \frac{d}{d\alpha}\,\lVert x - \alpha u\rVert^2 = \frac{d}{d\alpha}\left(\lVert x\rVert^2 - 2\alpha\langle x,u\rangle + \alpha^2\lVert u\rVert^2\right) = -2\langle x,u\rangle + 2\alpha\lVert u\rVert^2 = 0 . \]

So $\alpha^\star = \langle x,u\rangle/\lVert u\rVert^2$ and the projection is \(\mathrm{proj}_u(x) = \frac{\langle x,u\rangle}{\lVert u\rVert^2}\,u\). The residual is automatically orthogonal to $u$, since $\langle x - \alpha^\star u,\ u\rangle = \langle x,u\rangle - \alpha^\star\lVert u\rVert^2 = 0$. Least squares is this same argument in higher dimensions: the fitted values are the projection of $y$ onto the column space of the design matrix, and the residual vector is orthogonal to every regressor.

Projection of a vector onto a line, with orthogonal residualA two-dimensional plot. The vector x equals (3, 4) is drawn from the origin. A line through the origin spans the direction u equals (1, 1). The projection of x onto that line lands at the point (3.5, 3.5), and the dashed residual from (3.5, 3.5) back to (3, 4) equals (minus 0.5, 0.5), meeting the line at a right angle. The angle between x and u is 8.13 degrees. span{u}, u = (1,1) x = (3, 4) proj = (3.5, 3.5) residual (−0.5, 0.5) ⟂ u θ = 8.13° 0 x₁ x₂
Notice that the projection is the unique point on the line whose residual is perpendicular to it — that orthogonality is the optimality condition, not a coincidence of the picture.

With $x = (3,4)$ and $u = (1,1)$: $\langle x,u\rangle = 7$, $\lVert u\rVert^2 = 2$, so $\alpha^\star = 3.5$ and the projection is $(3.5, 3.5)$. The residual $(-0.5, 0.5)$ has inner product $-0.5 + 0.5 = 0$ with $u$, as promised. The angle satisfies $\cos\theta = 7/(5\sqrt{2}) = 0.98995$, so $\theta = 8.13^\circ$.

Orthonormal bases and Gram–Schmidt

A basis \(\\{q_1,\dots,q_n\\}\) is orthonormal when \(\langle q_i,q_j\rangle = \delta_{ij}\). It is the convenient case: coordinates are just inner products, $x = \sum_i \langle x, q_i\rangle q_i$, and lengths are preserved by Parseval, $\lVert x\rVert^2 = \sum_i \langle x,q_i\rangle^2$. Gram–Schmidt manufactures one from any basis \(\\{a_1,\dots,a_n\\}\) by repeatedly subtracting off what is already explained:

\[ v_k = a_k - \sum_{j<k} \langle a_k, q_j\rangle\, q_j, \qquad q_k = \frac{v_k}{\lVert v_k\rVert}. \]

Each step is the projection formula applied and subtracted. In floating point, classical Gram–Schmidt loses orthogonality badly on ill-conditioned inputs; the modified variant, or a Householder QR, is what numerical libraries use.

Hyperplanes and the margin

A hyperplane is the set \(H = \\{x : \mathbf{w}^\top x + b = 0\\}\) with $\mathbf{w} \neq 0$. It is an $(n{-}1)$-dimensional affine subspace, and $\mathbf{w}$ is normal to it: for any $x_1, x_2 \in H$, $\mathbf{w}^\top(x_1 - x_2) = 0$. To find the distance from an arbitrary $x_0$, step from $x_0$ along $\mathbf{w}$ until you hit $H$. Write the foot as $x_0 - t\mathbf{w}$ and solve $\mathbf{w}^\top(x_0 - t\mathbf{w}) + b = 0$, giving $t = (\mathbf{w}^\top x_0 + b)/\lVert \mathbf{w}\rVert^2$. The distance is $\lVert t\mathbf{w}\rVert$:

\[ d(x_0, H) = \frac{\lvert \mathbf{w}^\top x_0 + b\rvert}{\lVert \mathbf{w}\rVert}. \]

Take $\mathbf{w} = (2,1)$, $b = -4$ and $x_0 = (3,3)$: the numerator is $\lvert 6 + 3 - 4\rvert = 5$ and $\lVert\mathbf{w}\rVert = \sqrt 5$, so the distance is exactly $\sqrt 5 \approx 2.236$.

Key Insight — where the SVM's \(2/\lVert \mathbf{w}\rVert\) comes from: the pair \((\mathbf{w}, b)\) is over-parameterised, since scaling both by \(c>0\) leaves the hyperplane unchanged. The hard-margin SVM removes that freedom by fixing the scale with the constraints \(y_i(\mathbf{w}^\top x_i + b)\ge 1\), so the closest points sit at \(\lvert \mathbf{w}^\top x + b\rvert = 1\). By the distance formula they are \(1/\lVert\mathbf{w}\rVert\) away, the two margins are \(2/\lVert\mathbf{w}\rVert\) apart, and maximising the margin becomes minimising \(\lVert\mathbf{w}\rVert^2\). The quadratic objective is not a modelling choice — it is the distance formula in disguise.

What this means for a trained classifier

The last layer of a neural classifier is a set of hyperplanes, one per class in the binary-versus-rest reading. The logit \(\mathbf{w}_c^\top h + b_c\) is a signed, unnormalised distance from the penultimate feature $h$ to class $c$’s boundary; dividing by \(\lVert \mathbf{w}_c\rVert\) turns it into an actual distance. This is why logit magnitudes are not comparable across classes with differently scaled weight vectors, and why weight normalisation and cosine-softmax variants exist at all.

Interview trap: the formula \(\lvert\mathbf{w}^\top x_0 + b\rvert/\lVert\mathbf{w}\rVert\) is a distance only in the norm induced by the inner product you used. Under a different metric — a Mahalanobis inner product \(\langle x,y\rangle_A = x^\top A y\) with \(A \succ 0\), say — orthogonality, angles and the closest point on \(H\) all change. "Perpendicular" is a statement about a chosen inner product, never about the coordinates on the page.

Recap

  • Norm, angle and orthogonality are all defined from the inner product; change the inner product and all three change.
  • \(\mathrm{proj}_u(x) = \frac{\langle x,u\rangle}{\lVert u\rVert^2}u\), derived by minimising \(\lVert x - \alpha u\rVert^2\); the residual is orthogonal to \(u\) by construction. Least squares is the same statement in matrix form.
  • Gram–Schmidt is repeated projection-and-subtract; use the modified version or QR in floating point.
  • Distance from \(x_0\) to \(\{\mathbf{w}^\top x + b = 0\}\) is \(\lvert\mathbf{w}^\top x_0 + b\rvert/\lVert\mathbf{w}\rVert\). Fixing the scale with \(y_i(\mathbf{w}^\top x_i + b)\ge 1\) turns margin maximisation into minimising \(\lVert\mathbf{w}\rVert^2\).

Next: what happens when you move these vectors around — see linear transformations.

References

  1. Boyd, S., & Vandenberghe, L. Convex Optimization. Cambridge University Press, 2004.
  2. Cortes, C., & Vapnik, V. Support-Vector Networks. Machine Learning, 20(3), 1995.
  3. Trefethen, L. N., & Bau, D. Numerical Linear Algebra. SIAM, 1997.

You May Also Enjoy

Noether’s Theorem: Symmetry, Conservation, and Equivariant Networks

7 minute read

Published:

Energy is conserved because the laws of physics do not care what time it is. That single sentence is Noether’s theorem, and its machine learning descendant is the reason an equivariant network needs less data than one that must learn the symmetry from examples.

Entropy and Free Energy: From the Second Law to the ELBO

7 minute read

Published:

Thermodynamic entropy and Shannon entropy differ by a constant with units. Once you accept that, the ELBO stops being an inference trick and becomes a free energy — and diffusion models stop being a clever architecture and become a driven non-equilibrium process.

Statistical Mechanics: The Boltzmann Distribution and the Cost of Z

6 minute read

Published:

Counting microstates gives you the Boltzmann distribution, and the Boltzmann distribution gives you softmax, simulated annealing and energy-based models. The partition function is not a bookkeeping constant — it is the object that contains every thermodynamic quantity, and it is intractable for exactly that reason.