Linear Maps as Geometry: Rotate, Scale, Rotate
Published:
Reading a matrix as a motion
A linear map \(T:\mathbb{R}^n\to\mathbb{R}^m\) satisfies \(T(\alpha x + \beta y) = \alpha T(x) + \beta T(y)\), and once you fix bases it is a matrix. Its columns are the images of the basis vectors, which is the fastest way to see what a small matrix does: apply it to \(e_1\) and \(e_2\) and look at where they land.
The four motions worth having memorised in \(\mathbb{R}^2\):
| Map | Matrix | \(\det\) | Effect |
|---|---|---|---|
| Rotation by \(\theta\) | \(\begin{pmatrix}\cos\theta & -\sin\theta\\ \sin\theta & \cos\theta\end{pmatrix}\) | \(+1\) | lengths, angles, orientation preserved |
| Reflection in the line at angle \(\theta\) | \(\begin{pmatrix}\cos 2\theta & \sin 2\theta\\ \sin 2\theta & -\cos 2\theta\end{pmatrix}\) | \(-1\) | lengths preserved, orientation flipped |
| Scaling | \(\mathrm{diag}(s_1, s_2)\) | \(s_1 s_2\) | axis-aligned stretch |
| Shear | \(\begin{pmatrix}1 & k\\ 0 & 1\end{pmatrix}\) | \(1\) | area preserved, angles destroyed |
The shear is the instructive one. It has determinant \(1\), so it preserves area, yet it distorts every angle in the plane and turns the unit circle into a tilted ellipse. Volume preservation and shape preservation are unrelated properties.
Orthogonal matrices are the rigid motions
\(Q\) is orthogonal when \(Q^\top Q = I\). That single condition is equivalent to preserving the inner product:
Preserving inner products means preserving norms (\(y = x\)) and therefore all distances, and preserving angles because both the numerator and denominator of \(\cos\theta\) are unchanged. So orthogonal matrices are precisely the linear isometries: the rigid motions that fix the origin. Since \(\det(Q^\top Q) = (\det Q)^2 = 1\), we get \(\det Q = \pm 1\), and the two cases are rotations (\(+1\), the group \(SO(n)\)) and orientation-reversing reflections (\(-1\)).
This is why orthogonal or unitary weight matrices are attractive in deep nets: they cannot amplify or shrink a signal, so gradients neither explode nor vanish through them.
The determinant is a volume
For a linear map \(A\) and any measurable set \(S\),
\(\det A = 0\) means the image is squashed into a lower-dimensional subspace and volume vanishes, the map is not invertible. The sign records orientation: a positive determinant preserves handedness, a negative one turns a right-handed frame into a left-handed one.
SVD: every linear map is rotate–scale–rotate
The singular value decomposition says that for any real \(A \in \mathbb{R}^{m\times n}\) there exist orthogonal \(U \in \mathbb{R}^{m\times m}\), \(V\in\mathbb{R}^{n\times n}\) and a nonnegative diagonal \(\Sigma\) with
Geometrically: \(V^\top\) rotates (or reflects) the input so that the special directions line up with the axes, \(\Sigma\) stretches each axis by its singular value, and \(U\) rotates the result into place. The picture to keep is that \(A\) maps the unit sphere to an ellipsoid whose semi-axes have lengths \(\sigma_i\) and point along the columns of \(U\). For a square matrix, \(\lvert\det A\rvert = \prod_i \sigma_i\), since the two orthogonal factors contribute \(\pm 1\) each.
Work it out for the shear \(A = \begin{pmatrix}1&1\\0&1\end{pmatrix}\). Then \(A^\top A = \begin{pmatrix}1&1\\1&2\end{pmatrix}\), whose characteristic polynomial is \(\lambda^2 - 3\lambda + 1\), so \(\lambda = (3\pm\sqrt 5)/2 = 2.6180\) and \(0.3820\). The singular values are their square roots:
| Quantity | Value |
|---|---|
| \(\sigma_1\) | \(1.6180\) (the golden ratio \(\varphi\)) |
| \(\sigma_2\) | \(0.6180 = 1/\varphi\) |
| \(\sigma_1\sigma_2\) | \(1.0000\) |
| \(\det A\) | \(1\) |
| Most-stretched input direction | \(\propto (1,\ 1.618)\), at \(58.28^\circ\) |
A map that “preserves area” stretches one direction by \(62\%\) and squeezes another by \(38\%\). The condition number \(\sigma_1/\sigma_2 = 2.618\) is what an optimiser actually feels, and the determinant tells you nothing about it.
Affine versus linear
The layer you actually write is \(x \mapsto Wx + b\), which is affine, not linear: it fails \(T(0) = 0\) whenever \(b \neq 0\). Affine maps preserve straight lines, parallelism, and ratios of lengths along a line, but not lengths, angles or the origin. Composing affine maps gives affine maps, which is why a network of affine layers with no nonlinearity collapses to a single affine map. The standard trick for handling them uniformly is homogeneous coordinates: append a \(1\) to \(x\) and use the \((n{+}1)\times(n{+}1)\) block matrix \(\begin{pmatrix}W & b\\ 0 & 1\end{pmatrix}\), so translation becomes linear one dimension up. Graphics pipelines and \(SE(3)\) pose representations both run on exactly this.
Recap
- Orthogonal \(\Leftrightarrow\) \(Q^\top Q = I\) \(\Leftrightarrow\) preserves inner products \(\Leftrightarrow\) preserves all lengths and angles. \(\det = +1\) rotation, \(\det = -1\) reflection.
- \(\mathrm{vol}(A(S)) = \lvert\det A\rvert\,\mathrm{vol}(S)\); the sign records orientation, and orientation-reversal is a genuine physical distinction, not a bookkeeping detail.
- \(A = U\Sigma V^\top\): rotate, scale by \(\sigma_i\), rotate. The unit sphere becomes an ellipsoid with semi-axes \(\sigma_i\) along the columns of \(U\).
- \(\lvert\det A\rvert = \prod\sigma_i\), so determinant \(1\) is compatible with arbitrarily bad conditioning, shear is the standard counterexample.
- \(x\mapsto Wx+b\) is affine, not linear; homogeneous coordinates make it linear in one extra dimension.
Flat space is now finished. Curvature asks what changes when the space itself bends; the algebraic side of the SVD, how it is computed and how it relates to eigendecomposition, sits in eigenvalues and the SVD.
References
- Strang, G. Introduction to Linear Algebra, 6th ed. Wellesley–Cambridge Press, 2023.
- Trefethen, L. N., & Bau, D. Numerical Linear Algebra. SIAM, 1997.
- Miyato, T., Kataoka, T., Koyama, M., & Yoshida, Y. Spectral Normalization for Generative Adversarial Networks. ICLR 2018.