Matrices as Linear Maps: Span, Rank, and the Subspaces They Create
Published:
Two readings of $Ax$
The mechanical reading of $Ax$ is “row $i$ of $A$ dotted with $x$”. It is correct and it explains nothing.
The useful reading is by columns. If $A$ has columns $a_1,\dots,a_n$ and $x = (x_1,\dots,x_n)^\top$, then
So $A$ takes the coordinates in $x$ and uses them as mixing weights on a fixed set of output vectors. The set of everything you can produce this way is the span of the columns — the column space $\operatorname{col}(A)$. Nothing outside it is reachable, which is exactly why $Ax = b$ has a solution if and only if $b \in \operatorname{col}(A)$.
A basis for a subspace is a spanning set with no redundancy, and every basis of a given subspace has the same size — that size is the dimension. The rank of $A$ is $\dim \operatorname{col}(A)$: the number of genuinely independent output directions. A $1000 \times 1000$ matrix of rank 3 maps a thousand-dimensional space onto a three-dimensional plane inside it.
What the map destroys
If the columns are dependent, some non-zero combination of them gives zero. Those coefficient vectors form the null space (or kernel) \(\ker(A) = \{x : Ax = 0\}\), a subspace of the domain. Every vector in it is annihilated.
That gives the conservation law. Each input dimension either survives to contribute an independent output direction, or it is collapsed:
This is rank–nullity, and read this way it is almost a tautology rather than a theorem. The genuinely non-obvious fact sitting alongside it is that row rank equals column rank: the number of independent rows and the number of independent columns are the same, so $\operatorname{rank}(A) = \operatorname{rank}(A^\top)$ even when $A$ is rectangular and the two spaces live in different places.
A worked example
Take
The first two columns $(1,4)^\top$ and $(2,5)^\top$ are independent, so the column space is all of $\mathbb{R}^2$ and $\operatorname{rank}(A) = 2$. The third column is redundant: $(3,6)^\top = -(1,4)^\top + 2\,(2,5)^\top$, since $-1 + 4 = 3$ and $-4 + 10 = 6$.
That dependency is the null space. It says $-1 \cdot a_1 + 2 \cdot a_2 - 1 \cdot a_3 = 0$, so $x = (1,-2,1)^\top$ satisfies $Ax = 0$: check $1 - 4 + 3 = 0$ and $4 - 10 + 6 = 0$. The null space is the line spanned by $(1,-2,1)^\top$, of dimension 1.
| Quantity | Value | Lives in |
|---|---|---|
| $\operatorname{rank}(A)$ | 2 | — |
| $\operatorname{col}(A)$ | all of $\mathbb{R}^2$ | codomain $\mathbb{R}^2$ |
| $\ker(A)$ | \(\operatorname{span}\{(1,-2,1)^\top\}\) | domain $\mathbb{R}^3$ |
| Rank–nullity | $2 + 1 = 3$ | $= n$, the 3 columns |
Geometrically: $A$ flattens $\mathbb{R}^3$ onto the plane $\mathbb{R}^2$ by crushing one particular line to the origin. Solutions to $Ax = b$ are never unique — add any multiple of $(1,-2,1)^\top$ and you get another.
Multiplication is composition; conjugation is a change of basis
Because $(AB)x = A(Bx)$, the product $AB$ is the map “do $B$, then do $A$”. Non-commutativity stops being strange: rotating then projecting is not projecting then rotating. It also explains the shape rule — the output dimension of $B$ must be the input dimension of $A$ — and the bound $\operatorname{rank}(AB) \le \min(\operatorname{rank}A, \operatorname{rank}B)$, since a composition cannot recover dimensions an earlier stage destroyed.
The second reading of a product is a change of coordinates. Let $P$ have as its columns a new basis, written in the old coordinates. Then $P$ converts new coordinates into old ones, and $P^{-1}$ does the reverse. So
reads right to left as: take a vector in the new coordinates, convert to old ($P$), apply the map ($A$), convert back ($P^{-1}$). $A$ and $B$ are the same linear map described in two languages. This is why similar matrices share rank, trace, determinant and eigenvalues — those are properties of the map, not of the description. It is also the whole point of diagonalisation: find the basis in which the description is as simple as possible.
Why this matters in practice
Rank deficiency is what makes $A^\top A$ singular in least squares, forcing ridge regularisation. Rank collapse in a transformer’s attention matrices is a real failure mode. And the entire family of parameter-efficient fine-tuning methods is a bet that the update to a weight matrix has far lower rank than the matrix itself.
Recap
- \(Ax\) is a linear combination of the columns of \(A\); the column space is therefore the set of all reachable outputs, and the rank is its dimension.
- The null space collects the input directions the map destroys. Rank–nullity, \(\operatorname{rank}(A) + \dim\ker(A) = n\), partitions the domain.
- Row rank equals column rank, so \(\operatorname{rank}(A) = \operatorname{rank}(A^\top)\), even for rectangular \(A\).
- \(AB\) means "\(B\) then \(A\)"; \(P^{-1}AP\) is the same map written in a different basis, which is why similar matrices share rank, trace, determinant and eigenvalues.
- Null space lives in the domain, column space in the codomain — do not compare them directly.
References
- Strang, G. Introduction to Linear Algebra, 6th ed. Wellesley-Cambridge Press, 2023. See also MIT 18.06, lectures 5–10.
- Axler, S. Linear Algebra Done Right, 4th ed. Springer, 2024 — the map-first treatment, free to read online.
- Deisenroth, M. P., Faisal, A. A., & Ong, C. S. Mathematics for Machine Learning, ch. 2. Cambridge University Press, 2020.
- Hu, E. J., et al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022.