Convolutions and CNNs: Weight Sharing as a Prior
Published:
Start from the layer you already know
Take a dense layer. Every output unit sees every input, with its own private weight for each. That is the most expressive linear map available, and on images it is a disaster — not because it cannot represent the right function, but because it can represent far too many wrong ones, and you would need an implausible amount of data to pick out the right one.
So impose two constraints.
Local connectivity. An output unit at position \((i,j)\) may read only a small window of the input centred on \((i,j)\) — say \(3 \times 3\). Every weight connecting it to a distant pixel is forced to zero. The assumption: whatever an edge or a corner is, you can tell it is there by looking at a small patch. You do not need pixel \((3,7)\) to decide whether there is an edge at \((180, 200)\).
Weight sharing. Every output unit in the same channel uses the same window of weights. The unit at \((0,0)\) and the unit at \((180,200)\) are computed by the identical \(3 \times 3\) pattern of numbers. The assumption: a vertical edge in the top-left corner is the same thing as a vertical edge in the bottom-right, so it should be detected by the same detector.
That is the entire content of a convolutional layer. Everything else — stride, padding, dilation, channels — is bookkeeping about which windows and how many detectors. The two assumptions above are the prior, and the prior is what does the work.
The confession: it is cross-correlation
The operation every framework calls conv2d is not convolution. Convolution, in the signal-processing sense, flips the kernel:
What PyTorch, TensorFlow, JAX and every other library actually compute is cross-correlation, which does not:
Say it once, plainly: the flip is missing, and it does not matter for learning. The kernel \(w\) is a free parameter. Whatever the flipped version would have learned, the unflipped version learns the mirror image of it, and the two networks compute exactly the same set of functions. The function class is identical, so nobody bothers to flip.
It does matter in three places. First, if you hand-set a kernel — a Sobel operator, a Gaussian derivative — the sign and orientation conventions from a textbook will come out reversed. Second, true convolution is commutative and associative and satisfies the convolution theorem; cross-correlation does not, so any proof that leans on those properties needs the flip put back. Third, the gradient of a cross-correlation with respect to its input is a convolution (a genuine one), which is why the backward pass and the “transposed convolution” used for upsampling look flipped relative to the forward pass.
For the rest of this post, “convolution” means what the framework means.
The parameter argument, done properly
Take the standard ImageNet input shape: \(224 \times 224\) pixels, 3 colour channels. That is
Dense layer to 1000 units. Every one of the 1000 units gets its own weight for each of the 150,528 inputs, plus a bias:
Roughly 150.5 million weights in a single layer, before you have computed a single nonlinearity.
Convolution, \(3 \times 3\), 64 filters. Each filter spans the full input depth, so it is \(3 \times 3 \times 3\), and there are 64 of them, each with a bias:
The ratio of the weight counts is \(150{,}528{,}000 / 1728 = 87{,}111.1\ldots\); including biases, \(150{,}529{,}000 / 1792 \approx 84{,}000\). Either way: roughly five orders of magnitude.
That comparison is a little unfair in the convolution’s favour, because the two layers do not produce the same shape of output. Make it fair. The convolution with “same” padding produces a \(224 \times 224 \times 64\) volume, which is \(3{,}211{,}264\) values. A dense layer producing that same volume from the same input would need
about 280 million times as many as the 1,728 the convolution uses. The like-for-like saving is far larger than the headline comparison suggests.
That reuse factor is the honest statement of what weight sharing does. It is not that the layer does less work; it is that the same small set of numbers is held accountable for 50,176 separate predictions, so the data has 50,176 times as much leverage on each one.
Channels: why a kernel is three-dimensional
A common confusion: people picture a \(3 \times 3\) kernel as a \(3 \times 3\) grid of numbers. On a colour image it is not — it is \(3 \times 3 \times 3\), a small cuboid, because it must span all input channels.
The rule generalises. With \(C_{\text{in}}\) input channels, one filter is a \(k \times k \times C_{\text{in}}\) tensor, and it produces one output channel. “64 filters” means 64 such cuboids, stacked into a weight tensor of shape \((C_{\text{out}}, C_{\text{in}}, k, k)\), producing 64 output channels:
Check it: \(64 \times (3 \times 9 + 1) = 64 \times 28 = 1792\). Matches.
There is a structural point hiding here. The layer is locally connected in space and fully connected across channels — every output channel reads every input channel. Locality was imposed on the spatial axes only, because that is where the prior applies: neighbouring pixels are related, but “channel 17” and “channel 18” have no such relationship. Architectures that break the channel-wise density on purpose (depthwise separable convolutions, grouped convolutions) do so for efficiency, and they pay for it with a \(1 \times 1\) convolution to mix channels back together.
Stride, padding, dilation, and the output size
Three knobs change which windows are taken.
- Stride \(s\): move the window \(s\) positions between outputs instead of 1. Stride 2 halves the resolution.
- Padding \(p\): add \(p\) rows and columns of (usually) zeros to each side before sliding, so that the output can stay the same size as the input.
- Dilation \(d\): spread the kernel taps \(d\) apart, so a \(3 \times 3\) kernel with \(d = 2\) covers a \(5 \times 5\) footprint while still holding only 9 weights.
Deriving the output size
Work in one dimension; the two-dimensional case is the same formula applied per axis. After padding, the signal has length \(n + 2p\). A dilated kernel with \(k\) taps spaced \(d\) apart has an effective size
(with \(d = 1\) this is just \(k\)). Place the window at offset \(i\), counting from zero. It occupies positions \(i, i+1, \ldots, i + k' - 1\), and it fits entirely inside the padded signal exactly when
The legal offsets are \(i = 0, s, 2s, 3s, \ldots\), so the number of them is the number of multiples of \(s\) in \([0,\, n + 2p - k']\), which is
For the ordinary undilated case this is the formula everyone memorises:
The \(+1\) is the window at offset zero; the floor is the windows that would hang off the end being discarded. Both parts fall straight out of “count the legal offsets”, which is why it is worth deriving once rather than memorising.
Checking it
| n | k | p | s | d | formula | output |
|---|---|---|---|---|---|---|
| 224 | 3 | 1 | 1 | 1 | floor(223/1)+1 | 224 |
| 224 | 3 | 0 | 1 | 1 | floor(221/1)+1 | 222 |
| 224 | 3 | 1 | 2 | 1 | floor(223/2)+1 | 112 |
| 224 | 7 | 3 | 2 | 1 | floor(223/2)+1 | 112 |
| 224 | 3 | 2 | 1 | 2 | floor(223/1)+1 | 224 |
| 5 | 3 | 0 | 1 | 1 | floor(2/1)+1 | 3 |
The first row is the “same” convolution: \(k = 3\), \(p = 1\), \(s = 1\) preserves resolution, which is why \(3 \times 3\) with padding 1 is the default building block of almost every modern architecture. The third and fourth rows show a \(7 \times 7\) stride-2 stem and a \(3 \times 3\) stride-2 convolution reaching the same \(112\) — one sees a much larger footprint per output for the same downsampling. The fifth row shows how to keep resolution while dilating: \(k' = 2(3-1)+1 = 5\), so \(p = 2\) restores the size. The last row is the figure below.
I checked the formula against brute-force enumeration of the legal window offsets for every combination of \(n \le 59\), \(k \le 7\), \(p \le 3\), \(s \le 4\), \(d \le 3\) with a non-degenerate output: zero mismatches.
Equivariance, stated correctly
This is the part most explanations get backwards, so here it is precisely.
Let \(T_v\) be the operator that translates a signal by \(v\): \((T_v x)[i] = x[i - v]\). Write \(\mathrm{Conv}_w\) for cross-correlation with kernel \(w\). Then, on an infinite or circular domain,
Shift the input, and the output shifts by the same amount. That is equivariance: the operation commutes with the symmetry. It is not invariance, which would read
If a convolutional layer were translation-invariant, its output would be unchanged by moving the cat to the other side of the photo — which would mean the feature map contains no information about where anything is, after a single layer. That would destroy the network. Equivariance is the useful property: spatial information is preserved, in a form that moves with the content.
I verified this numerically on a 20-sample signal with a 3-tap kernel: shifting the input by three positions shifts the output by exactly three, entry for entry, everywhere the window stays inside the signal.
- Boundaries. On a finite input with zero padding, content shifted near the edge meets zeros it did not meet before. Equivariance holds exactly only in the interior.
- Stride. A stride-\(s\) layer is equivariant only to shifts that are multiples of \(s\). I checked this: with stride 2, shifting the input by 2 shifts the output by exactly 1; shifting the input by 1 produces an output that matches neither the shifted nor the unshifted result. It samples a different sub-grid.
- Downsampling in general. Pooling and strided convolution both sample below the Nyquist rate of the signal they receive, so they alias. This is why real CNNs are measurably less shift-stable than the theory promises.
Where invariance actually comes from
If equivariance is what convolution gives, invariance has to be bought elsewhere, and there are exactly two places it is usually bought.
Local pooling. Max-pooling over a \(2 \times 2\) window returns the same value for any shift that keeps the argmax inside the window. That is a small, approximate, local invariance — and it is the honest description, not “pooling makes the network translation-invariant”.
Global pooling. Global average pooling collapses the whole \(H \times W\) map to one number per channel:
A sum over all positions is unchanged by any circular shift of its argument, so this is exactly invariant (and approximately so at real boundaries). This is where a classifier’s translation invariance really comes from: an equivariant stack, then one global reduction at the end.
Note the corollary. If you replace global pooling with flatten-then-dense, you reintroduce a position-dependent weight for every location and throw the invariance away. That is one reason the flatten-and-dense head fell out of fashion.
Receptive field: how far a unit can see
Define the receptive field \(r_\ell\) of a unit at layer \(\ell\) as the number of input positions (per axis) that can affect it, and the jump \(j_\ell\) as the spacing, in input positions, between adjacent layer-\(\ell\) units. Starting from \(r_0 = 1\), \(j_0 = 1\):
The reasoning: each of the \(k'_\ell\) inputs to a layer-\(\ell\) unit is a layer-\((\ell-1)\) unit, adjacent ones separated by \(j_{\ell-1}\) input positions, so the extremes are \((k'_\ell - 1) j_{\ell-1}\) apart, and each carries its own \(r_{\ell-1}\)-wide field.
For a stack of \(L\) layers of \(3 \times 3\), stride 1, no dilation, \(j\) stays 1 and the recursion collapses to
Three layers see \(7\) input pixels per axis; the figure below traces exactly that.
A stack with downsampling in it
Now put a pooling layer in the middle: \(3 \times 3\) convolution, \(3 \times 3\) convolution, \(2 \times 2\) max-pool with stride 2, \(3 \times 3\) convolution. Running the recursion:
| layer | k | s | r after | j after |
|---|---|---|---|---|
| conv 3, s1 | 3 | 1 | 1 + 2(1) = 3 | 1 |
| conv 3, s1 | 3 | 1 | 3 + 2(1) = 5 | 1 |
| pool 2, s2 | 2 | 2 | 5 + 1(1) = 6 | 2 |
| conv 3, s1 | 3 | 1 | 6 + 2(2) = 10 | 2 |
The final convolution adds 4 to the receptive field rather than 2, because after the stride-2 pool each of its inputs is 2 input pixels apart. That is the real reason downsampling is in the architecture: it makes every subsequent layer count double. Continue the pattern (two more \(3 \times 3\) convolutions and a second pool) and you get \(r = 10, 14, 16\) with \(j = 4\).
Pooling, and why it is going out of fashion
Max-pooling with a \(2 \times 2\) window and stride 2 takes the largest activation in each disjoint block, halving both spatial axes: \(224 \to 112 \to 56 \to 28 \to 14 \to 7\) across five stages. What it buys:
- Downsampling, which cuts the compute of every subsequent layer by four.
- Faster receptive-field growth, via the jump doubling shown above.
- Small local shift tolerance, since the max is unchanged by shifts that keep the winner in the window.
- No parameters at all.
What it costs, and why the field has drifted away from it: the operation is fixed, not learned, and it discards which of the four positions won. A stride-2 convolution achieves the same downsampling with a learned combination — it is strictly more general, since the right weights make it approximate an average pool — at the price of \(C_{\text{out}}(C_{\text{in}} k^2 + 1)\) extra parameters and a real increase in compute. Modern designs commonly use stride-2 convolutions between stages and keep a single global average pooling at the head. Both approaches alias, so neither is a free source of shift-invariance.
What CNNs do not give you
Every prior that helps on the data it fits hurts on the data it does not. Four honest limitations.
No rotation equivariance. Convolution commutes with translation, and with nothing else. A concrete demonstration: take the Sobel-x kernel \([[1,0,-1],[2,0,-2],[1,0,-1]]\) and a \(6 \times 6\) image containing a single vertical line of ones. The valid convolution gives a response map whose rows are all \([0, -4, 0, 4]\) — a clear detection. Rotate the image by 90 degrees, so the line is horizontal, and the same kernel returns all zeros. Rotating the input did not rotate the feature map; it deleted the feature. A CNN handles rotation only by learning separate filters for separate orientations — paid for in parameters and in data — or by building the symmetry in explicitly, which is what group-equivariant convolutions do.
No scale equivariance. The kernel has a fixed pixel size. An object at twice the scale is a different pattern to a \(3 \times 3\) filter. The usual answers are architectural (feature pyramids, multiple parallel kernel sizes) or data-driven (scale augmentation), never free.
A receptive field that grows only linearly. At fixed resolution, \(r_L = 2L + 1\). To make one unit see across a 224-pixel image you would need \(L = 112\) stacked \(3 \times 3\) layers, since \(2(112) + 1 = 225 \ge 224\). The two escape routes both cost something. Downsampling makes growth geometric but throws away resolution — a \(16 \times 16\) receptive field that arrived via two stride-2 stages cannot recover fine detail it has already discarded. Dilation grows the field geometrically at full resolution — a stack with dilations \(1, 2, 4, 8\) reaches \(r = 1 + 2(1 + 2 + 4 + 8) = 31\) where four undilated layers would reach 9 — but the taps become sparse, and consecutive dilated layers sample a grid that can miss what falls between the taps.
A locality prior that is sometimes simply wrong. Apply one fixed random permutation to the pixels of every image in a dataset. A dense network is entirely unaffected: permuting the inputs permutes its first-layer weights and nothing else changes. A convolutional network is wrecked, because its neighbourhoods no longer contain neighbours. The prior is not a property of the architecture; it is a claim about the data, and it pays only when the claim is true. On inputs with no meaningful spatial ordering — most tabular data, arbitrary feature vectors — the claim is false and the constraint only removes capacity.
Forward: the constraint attention removes
Of the two constraints, weight sharing is the one that keeps paying — the same detector applied everywhere is a genuine statement about images. Local connectivity is the one with a hard ceiling: it is why the receptive field grows by a fixed two pixels per layer, and why relating two distant parts of an image requires depth you would rather spend elsewhere.
Self-attention lifts exactly that constraint. It keeps a form of weight sharing — the same query, key, and value projections are applied at every position — but replaces the fixed \(k \times k\) neighbourhood with a weighting over all positions, computed from the content itself. The receptive field is the whole input at the first layer. What that costs, and what has to be added back to replace the locality prior it discards, is the subject of Book I.
✅ Key Takeaways
- A convolutional layer is a dense layer with local connectivity and weight sharing imposed. Those two constraints are the prior; everything else is bookkeeping.
- Frameworks implement cross-correlation, not convolution — the kernel is not flipped. It makes no difference to a learned kernel, and it does make a difference when you hand-set one or invoke the convolution theorem.
- Dense from \(224 \times 224 \times 3\) to 1000 units: 150,529,000 parameters. A \(3 \times 3\), 64-filter convolution: 1,792. That is about 84,000 times fewer — but only about 1.7 times less arithmetic. Weight sharing buys sample efficiency and memory, not FLOPs.
- Convolution is equivariant to translation, not invariant: shift the input, the output shifts too. Local pooling buys small approximate invariance; global average pooling buys the real thing.
- Output size is \(\lfloor (n + 2p - k)/s \rfloor + 1\), and it is just a count of the kernel offsets that fit. With dilation, replace \(k\) by \(d(k-1)+1\).
- A kernel spans all input channels, so it is \(k \times k \times C_{\text{in}}\); "64 filters" means 64 of those, giving 64 output channels and \(C_{\text{out}}(C_{\text{in}}k^2 + 1)\) parameters.
- Receptive field: \(r_\ell = r_{\ell-1} + (k'_\ell - 1) j_{\ell-1}\), \(j_\ell = j_{\ell-1} s_\ell\). Stride-1 \(3 \times 3\) stacks give \(r_L = 2L+1\) — 112 layers to span a 224-pixel image.
- Not included: rotation equivariance, scale equivariance, or long-range interaction without depth. The locality prior is a claim about your data, and it is worth nothing if the claim is false.