Published:
Many real-world graphs are embedded in 3D space, molecules, proteins, point clouds, crystal structures. Standard GNNs ignore coordinates and only use connectivity. Geometric GNNs incorporate spatial positions and must respect physical symmetries.
Published:
Graph neural networks are transforming computational drug discovery. Molecules are natural graphs, and GNNs learn molecular representations that predict toxicity, solubility, binding affinity, and synthesis feasibility, tasks that previously required expensive laboratory experiments.
Published:
APPNP decouples feature transformation from propagation. A neural network transforms features first; then Personalized PageRank propagates the result. This enables deep propagation without over-smoothing.
Published:
BERT, GPT, and T5 are all Transformers, but their architectures are fundamentally different. One comparison table clarifies the entire landscape.
Published:
The difference between GPT, BERT, and T5 is largely a masking decision. Learn how causal, padding, and bidirectional masks shape what each token is allowed to see.
Published:
GAT weights a neighbour by a scalar. SheafAN keeps the scalar and adds a learned orthogonal transport matrix alongside it, recovering GAT exactly at d = 1, and turning a model that goes numerically unstable past eight layers into one that runs to sixty-four.
Published:
GNNs fail on heterophilic graphs and they oversmooth with depth. Bodnar et al. show these are one failure with one cause, the graph is implicitly equipped with a trivial sheaf, and prove a hierarchy of exactly what richer sheaves buy you.
Published:
Neural Sheaf Diffusion (Bodnar et al., 2022) learns the sheaf restriction maps from data using a neural network, then performs diffusion with the learned Sheaf Laplacian. This gives a principled, topology-grounded GNN that handles heterophily without heuristic fixes.
Published:
Neural ODEs replace discrete layer-by-layer computation with continuous dynamics governed by a differential equation. Graph Neural ODEs apply this to graph data, treating node embeddings as a dynamical system evolving in continuous time.
Published:
| ChebNet avoids the expensive full eigendecomposition by approximating spectral filters with Chebyshev polynomials, achieving O( | E | ) computation and spatial locality without sacrificing expressiveness. |
Published:
| ChebNet avoids the expensive full eigendecomposition by approximating spectral filters with Chebyshev polynomials, achieving O( | E | ) computation and spatial locality without sacrificing expressiveness. |
Published:
Knowledge graph completion can be solved with shallow KG embeddings (TransE, DistMult, ComplEx) or with structural GNNs (R-GCN, CompGCN). Each approach has different inductive biases and failure modes. Understanding when to use each is the central design decision for KG tasks.
Published:
Traffic prediction is a canonical spatio-temporal graph task: sensors on roads form a fixed graph, and speed/volume measurements evolve over time. GNNs capture spatial correlations between sensors; RNNs or convolutions capture temporal patterns. Together they achieve state-of-the-art traffic forecasting.
Published:
Spatio-temporal GNNs combine spatial message passing with temporal sequence modelling. They are the dominant approach for traffic forecasting, weather prediction, and any task where measurements at sensor nodes evolve over time on a fixed graph.
Published:
| ChebNet avoids the expensive full eigendecomposition by approximating spectral filters with Chebyshev polynomials, achieving O( | E | ) computation and spatial locality without sacrificing expressiveness. |
Published:
Molecules are graphs. Molecular GNNs predict chemical properties from structure. The best models use 3D coordinates and bond angles, not just connectivity.
Published:
The Sheaf Laplacian generalises the graph Laplacian by incorporating per-edge restriction maps. Its spectrum reveals how consistent data is under the sheaf. Sheaf diffusion with this Laplacian generalises GCN to handle heterophilic graphs.
Published:
Knowledge graph completion can be solved with shallow KG embeddings (TransE, DistMult, ComplEx) or with structural GNNs (R-GCN, CompGCN). Each approach has different inductive biases and failure modes. Understanding when to use each is the central design decision for KG tasks.
Published:
EGNN achieves E(n)-equivariance with a simple update rule: positions updated via weighted sums of relative position vectors, features updated via invariant distances. No spherical harmonics needed.
Published:
Equivariance formalises the idea that a function should ‘commute with symmetry transformations.’ A rotation-equivariant model applied to rotated input gives the rotated output, no extra training needed. This is the foundation for geometric deep learning.
Published:
EGNN achieves E(n)-equivariance with a simple update rule: positions updated via weighted sums of relative position vectors, features updated via invariant distances. No spherical harmonics needed.
Published:
A single Transformer block combines attention, residuals, layer norm, and an FFN into one reusable unit. Understanding this block is understanding the Transformer.
Published:
The FFN block holds two-thirds of a Transformer’s parameters and does most of its factual recall. Yet it is almost always overlooked in introductions to attention.
Published:
FoPE rethinks long-context positional encoding from a frequency-domain perspective. Instead of only stretching RoPE heuristically, it explicitly improves attention’s periodic extension so Transformers generalize more gracefully to longer sequences.
Published:
FoPE rethinks long-context positional encoding from a frequency-domain perspective. Instead of only stretching RoPE heuristically, it explicitly improves attention’s periodic extension so Transformers generalize more gracefully to longer sequences.
Published:
The Graph Fourier Transform decomposes a signal on a graph into frequency components using the Laplacian’s eigenvectors. This spectral view is the mathematical foundation behind spectral GNNs like ChebNet and GCN.
Published:
Knowledge graphs encode human knowledge as typed entity-relation triples. GNNs enable structure-aware entity representation, multi-hop reasoning, knowledge base completion, and entity alignment, tasks that shallow embedding methods cannot fully solve.
Published:
The MPNN framework (Gilmer et al., 2017) unifies GCN, GAT, GIN, GraphSAGE, and almost all spatial GNNs under one abstraction: message functions, aggregation, and update. Understanding MPNN means understanding the whole GNN family.
Published:
The MPNN framework (Gilmer et al., 2017) unifies GCN, GAT, GIN, GraphSAGE, and almost all spatial GNNs under one abstraction: message functions, aggregation, and update. Understanding MPNN means understanding the whole GNN family.
Published:
In Transformers, depth = expressiveness. In GNNs, depth = both expressiveness AND over-smoothing. The optimal GNN depth is rarely more than 3-4 layers, fundamentally different from the hundreds of layers in modern LLMs.
Published:
Graph classification is the task of predicting a label for an entire graph. It requires composing message passing (node embeddings), readout (graph embedding), and a classifier, and all three choices interact to determine model expressiveness.
Published:
The 1-WL graph isomorphism test provides the exact upper bound on message-passing GNN expressivity. GIN achieves this bound. Any pair of graphs that 1-WL cannot distinguish cannot be distinguished by any MPNN.
Published:
The MPNN framework (Gilmer et al., 2017) unifies GCN, GAT, GIN, GraphSAGE, and almost all spatial GNNs under one abstraction: message functions, aggregation, and update. Understanding MPNN means understanding the whole GNN family.
Published:
Message-passing GNNs are permutation-equivariant by design, they cannot assign unique positions to nodes. Without positional encodings, symmetric nodes are indistinguishable. Here is why that matters and how to fix it.
Published:
In Transformers, depth = expressiveness. In GNNs, depth = both expressiveness AND over-smoothing. The optimal GNN depth is rarely more than 3-4 layers, fundamentally different from the hundreds of layers in modern LLMs.
Published:
Oversmoothing and oversquashing are both problems with deep GNNs, but they affect different nodes, have different causes, and require different fixes. Confusing them leads to applying the wrong solution.
Published:
Stack enough GNN layers and all node embeddings converge to the same vector, making the model useless. Oversmoothing is not a training problem; it is a mathematical inevitability of iterated averaging.
Published:
Most GNNs assume nearby nodes are similar, the homophily assumption. When this breaks (heterophilic graphs), standard message passing hurts performance. Understanding this distinction is essential for modern GNN design.
Published:
Certain graph structures are invisible to message-passing GNNs, not because of bad training, but because of fundamental mathematical limits. Two structurally distinct graphs can produce identical embeddings in any MPNN.
Published:
BERT, GPT, and T5 are all Transformers, but their architectures are fundamentally different. One comparison table clarifies the entire landscape.
Published:
The difference between GPT, BERT, and T5 is largely a masking decision. Learn how causal, padding, and bidirectional masks shape what each token is allowed to see.
Published:
Shortest-path distances between nodes can be encoded as attention biases or node features, directly informing the model about graph proximity without requiring message passing.
Published:
Graphormer encodes graph structure directly into Transformer attention via three biases: node centrality, spatial encoding (shortest paths), and edge encoding. It won the OGB-LSC 2021 competition on molecular property prediction.
Published:
Sheaf GNNs are the principled solution to heterophily: by learning per-edge maps that transform features before comparison, they can perform diffusion that converges within classes and diverges across classes, the exact opposite of standard GCN’s collapse.
Published:
HAN combines meta-path decomposition with two levels of attention: node-level attention weights neighbours along a meta-path, and semantic-level attention weights different meta-paths. This lets the model learn which relationships matter most for a given task.
Published:
Most real-world graphs are heterogeneous, they contain multiple node types (users, items, tags) and edge types (clicks, rates, authors). Standard GNNs treat all nodes and edges identically, making them blind to this type structure.
Published:
Molecules are graphs. Molecular GNNs predict chemical properties from structure. The best models use 3D coordinates and bond angles, not just connectivity.
Published:
Oversquashing occurs when exponentially many node features must be compressed into a fixed-size embedding through a bottleneck edge. It is the reason GNNs struggle with long-range dependencies, not just oversmoothing.
Published:
YaRN combines NTK scaling for high-frequency dimensions with linear interpolation for low-frequency ones, plus a temperature correction, achieving better long-context performance with minimal fine-tuning.
Published:
Mean and sum readout treat all nodes equally. Attention readout learns which nodes matter most for a given task. Set2Set goes further, it uses an LSTM to iteratively query the node set, producing richer graph representations than single-pass pooling.
Published:
Laplacian eigenvectors are only defined up to sign: if u is an eigenvector, so is -u. This seemingly minor issue creates a fundamental problem for learning with LapPE. Here is the problem, its consequences, and how SignNet solves it.
Published:
The k smallest eigenvectors of the graph Laplacian form a natural positional embedding space, the graph’s own coordinate system. They capture global structure, symmetry, and community membership.
Published:
The k smallest eigenvectors of the graph Laplacian form a natural positional embedding space, the graph’s own coordinate system. They capture global structure, symmetry, and community membership.
Published:
Stack enough GNN layers and all node embeddings converge to the same vector, making the model useless. Oversmoothing is not a training problem; it is a mathematical inevitability of iterated averaging.
Published:
The Graph Fourier Transform decomposes a signal on a graph into frequency components using the Laplacian’s eigenvectors. This spectral view is the mathematical foundation behind spectral GNNs like ChebNet and GCN.
Published:
Recommendation is naturally a graph problem: users and items are nodes, interactions are edges. GNNs on bipartite user-item graphs capture higher-order collaborative filtering signals, friends of friends liked this, that matrix factorisation cannot represent.
Published:
LongRoPE (Microsoft, 2024) pushes RoPE-based context to 2M tokens by searching for optimal per-dimension rescaling factors, far outperforming NTK or YaRN at extreme lengths.
Published:
Tensor Field Networks (TFN) were the first architecture to achieve SE(3) equivariance using spherical harmonics and Clebsch-Gordan tensor products. They laid the theoretical foundation for NequIP and MACE, the current state-of-the-art in equivariant molecular force fields.
Published:
The FFN block holds two-thirds of a Transformer’s parameters and does most of its factual recall. Yet it is almost always overlooked in introductions to attention.
Published:
The MPNN framework (Gilmer et al., 2017) unifies GCN, GAT, GIN, GraphSAGE, and almost all spatial GNNs under one abstraction: message functions, aggregation, and update. Understanding MPNN means understanding the whole GNN family.
Published:
LongRoPE (Microsoft, 2024) pushes RoPE-based context to 2M tokens by searching for optimal per-dimension rescaling factors, far outperforming NTK or YaRN at extreme lengths.
Published:
GNNs fail on heterophilic graphs and they oversmooth with depth. Bodnar et al. show these are one failure with one cause, the graph is implicitly equipped with a trivial sheaf, and prove a hierarchy of exactly what richer sheaves buy you.
Published:
Neural Sheaf Diffusion (Bodnar et al., 2022) learns the sheaf restriction maps from data using a neural network, then performs diffusion with the learned Sheaf Laplacian. This gives a principled, topology-grounded GNN that handles heterophily without heuristic fixes.
Published:
NTK-Aware Scaling extends the context window of RoPE-based models by rescaling frequencies using Neural Tangent Kernel theory, with no fine-tuning required.
Published:
Tensor Field Networks (TFN) were the first architecture to achieve SE(3) equivariance using spherical harmonics and Clebsch-Gordan tensor products. They laid the theoretical foundation for NequIP and MACE, the current state-of-the-art in equivariant molecular force fields.
Published:
GNNs fail on heterophilic graphs and they oversmooth with depth. Bodnar et al. show these are one failure with one cause, the graph is implicitly equipped with a trivial sheaf, and prove a hierarchy of exactly what richer sheaves buy you.
Published:
GAT weights a neighbour by a scalar. SheafAN keeps the scalar and adds a learned orthogonal transport matrix alongside it, recovering GAT exactly at d = 1, and turning a model that goes numerically unstable past eight layers into one that runs to sixty-four.
Published:
Graphormer encodes graph structure directly into Transformer attention via three biases: node centrality, spatial encoding (shortest paths), and edge encoding. It won the OGB-LSC 2021 competition on molecular property prediction.
Published:
Polynomial Neural Sheaf Diffusion (PNSD) replaces the fixed diffusion operator (I - Δ_F) with a learnable polynomial of the Sheaf Laplacian. This gives the model spectral flexibility, it can learn to amplify or suppress different frequency components of the sheaf signal.
Published:
APPNP decouples feature transformation from propagation. A neural network transforms features first; then Personalized PageRank propagates the result. This enables deep propagation without over-smoothing.
Published:
Recommendation is naturally a graph problem: users and items are nodes, interactions are edges. GNNs on bipartite user-item graphs capture higher-order collaborative filtering signals, friends of friends liked this, that matrix factorisation cannot represent.
Published:
Layer norm is not optional plumbing. It determines training stability, gradient flow, and whether deep Transformers converge at all. Pre-LN vs Post-LN is not a detail, it changes training dynamics fundamentally.
Published:
Layer norm is not optional plumbing. It determines training stability, gradient flow, and whether deep Transformers converge at all. Pre-LN vs Post-LN is not a detail, it changes training dynamics fundamentally.
Published:
Q, K, and V are not arbitrary labels. They map precisely onto search queries, database labels, and retrieved content, a framework you already understand.
Published:
Molecules are graphs. Molecular GNNs predict chemical properties from structure. The best models use 3D coordinates and bond angles, not just connectivity.
Published:
Graph neural networks are transforming computational drug discovery. Molecules are natural graphs, and GNNs learn molecular representations that predict toxicity, solubility, binding affinity, and synthesis feasibility, tasks that previously required expensive laboratory experiments.
Published:
Knowledge graph completion can be solved with shallow KG embeddings (TransE, DistMult, ComplEx) or with structural GNNs (R-GCN, CompGCN). Each approach has different inductive biases and failure modes. Understanding when to use each is the central design decision for KG tasks.
Published:
R-GCN extends GCN to multi-relational graphs by learning a separate weight matrix for each relation type. It handles knowledge graphs with typed edges and powers both entity classification and link prediction tasks.
Published:
Random walk positional encodings encode each node’s structural context by computing the probability of returning to it from itself in k steps, a computationally efficient alternative to Laplacian eigenvectors with no sign ambiguity.
Published:
FoPE rethinks long-context positional encoding from a frequency-domain perspective. Instead of only stretching RoPE heuristically, it explicitly improves attention’s periodic extension so Transformers generalize more gracefully to longer sequences.
Published:
Position Interpolation rescales positions before applying RoPE so a model trained on short contexts can be adapted to longer ones with surprisingly little fine-tuning. It became the reference baseline for long-context RoPE extension.
Published:
XPos modifies RoPE with a multiplicative decay that keeps relative rotations while stabilising magnitude at long distance. It is one of the cleanest attempts to make rotary embeddings extrapolate better.
Published:
This paper does two things at once: it explains what RoPE is really doing inside a trained LLM, and it proposes p-RoPE, a partial rotary variant that drops the lowest frequencies to preserve stronger semantic channels.
Published:
LongRoPE (Microsoft, 2024) pushes RoPE-based context to 2M tokens by searching for optimal per-dimension rescaling factors, far outperforming NTK or YaRN at extreme lengths.
Published:
YaRN combines NTK scaling for high-frequency dimensions with linear interpolation for low-frequency ones, plus a temperature correction, achieving better long-context performance with minimal fine-tuning.
Published:
NTK-Aware Scaling extends the context window of RoPE-based models by rescaling frequencies using Neural Tangent Kernel theory, with no fine-tuning required.
Published:
SE(3)-Transformers extend self-attention to 3D point clouds and molecular graphs while maintaining SE(3) equivariance. Attention weights are learned between node pairs; values are equivariant features built from spherical harmonics.
Published:
SGC removes all nonlinearities between GCN layers and collapses the entire propagation into a single pre-computed matrix power. Surprisingly, it matches GCN on most benchmarks, revealing that nonlinearities between layers may be unnecessary.
Published:
If a sheaf neural network learns its geometry from data, it can learn the wrong geometry and have no way of knowing. Treating the sheaf Laplacian as a latent random variable fixes that, but requires a reparameterisable distribution on SO(n) with a tractable density, which did not exist.
Published:
Tensor Field Networks (TFN) were the first architecture to achieve SE(3) equivariance using spherical harmonics and Clebsch-Gordan tensor products. They laid the theoretical foundation for NequIP and MACE, the current state-of-the-art in equivariant molecular force fields.
Published:
Shortest-path distances between nodes can be encoded as attention biases or node features, directly informing the model about graph proximity without requiring message passing.
Published:
Traffic prediction is a canonical spatio-temporal graph task: sensors on roads form a fixed graph, and speed/volume measurements evolve over time. GNNs capture spatial correlations between sensors; RNNs or convolutions capture temporal patterns. Together they achieve state-of-the-art traffic forecasting.
Published:
Spatio-temporal GNNs combine spatial message passing with temporal sequence modelling. They are the dominant approach for traffic forecasting, weather prediction, and any task where measurements at sensor nodes evolve over time on a fixed graph.
Published:
Molecules are graphs. Molecular GNNs predict chemical properties from structure. The best models use 3D coordinates and bond angles, not just connectivity.
Published:
GAT weights a neighbour by a scalar. SheafAN keeps the scalar and adds a learned orthogonal transport matrix alongside it, recovering GAT exactly at d = 1, and turning a model that goes numerically unstable past eight layers into one that runs to sixty-four.
Published:
Laplacian eigenvectors are only defined up to sign: if u is an eigenvector, so is -u. This seemingly minor issue creates a fundamental problem for learning with LapPE. Here is the problem, its consequences, and how SignNet solves it.
Published:
The FFN block holds two-thirds of a Transformer’s parameters and does most of its factual recall. Yet it is almost always overlooked in introductions to attention.
Published:
BERT, GPT, and T5 are all Transformers, but their architectures are fundamentally different. One comparison table clarifies the entire landscape.
Published:
This book has two halves. One computes topological summaries of data and feeds them to a model. The other makes the topology itself the domain the network runs on. This guide sets out both, and the pipeline that connects them.
Published:
Tensor Field Networks (TFN) were the first architecture to achieve SE(3) equivariance using spherical harmonics and Clebsch-Gordan tensor products. They laid the theoretical foundation for NequIP and MACE, the current state-of-the-art in equivariant molecular force fields.
Published:
TGN (Temporal Graph Network) is the leading framework for continuous-time dynamic graphs. It maintains a per-node memory that is updated upon each interaction, enabling efficient inductive link prediction on event streams.
Published:
Most knowledge graphs treat facts as timeless, but facts change. Barack Obama was president from 2009 to 2017. Temporal Knowledge Graphs add timestamps to triples, requiring models to reason about what was true when.
Published:
Knowledge graph completion can be solved with shallow KG embeddings (TransE, DistMult, ComplEx) or with structural GNNs (R-GCN, CompGCN). Each approach has different inductive biases and failure modes. Understanding when to use each is the central design decision for KG tasks.
Published:
Certain graph structures are invisible to message-passing GNNs, not because of bad training, but because of fundamental mathematical limits. Two structurally distinct graphs can produce identical embeddings in any MPNN.
Published:
The 1-WL graph isomorphism test provides the exact upper bound on message-passing GNN expressivity. GIN achieves this bound. Any pair of graphs that 1-WL cannot distinguish cannot be distinguished by any MPNN.
Published:
Knowledge graphs encode human knowledge as typed entity-relation triples. GNNs enable structure-aware entity representation, multi-hop reasoning, knowledge base completion, and entity alignment, tasks that shallow embedding methods cannot fully solve.
Published:
XPos modifies RoPE with a multiplicative decay that keeps relative rotations while stabilising magnitude at long distance. It is one of the cleanest attempts to make rotary embeddings extrapolate better.
Published:
The 1-WL graph isomorphism test provides the exact upper bound on message-passing GNN expressivity. GIN achieves this bound. Any pair of graphs that 1-WL cannot distinguish cannot be distinguished by any MPNN.
Published:
YaRN combines NTK scaling for high-frequency dimensions with linear interpolation for low-frequency ones, plus a temperature correction, achieving better long-context performance with minimal fine-tuning.
Published:
The FFN block holds two-thirds of a Transformer’s parameters and does most of its factual recall. Yet it is almost always overlooked in introductions to attention.
Published:
Once ReLU became the default, researchers started asking a better question: can we keep the easy optimization while making the activation smoother, softer, and more expressive? This chapter covers the modern answers.
Published:
Activation functions are the reason neural networks can model curved decision boundaries instead of collapsing into one giant linear map. This chapter builds the intuition first, then walks through the classical functions that shaped deep learning.
Published:
Training is one loop: measure the loss, ask backpropagation which way is downhill, take a small step. This chapter derives exactly how small that step has to be, why the answer is 2/a for a quadratic, and why reverse-mode differentiation gets you every gradient for roughly the price of one forward pass.
Published:
Before understanding GNNs, you need to understand how graphs are represented mathematically. The adjacency matrix is the foundation, a simple grid that tells you which nodes are connected.
Published:
To predict a property of an entire graph, node embeddings must be aggregated into a single vector. The choice of global pooling, mean, sum, or max, is not arbitrary: each has distinct expressive power and fits different tasks.
Published:
Most GNNs assume nearby nodes are similar, the homophily assumption. When this breaks (heterophilic graphs), standard message passing hurts performance. Understanding this distinction is essential for modern GNN design.
Published:
An ML interview that asks you to invert a binary tree is not testing your tree knowledge, it is testing whether you can state a cost, defend it, and write a loop whose boundaries are right the first time. Here is the eight-post map of what that requires.
Published:
ALiBi skips traditional positional embeddings entirely and just subtracts a distance penalty from attention scores. Zero extra parameters, excellent extrapolation. Press et al., 2022.
Published:
A linked list has O(1) insertion and loses to an array anyway. Appending is O(1) but a single append can copy a million elements. Hash lookup is O(1) until every key lands in one bucket. All three claims are true, with the qualifier that got dropped.
Published:
Big-O is an upper bound on growth, not a running time, and saying an algorithm is O(n²) is compatible with it being O(n). Here is what each symbol actually claims, how to read a cost off loops and recursions, and why the asymptotically better algorithm sometimes loses on real hardware.
Published:
A single Transformer block combines attention, residuals, layer norm, and an FFN into one reusable unit. Understanding this block is understanding the Transformer.
Published:
BERT, GPT, and T5 are all Transformers, but their architectures are fundamentally different. One comparison table clarifies the entire landscape.
Published:
A self-contained guide to the Transformer, the engine behind GPT, BERT, and modern AI. Learn how attention replaces recurrence and why every major AI system uses it.
Published:
A linked list has O(1) insertion and loses to an array anyway. Appending is O(1) but a single append can copy a million elements. Hash lookup is O(1) until every key lands in one bucket. All three claims are true, with the qualifier that got dropped.
Published:
Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first, which is faster, shorter, and free of race conditions.
Published:
Message passing on a sparse graph does asymptotically less work than attention over every pair. It is still the slower one to train. This chapter is about why the architecture that wins is the one your hardware happens to like.
Published:
Write self-attention in the message-passing template and you get the Graph Attention Network equations with one substitution: the neighbourhood becomes the whole input. The two architectures are not analogous, they are the same operator on different graphs.
Published:
GAPE is a drop-in RoPE augmentation that adds content-aware attention logit biases: a query-gate suppresses irrelevant distant context while a key-gate preserves salient distant tokens. Provably sharper attention and improved long-context robustness, no architecture changes needed.
Published:
A single Transformer block combines attention, residuals, layer norm, and an FFN into one reusable unit. Understanding this block is understanding the Transformer.
Published:
Cross-attention lets one sequence query information from a completely different sequence. It is the bridge between encoder and decoder, and the core of multimodal AI.
Published:
The difference between GPT, BERT, and T5 is largely a masking decision. Learn how causal, padding, and bidirectional masks shape what each token is allowed to see.
Published:
Q, K, and V are not arbitrary labels. They map precisely onto search queries, database labels, and retrieved content, a framework you already understand.
Published:
Dividing by √d_k is not just a trick, it prevents softmax from saturating and dying in high-dimensional spaces. Here’s the math and the intuition.
Published:
One attention head sees one relationship. Multiple heads running in parallel let the model capture syntax, semantics, and coreference simultaneously, here’s how.
Published:
Self-attention is the core of every Transformer. Learn how Query, Key, and Value vectors let every token directly attend to every other, and why that matters.
Published:
GAT weights a neighbour by a scalar. SheafAN keeps the scalar and adds a learned orthogonal transport matrix alongside it, recovering GAT exactly at d = 1, and turning a model that goes numerically unstable past eight layers into one that runs to sixty-four.
Published:
SE(3)-Transformers extend self-attention to 3D point clouds and molecular graphs while maintaining SE(3) equivariance. Attention weights are learned between node pairs; values are equivariant features built from spherical harmonics.
Published:
HAN combines meta-path decomposition with two levels of attention: node-level attention weights neighbours along a meta-path, and semantic-level attention weights different meta-paths. This lets the model learn which relationships matter most for a given task.
Published:
Graph Transformers replace or augment local message passing with full pairwise attention, every node attends to every other node. This solves long-range dependencies and over-squashing at the cost of O(N²) computation.
Published:
GCN assigns the same (degree-based) weight to every neighbour. GAT learns which neighbours actually matter, using attention coefficients on edges. More expressive, more interpretable.
Published:
ALiBi skips traditional positional embeddings entirely and just subtracts a distance penalty from attention scores. Zero extra parameters, excellent extrapolation. Press et al., 2022.
Published:
Mean and sum readout treat all nodes equally. Attention readout learns which nodes matter most for a given task. Set2Set goes further, it uses an LSTM to iteratively query the node set, producing richer graph representations than single-pass pooling.
Published:
Treating the derivative as a slope stops working the moment there is more than one input. Treating it as the best linear approximation keeps working forever, and makes backpropagation an obvious consequence of the chain rule rather than an algorithm to memorise.
Published:
Most of the bits in a photograph encode texture no one can see. Latent diffusion throws them away first with an autoencoder, then runs the entire diffusion process in a space roughly forty-eight times smaller, which is how Stable Diffusion fits on a consumer GPU.
Published:
Probability is not defined as a long-run frequency or a degree of belief. It is defined as a function on sets that obeys three rules, and the third rule, countable additivity, is the one doing all the work.
Published:
Treating the derivative as a slope stops working the moment there is more than one input. Treating it as the best linear approximation keeps working forever, and makes backpropagation an obvious consequence of the chain rule rather than an algorithm to memorise.
Published:
Training is one loop: measure the loss, ask backpropagation which way is downhill, take a small step. This chapter derives exactly how small that step has to be, why the answer is 2/a for a quadratic, and why reverse-mode differentiation gets you every gradient for roughly the price of one forward pass.
Published:
A decision tree chops feature space into axis-aligned boxes and predicts one number per box, which explains why it needs no feature scaling, why it approximates a diagonal boundary as a staircase, and why it cannot extrapolate a single step beyond the training range.
Published:
A test catches 99% of a disease and false-alarms only 5% of the time. You test positive. The chance you are ill is one in six. Here is the computation, and why almost everyone guesses ten times too high.
Published:
A matrix is not a grid of numbers, it is a map. Once you read it that way, rank, column space, null space and rank–nullity stop being definitions to memorise and become one geometric statement about what the map keeps and what it destroys.
Published:
Layer norm is not optional plumbing. It determines training stability, gradient flow, and whether deep Transformers converge at all. Pre-LN vs Post-LN is not a detail, it changes training dynamics fundamentally.
Published:
A test catches 99% of a disease and false-alarms only 5% of the time. You test positive. The chance you are ill is one in six. Here is the computation, and why almost everyone guesses ten times too high.
Published:
One school says probability is a long-run frequency, so parameters cannot have probabilities. The other says probability is a degree of belief, so they can. Everything else, priors, credible intervals, the whole argument, follows from that one disagreement.
Published:
If a sheaf neural network learns its geometry from data, it can learn the wrong geometry and have no way of knowing. Treating the sheaf Laplacian as a latent random variable fixes that, but requires a reparameterisable distribution on SO(n) with a tractable density, which did not exist.
Published:
Q, K, and V are not arbitrary labels. They map precisely onto search queries, database labels, and retrieved content, a framework you already understand.
Published:
Graph classification is the task of predicting a label for an entire graph. It requires composing message passing (node embeddings), readout (graph embedding), and a classifier, and all three choices interact to determine model expressiveness.
Published:
Dijkstra is greedy: it finalises the closest unvisited vertex and never revisits it. That is valid only because every edge adds weight, put one negative edge in the graph and the algorithm returns a wrong answer without complaining.
Published:
An estimator is a random variable, so it has a mean and a spread. Squared error splits exactly into those two pieces, and once you see the split, it becomes obvious that deliberately biasing an estimator can make it strictly better.
Published:
A model can fail for two opposite reasons: it is too rigid to represent the truth, or so flexible that it chases the noise. Squared error splits cleanly into exactly those two terms plus a floor you cannot beat.
Published:
Big-O is an upper bound on growth, not a running time, and saying an algorithm is O(n²) is compatible with it being O(n). Here is what each symbol actually claims, how to read a cost off loops and recursions, and why the asymptotically better algorithm sometimes loses on real hardware.
Published:
No comparison sort can beat n log n, that is a theorem with a two-line counting proof, not a limit of current cleverness. Counting sort beats it anyway, because it never compares two elements. Also: how to write a binary search whose boundaries are right.
Published:
Recommendation is naturally a graph problem: users and items are nodes, interactions are edges. GNNs on bipartite user-item graphs capture higher-order collaborative filtering signals, friends of friends liked this, that matrix factorisation cannot represent.
Published:
Collaborative filtering represents every user and item as one static vector. Sheaf4Rec replaces each with a vector space, and reports consistent gains on ranking metrics, though the wins come from recall rather than precision, and the headline efficiency claim is hard to reconcile with the timing table.
Published:
CNNs, GNNs, Transformers and sheaf models look like separate inventions. They are the same recipe applied to different domains: identify the symmetry of your data, then build layers that respect it. This book is that recipe, and the arguments that follow from it.
Published:
Counting microstates gives you the Boltzmann distribution, and the Boltzmann distribution gives you softmax, simulated annealing and energy-based models. The partition function is not a bookkeeping constant, it is the object that contains every thermodynamic quantity, and it is intractable for exactly that reason.
Published:
If you cannot resample from the population, resample from your sample instead. That one substitution gives standard errors and intervals for statistics whose sampling distributions nobody can write down, and it fails in ways worth memorising.
Published:
Oversquashing occurs when exponentially many node features must be compressed into a fixed-size embedding through a bottleneck edge. It is the reason GNNs struggle with long-range dependencies, not just oversmoothing.
Published:
Collaborative filtering represents every user and item as one static vector. Sheaf4Rec replaces each with a vector space, and reports consistent gains on ranking metrics, though the wins come from recall rather than precision, and the headline efficiency claim is hard to reconcile with the timing table.
Published:
A Langevin step is a gradient step plus noise. Change the potential to the negative log-density and it samples from that density instead of minimising it, which is the entire sampler behind score-based generative models.
Published:
A binary search tree gives O(log n) search only if it is balanced, and inserting sorted keys, the most natural thing a caller ever does, makes it a linked list with extra pointers. Here is what each traversal is for, and what AVL and red-black trees actually guarantee.
Published:
Almost every mathematical question asked in an ML interview reduces to two things: what a matrix does to space, and how to differentiate a composition. This book covers those two things properly and is honest about what you can safely forget.
Published:
The difference between GPT, BERT, and T5 is largely a masking decision. Learn how causal, padding, and bidirectional masks shape what each token is allowed to see.
Published:
If a sheaf neural network learns its geometry from data, it can learn the wrong geometry and have no way of knowing. Treating the sheaf Laplacian as a latent random variable fixes that, but requires a reparameterisable distribution on SO(n) with a tractable density, which did not exist.
Published:
A probability density can be 2, or 200, and nothing is wrong. Getting clear on what a PDF actually is fixes half the confusion about continuous distributions, and explains the Jacobian term that makes normalising flows work.
Published:
A simplicial complex cannot hold a benzene ring as a single cell, filling the hexagon costs three edges between atoms that share no bond, and this one constraint is what cell and combinatorial complexes exist to remove.
Published:
Standard GNNs assume neighbouring nodes should agree. Sheaf Neural Networks replace that assumption with a learned linear map on every edge, which turns heterophily, oversmoothing, and directional structure into one operator: the sheaf Laplacian.
Published:
A sheaf is a mathematical object from algebraic topology that assigns vector spaces to cells and linear maps between them. On graphs, sheaves assign feature spaces to nodes and edges, with restriction maps encoding how node features relate across edges.
Published:
Standard message passing aggregates neighbour features and averages. On heterophilic graphs (where neighbours often disagree), this is harmful. Cellular sheaves provide a mathematically principled framework to model per-edge relationships between node features, going beyond mere averaging.
Published:
Treating the derivative as a slope stops working the moment there is more than one input. Treating it as the best linear approximation keeps working forever, and makes backpropagation an obvious consequence of the chain rule rather than an algorithm to memorise.
Published:
A probability density can be 2, or 200, and nothing is wrong. Getting clear on what a PDF actually is fixes half the confusion about continuous distributions, and explains the Jacobian term that makes normalising flows work.
Published:
Every sheaf neural network is built on an operator defined in this 2019 paper. It is not a machine learning paper but a programme for lifting spectral graph theory to cellular sheaves, and it is unusually candid about which parts of the lift fail.
Published:
A Python class is a factory for namespaces, not a sealed blueprint. Once you see where an attribute actually lives, on the instance or on the class, the mutable-default trap, the point of self, and the reason composition usually beats inheritance all fall out of the same rule.
Published:
Among all the hyperplanes that separate two classes, one sits furthest from both. Finding it turns out to depend on the data only through inner products, and that single fact is what lets you work in a space you never build.
Published:
Logistic regression is not a squashed linear regression, it is a straight line drawn in log-odds space, which is why one coefficient means one multiplication of the odds, and why perfectly separable data drives that coefficient to infinity.
Published:
A def is executed, not declared: it builds a function object once, evaluating the default arguments there and then. Everything odd about mutable defaults and late-binding closures follows from taking that seriously.
Published:
The central limit theorem is an asymptotic statement about the centre of a distribution. It says nothing about your sample size, and nothing about the tails you actually care about, which is why concentration inequalities exist.
Published:
k-means says a cluster is a ball around a centroid, DBSCAN says it is a connected dense region, and a Gaussian mixture says it is a bump in a density, pick the algorithm and you have already picked the answer.
Published:
DiffPool learns to hierarchically cluster nodes into super-nodes across layers, like a convolutional pyramid for graphs. Unlike flat global pooling, it captures multi-scale graph structure by differentiably assigning nodes to clusters.
Published:
Convolution is not a clever idea someone had about images. It is the only linear map that commutes with translation, a theorem, not a design choice, and one you can verify by exhaustion on a small enough case.
Published:
A dense layer from a 224-by-224 colour image to 1000 units holds 150.5 million weights; a 3-by-3, 64-filter convolution holds 1,792, and the two constraints that buy that factor of 84,000 are exactly the prior that makes it work on images.
Published:
The Sheaf Laplacian generalises the graph Laplacian by incorporating per-edge restriction maps. Its spectrum reveals how consistent data is under the sheaf. Sheaf diffusion with this Laplacian generalises GCN to handle heterophilic graphs.
Published:
Collaborative filtering represents every user and item as one static vector. Sheaf4Rec replaces each with a vector space, and reports consistent gains on ranking metrics, though the wins come from recall rather than precision, and the headline efficiency claim is hard to reconcile with the timing table.
Published:
Recommendation is naturally a graph problem: users and items are nodes, interactions are edges. GNNs on bipartite user-item graphs capture higher-order collaborative filtering signals, friends of friends liked this, that matrix factorisation cannot represent.
pip install Published:
Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed, most of what people reach for a package to solve is one import away.
Published:
A dict trades memory for the ability to skip the search entirely. Everything that follows, why keys must be hashable, why lists cannot be keys, and why 1, 1.0 and True collide, comes from that single trade.
Published:
A simplicial complex cannot hold a benzene ring as a single cell, filling the hexagon costs three edges between atoms that share no bond, and this one constraint is what cell and combinatorial complexes exist to remove.
Published:
Social networks are large sparse graphs with rich node features (user profiles) and heterogeneous edges (friendship, follow, retweet). GNNs predict user behaviour, detect communities, identify influential spreaders, and flag misinformation, tasks with significant real-world impact.
Published:
Oversmoothing and oversquashing are both problems with deep GNNs, but they affect different nodes, have different causes, and require different fixes. Confusing them leads to applying the wrong solution.
Published:
A Python list is a growable array of pointers, and that one implementation fact predicts every complexity in its API: appending is cheap, inserting at the front is not, and in costs a full scan.
Published:
Big-O is an upper bound on growth, not a running time, and saying an algorithm is O(n²) is compatible with it being O(n). Here is what each symbol actually claims, how to read a cost off loops and recursions, and why the asymptotically better algorithm sometimes loses on real hardware.
Published:
An ML interview that asks you to invert a binary tree is not testing your tree knowledge, it is testing whether you can state a cost, defend it, and write a loop whose boundaries are right the first time. Here is the eight-post map of what that requires.
Published:
A comprehension builds the whole result before you touch any of it. Change one pair of brackets to parentheses and nothing is built at all until you ask, the difference, on a million elements, is 40 MB against 432 bytes.
Published:
The central limit theorem is an asymptotic statement about the centre of a distribution. It says nothing about your sample size, and nothing about the tails you actually care about, which is why concentration inequalities exist.
Published:
Convexity buys one enormous guarantee, every local minimum is global, and it says nothing at all about speed. Deep learning throws the guarantee away and still works, and it is worth being precise about how much of that we actually understand.
Published:
A test catches 99% of a disease and false-alarms only 5% of the time. You test positive. The chance you are ill is one in six. Here is the computation, and why almost everyone guesses ten times too high.
Published:
Conditioning a diffusion model on a prompt is easy; making it actually obey the prompt is not. Classifier-free guidance fixes that by training one network to do two jobs and then extrapolating between its own answers.
Published:
A 95% confidence interval does not say the parameter is 95% likely to be inside it. It says the recipe that produced the interval succeeds 95% of the time. That distinction is the single most-failed question in statistics interviews.
Published:
A sheaf gives every node a matrix-valued say in how its neighbours reach it, but not in whether they do. Set a node’s restriction maps to zero to stop it listening and you also stop it speaking. Fixing that needs sheaves on directed graphs, and the fix costs the Laplacian its positive semi-definiteness.
Published:
One school says probability is a long-run frequency, so parameters cannot have probabilities. The other says probability is a degree of belief, so they can. Everything else, priors, credible intervals, the whole argument, follows from that one disagreement.
Published:
Sheaves with orthogonal restriction maps define a connection on the graph, a parallel transport structure over edges. This connects sheaf GNNs to differential geometry and enables equivariant processing of data with local coordinate frames at each node.
Published:
Laplacian eigenvector positional encodings tell a node where it sits in the graph, but the graph Laplacian only knows adjacency, so two structurally identical nodes get identical encodings no matter how different their features are. Swap in the sheaf Laplacian and the features enter the spectrum.
Published:
Neural Sheaf Diffusion learns the restriction maps by gradient descent. Conn-NSD computes them once, before training, by assuming the data lies on a manifold and optimally aligning neighbouring tangent spaces, matching the learned models on small graphs at roughly half the cost per epoch.
Published:
Energy is conserved because the laws of physics do not care what time it is. That single sentence is Noether’s theorem, and its machine learning descendant is the reason an equivariant network needs less data than one that must learn the symmetry from examples.
Published:
Better ODE solvers bottom out around ten evaluations because the trajectory is genuinely curved. To go lower you have to change the model, either teach a student to take two teacher steps at once, or train a network that jumps to the end of the trajectory from anywhere on it.
Published:
LongRoPE (Microsoft, 2024) pushes RoPE-based context to 2M tokens by searching for optimal per-dimension rescaling factors, far outperforming NTK or YaRN at extreme lengths.
Published:
YaRN combines NTK scaling for high-frequency dimensions with linear interpolation for low-frequency ones, plus a temperature correction, achieving better long-context performance with minimal fine-tuning.
Published:
NTK-Aware Scaling extends the context window of RoPE-based models by rescaling frequencies using Neural Tangent Kernel theory, with no fine-tuning required.
with Is Not Optional Published:
A file object that falls out of scope without with does get closed eventually, but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.
Published:
Continuous normalising flows were elegant and nearly untrainable, every gradient step needed an ODE solve and a divergence estimate. Flow matching removes both by regressing a velocity field against a target you can write down in closed form, one example at a time.
Published:
Neural ODEs replace discrete layer-by-layer computation with continuous dynamics governed by a differential equation. Graph Neural ODEs apply this to graph data, treating node embeddings as a dynamical system evolving in continuous time.
Published:
TGN (Temporal Graph Network) is the leading framework for continuous-time dynamic graphs. It maintains a per-node memory that is updated upon each interaction, enabling efficient inductive link prediction on event streams.
Published:
Most GNN research assumes a fixed graph. Real graphs evolve: edges appear and disappear, node features drift, new nodes arrive. Dynamic graph learning addresses how to model and predict on graphs whose structure changes over time.
is Is Not ==, and for Has an else Published:
== asks whether two objects are equal; is asks whether they are the same object. CPython’s caching of small integers makes the two agree just often enough to teach you the wrong lesson.
Published:
The central limit theorem is an asymptotic statement about the centre of a distribution. It says nothing about your sample size, and nothing about the tails you actually care about, which is why concentration inequalities exist.
Published:
Stack enough GNN layers and all node embeddings converge to the same vector, making the model useless. Oversmoothing is not a training problem; it is a mathematical inevitability of iterated averaging.
Published:
Convexity buys one enormous guarantee, every local minimum is global, and it says nothing at all about speed. Deep learning throws the guarantee away and still works, and it is worth being precise about how much of that we actually understand.
Published:
Convolution is not a clever idea someone had about images. It is the only linear map that commutes with translation, a theorem, not a design choice, and one you can verify by exhaustion on a small enough case.
Published:
A dense layer from a 224-by-224 colour image to 1000 units holds 150.5 million weights; a 3-by-3, 64-filter convolution holds 1,792, and the two constraints that buy that factor of 84,000 are exactly the prior that makes it work on images.
Published:
GCN (Kipf & Welling, 2016) is the ‘hello world’ of GNNs. It simplifies spectral graph convolution into a single elegant layer: normalised neighbourhood averaging with a learned linear transformation.
Published:
A sheaf gives every node a matrix-valued say in how its neighbours reach it, but not in whether they do. Set a node’s restriction maps to zero to stop it listening and you also stop it speaking. Fixing that needs sheaves on directed graphs, and the fix costs the Laplacian its positive semi-definiteness.
Published:
EGNN achieves E(n)-equivariance with a simple update rule: positions updated via weighted sums of relative position vectors, features updated via invariant distances. No spherical harmonics needed.
Published:
Many real-world graphs are embedded in 3D space, molecules, proteins, point clouds, crystal structures. Standard GNNs ignore coordinates and only use connectivity. Geometric GNNs incorporate spatial positions and must respect physical symmetries.
Published:
Expectation adds up no matter how tangled the dependence. Variance does not, and the correction term, covariance, is where most of the interesting behaviour of ensembles, portfolios and minibatch gradients lives.
Published:
The forward process looks like the trivial half of diffusion, but every term in it is load-bearing. Drop the shrink factor and the variance diverges; pick the wrong schedule and a third of your timesteps are spent denoising static.
Published:
L1 regularisation produces exact zeros and L2 does not. The reason is not statistical, it is geometric: the L1 unit ball has corners on the axes, and corners are what optimisation solutions stick to.
Published:
Probability is not defined as a long-run frequency or a degree of belief. It is defined as a function on sets that obeys three rules, and the third rule, countable additivity, is the one doing all the work.
Published:
Expectation adds up no matter how tangled the dependence. Variance does not, and the correction term, covariance, is where most of the interesting behaviour of ensembles, portfolios and minibatch gradients lives.
Published:
Python is not interpreted line by line, and python is not the language. Knowing what CPython compiles your file into, and where it puts the packages you install, explains most of the confusion beginners hit in their first week.
Published:
One school says probability is a long-run frequency, so parameters cannot have probabilities. The other says probability is a degree of belief, so they can. Everything else, priors, credible intervals, the whole argument, follows from that one disagreement.
Published:
Cross-attention lets one sequence query information from a completely different sequence. It is the bridge between encoder and decoder, and the core of multimodal AI.
Published:
Minimising cross-entropy is minimising a KL divergence plus a constant you cannot change. Once that clicks, maximum likelihood, variational inference and the mode-collapse of a VAE all turn out to be the same argument run in different directions.
Published:
Logistic regression is not a squashed linear regression, it is a straight line drawn in log-odds space, which is why one coefficient means one multiplication of the odds, and why perfectly separable data drives that coefficient to infinity.
Published:
If you cannot resample from the population, resample from your sample instead. That one substitution gives standard errors and intervals for statistics whose sampling distributions nobody can write down, and it fails in ways worth memorising.
Published:
A model can fail for two opposite reasons: it is too rigid to represent the truth, or so flexible that it chases the noise. Squared error splits cleanly into exactly those two terms plus a floor you cannot beat.
Published:
Curvature at a point is one over the radius of the circle that best hugs the curve there. Push that idea up to surfaces and Gauss’s Theorema Egregium falls out: some curvature is visible from inside the surface, which is why no map projection can ever get distances right.
Published:
The Jacobian tells you how a map distorts volume, which is exactly the term normalising flows have to pay. The Hessian tells you the shape of the valley you are descending. Both are indispensable to reason with and, at a billion parameters, hopeless to form.
Published:
Z-SASLM is a zero-shot, fine-tuning-free style blending pipeline that replaces linear latent interpolation with SLERP along the geodesic of the hypersphere, preserving latent manifold structure when blending multiple styles. Published at CVPR 2025 Workshop.
Published:
Python has almost no interfaces to implement and no operators to declare. Instead, every piece of syntax, len(x), x[i], for y in x, a + b, with r as f, is a documented call to a method with a double-underscore name. Learn the mapping and your own types stop being second-class citizens.
Published:
An ML interview that asks you to invert a binary tree is not testing your tree knowledge, it is testing whether you can state a cost, defend it, and write a loop whose boundaries are right the first time. Here is the eight-post map of what that requires.
Published:
A Python class is a factory for namespaces, not a sealed blueprint. Once you see where an attribute actually lives, on the instance or on the class, the mutable-default trap, the point of self, and the reason composition usually beats inheritance all fall out of the same rule.
Published:
k-means says a cluster is a ball around a centroid, DBSCAN says it is a connected dense region, and a Gaussian mixture says it is a bump in a density, pick the algorithm and you have already picked the answer.
Published:
DDPM’s objective never actually required the forward process to be Markov, only that its marginals be Gaussian. Dropping the Markov assumption exposes a whole family of samplers a trained model already supports, including a deterministic one that runs in 20 steps and gives an invertible latent space.
Published:
The DDPM objective starts as a variational bound with T+1 KL terms and ends as a plain mean-squared error on noise. Following the collapse shows why the discarded weighting term is not an approximation you tolerate but a reweighting that improves samples.
Published:
Destroying an image is easy and needs no learning at all. Diffusion models exploit that asymmetry: they define a trivial forward corruption, then train a network to walk it backwards one small step at a time.
Published:
A decision tree chops feature space into axis-aligned boxes and predicts one number per box, which explains why it needs no feature scaling, why it approximates a diagonal boundary as a staircase, and why it cannot extrapolate a single step beyond the training range.
Published:
BERT, GPT, and T5 are all Transformers, but their architectures are fundamentally different. One comparison table clarifies the entire landscape.
Published:
Neural Sheaf Diffusion has a theoretical guarantee against representation collapse that does not survive contact with depth. DNSD diagnoses why, the Laplacian’s disagreement signal vanishes as diffusion succeeds, and replaces the operator rather than patching around it.
Published:
A self-contained guide to the Transformer, the engine behind GPT, BERT, and modern AI. Learn how attention replaces recurrence and why every major AI system uses it.
Published:
Every later book on this site assumes you already know what a loss is, why gradient descent works, and what a convolution buys you. This book supplies that, and follows one thread through it: how much structure you build in versus how much you let the data decide.
Published:
Without residual connections, training a 96-layer Transformer would be practically impossible. The skip connection is a simple addition that solves the vanishing gradient problem and enables arbitrary depth.
Published:
In Transformers, depth = expressiveness. In GNNs, depth = both expressiveness AND over-smoothing. The optimal GNN depth is rarely more than 3-4 layers, fundamentally different from the hundreds of layers in modern LLMs.
Published:
Oversmoothing and oversquashing are both problems with deep GNNs, but they affect different nodes, have different causes, and require different fixes. Confusing them leads to applying the wrong solution.
Published:
Stack enough GNN layers and all node embeddings converge to the same vector, making the model useless. Oversmoothing is not a training problem; it is a mathematical inevitability of iterated averaging.
Published:
A summary statistic is a lossy compression of a sample. Knowing which information each one discards, and why sample variance divides by n minus 1 rather than n, is most of what descriptive statistics has to teach.
Published:
A matrix is not a table of numbers, it is a deformation of space. The SVD says every deformation is the same three moves in sequence: rotate, stretch along axes, rotate again.
Published:
DDPM’s objective never actually required the forward process to be Markov, only that its marginals be Gaussian. Dropping the Markov assumption exposes a whole family of samplers a trained model already supports, including a deterministic one that runs in 20 steps and gives an invertible latent space.
Published:
Dijkstra is greedy: it finalises the closest unvisited vertex and never revisits it. That is valid only because every edge adds weight, put one negative edge in the graph and the algorithm returns a wrong answer without complaining.
Published:
The restriction maps in a cellular sheaf can be constrained to different matrix classes: scalars, diagonal matrices, orthogonal matrices, or general matrices. Each class offers a different trade-off between expressivity and computational cost.
Published:
A dict trades memory for the ability to skip the search entirely. Everything that follows, why keys must be hashable, why lists cannot be keys, and why 1, 1.0 and True collide, comes from that single trade.
Published:
Curvature at a point is one over the radius of the circle that best hugs the curve there. Push that idea up to surfaces and Gauss’s Theorema Egregium falls out: some curvature is visible from inside the surface, which is why no map projection can ever get distances right.
Published:
DiffPool learns to hierarchically cluster nodes into super-nodes across layers, like a convolutional pyramid for graphs. Unlike flat global pooling, it captures multi-scale graph structure by differentiably assigning nodes to clusters.
Published:
Diffusion travels well between modalities, but not because it is generic. It fits wherever the answer is genuinely one-to-many, and the specific reason differs for a spectrogram, a protein backbone and a robot’s next ten actions.
Published:
Flow matching is often presented as the successor to diffusion. It is more accurate, and more useful, to say that diffusion is one particular probability path inside the flow-matching framework, and not the straightest one available.
Published:
Better ODE solvers bottom out around ten evaluations because the trajectory is genuinely curved. To go lower you have to change the model, either teach a student to take two teacher steps at once, or train a network that jumps to the end of the trajectory from anywhere on it.
Published:
Once you see the reverse process as an ODE, the thousand-step sampling loop stops being a fact of life and becomes what it really is, a first-order solver with a terrible step size. Everything after that is numerical analysis.
Published:
Most of the bits in a photograph encode texture no one can see. Latent diffusion throws them away first with an autoencoder, then runs the entire diffusion process in a space roughly forty-eight times smaller, which is how Stable Diffusion fits on a consumer GPU.
Published:
Conditioning a diffusion model on a prompt is easy; making it actually obey the prompt is not. Classifier-free guidance fixes that by training one network to do two jobs and then extrapolating between its own answers.
Published:
DDPM’s objective never actually required the forward process to be Markov, only that its marginals be Gaussian. Dropping the Markov assumption exposes a whole family of samplers a trained model already supports, including a deterministic one that runs in 20 steps and gives an invertible latent space.
Published:
Noise prediction and score estimation are the same network in different units. Taking the step size to zero turns the whole method into a stochastic differential equation, and reveals a deterministic ODE with identical marginals hiding inside it.
Published:
The DDPM objective starts as a variational bound with T+1 KL terms and ends as a plain mean-squared error on noise. Following the collapse shows why the discarded weighting term is not an approximation you tolerate but a reweighting that improves samples.
Published:
Reversing a corruption is generally intractable. Diffusion escapes by taking steps small enough that the reverse conditional is itself Gaussian, so the network only has to output a mean. Which mean it outputs, though, turns out to matter a great deal.
Published:
The forward process looks like the trivial half of diffusion, but every term in it is load-bearing. Drop the shrink factor and the variance diverges; pick the wrong schedule and a third of your timesteps are spent denoising static.
Published:
Destroying an image is easy and needs no learning at all. Diffusion models exploit that asymmetry: they define a trivial forward corruption, then train a network to walk it backwards one small step at a time.
Published:
The Sheaf Laplacian generalises the graph Laplacian by incorporating per-edge restriction maps. Its spectrum reveals how consistent data is under the sheaf. Sheaf diffusion with this Laplacian generalises GCN to handle heterophilic graphs.
Published:
Z-SASLM is a zero-shot, fine-tuning-free style blending pipeline that replaces linear latent interpolation with SLERP along the geodesic of the hypersphere, preserving latent manifold structure when blending multiple styles. Published at CVPR 2025 Workshop.
Published:
Dijkstra is greedy: it finalises the closest unvisited vertex and never revisits it. That is valid only because every edge adds weight, put one negative edge in the graph and the algorithm returns a wrong answer without complaining.
Published:
A manifold is a space that looks flat close up and can be curved or knotted globally. That gap between local and global is why charts exist, why a single autoencoder latent space sometimes cannot fit the data, and what dimensionality reduction is really exploiting.
Published:
PCA can be derived by asking for the directions of greatest spread, or by asking for the subspace that loses the least when you project onto it. The two questions look unrelated and have the same answer, which is the most useful thing to understand about it.
Published:
Not all graphs are equal. Directed edges, edge weights, multiple node/edge types, each variant requires different GNN design choices.
Published:
A sheaf gives every node a matrix-valued say in how its neighbours reach it, but not in whether they do. Set a node’s restriction maps to zero to stop it listening and you also stop it speaking. Fixing that needs sheaves on directed graphs, and the fix costs the Laplacian its positive semi-definiteness.
Published:
Ten families cover almost everything you will meet in a modelling interview. Nine of them fit in a table; the multivariate Gaussian deserves a page, because its covariance matrix is a statement about geometry.
Published:
A graph edge relates two things. A hyperedge relates any number of them, and hypergraph networks aggregate over it uniformly, every member contributes the same way. Attaching a sheaf gives each member its own linear map into the group, and turns forced consensus into apparent consensus.
Published:
Shortest-path distances between nodes can be encoded as attention biases or node features, directly informing the model about graph proximity without requiring message passing.
Published:
Better ODE solvers bottom out around ten evaluations because the trajectory is genuinely curved. To go lower you have to change the model, either teach a student to take two teacher steps at once, or train a network that jumps to the end of the trajectory from anywhere on it.
Published:
Positional encodings say where a node is in the graph. Structural encodings say what role it plays. They are complementary, and confusing them leads to poor design choices.
Published:
Ten families cover almost everything you will meet in a modelling interview. Nine of them fit in a table; the multivariate Gaussian deserves a page, because its covariance matrix is a statement about geometry.
Published:
Once you see the reverse process as an ODE, the thousand-step sampling loop stops being a fact of life and becomes what it really is, a first-order solver with a terrible step size. Everything after that is numerical analysis.
Published:
Graph neural networks are transforming computational drug discovery. Molecules are natural graphs, and GNNs learn molecular representations that predict toxicity, solubility, binding affinity, and synthesis feasibility, tasks that previously required expensive laboratory experiments.
Published:
Molecules are graphs. Molecular GNNs predict chemical properties from structure. The best models use 3D coordinates and bond angles, not just connectivity.
Published:
Python has almost no interfaces to implement and no operators to declare. Instead, every piece of syntax, len(x), x[i], for y in x, a + b, with r as f, is a documented call to a method with a double-underscore name. Learn the mapping and your own types stop being second-class citizens.
Published:
Most GNN research assumes a fixed graph. Real graphs evolve: edges appear and disappear, node features drift, new nodes arrive. Dynamic graph learning addresses how to model and predict on graphs whose structure changes over time.
Published:
Brain regions do not encode information in a shared feature space, which is exactly the assumption scalar message passing makes. BrainDyn puts learnable restriction maps between regions and integrates the result as a continuous-time system, the first pairing of cellular sheaves with neural ODEs.
Published:
Dynamic programming is not a trick, it is a diagnosis: if a problem has optimal substructure and overlapping subproblems, exhaustive recursion is doing the same work exponentially often and a table fixes it. Here is the diagnosis, and edit distance worked out cell by cell.
Published:
Translation is one group. Swap it for rotations, reflections, or the rigid motions of 3-D space and the same construction produces a different architecture, with the parameter count cut by exactly the size of the orbit.
Published:
Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first, which is faster, shorter, and free of race conditions.
Published:
A graph is a set of nodes connected by edges, but the power of GNNs comes from the features attached to nodes and edges, and the labels we want to predict.
Published:
Dynamic programming is not a trick, it is a diagnosis: if a problem has optimal substructure and overlapping subproblems, exhaustive recursion is doing the same work exponentially often and a table fixes it. Here is the diagnosis, and edit distance worked out cell by cell.
Published:
Brain regions do not encode information in a shared feature space, which is exactly the assumption scalar message passing makes. BrainDyn puts learnable restriction maps between regions and integrates the result as a continuous-time system, the first pairing of cellular sheaves with neural ODEs.
Published:
Every sheaf neural network is built on an operator defined in this 2019 paper. It is not a machine learning paper but a programme for lifting spectral graph theory to cellular sheaves, and it is unusually candid about which parts of the lift fail.
Published:
SGC removes all nonlinearities between GCN layers and collapses the entire propagation into a single pre-computed matrix power. Surprisingly, it matches GCN on most benchmarks, revealing that nonlinearities between layers may be unnecessary.
Published:
Sheaf networks move features through matrix-valued maps but ignore the symmetries of physical space; equivariant GNNs respect those symmetries but move vectors with scalars. ESNN does both, learned, directed, matrix-valued transport that is exactly E(n)-equivariant, and proves that when displacement is the only geometric input, the radial–tangential family is all the linear transport there is.
Published:
Eigenvectors are the directions a matrix does not rotate, when they exist. The SVD asks a weaker question that always has an answer, and that is exactly why it, not the eigendecomposition, is the workhorse of applied linear algebra.
Published:
PCA can be derived by asking for the directions of greatest spread, or by asking for the subspace that loses the least when you project onto it. The two questions look unrelated and have the same answer, which is the most useful thing to understand about it.
Published:
Laplacian eigenvectors are only defined up to sign: if u is an eigenvector, so is -u. This seemingly minor issue creates a fundamental problem for learning with LapPE. Here is the problem, its consequences, and how SignNet solves it.
Published:
The k smallest eigenvectors of the graph Laplacian form a natural positional embedding space, the graph’s own coordinate system. They capture global structure, symmetry, and community membership.
Published:
The Graph Fourier Transform decomposes a signal on a graph into frequency components using the Laplacian’s eigenvectors. This spectral view is the mathematical foundation behind spectral GNNs like ChebNet and GCN.
Published:
Thermodynamic entropy and Shannon entropy differ by a constant with units. Once you accept that, the ELBO stops being an inference trick and becomes a free energy, and diffusion models stop being a clever architecture and become a driven non-equilibrium process.
Published:
The DDPM objective starts as a variational bound with T+1 KL terms and ends as a plain mean-squared error on noise. Following the collapse shows why the discarded weighting term is not an approximation you tolerate but a reweighting that improves samples.
Published:
BERT, GPT, and T5 are all Transformers, but their architectures are fundamentally different. One comparison table clarifies the entire landscape.
Published:
Cross-attention lets one sequence query information from a completely different sequence. It is the bridge between encoder and decoder, and the core of multimodal AI.
Published:
Strings cannot be modified, so every method that looks like it edits one actually builds another. That single fact explains the whole str API, and why growing a string inside a loop can quietly become quadratic.
Published:
Graph classification is the task of predicting a label for an entire graph. It requires composing message passing (node embeddings), readout (graph embedding), and a classifier, and all three choices interact to determine model expressiveness.
Published:
Counting microstates gives you the Boltzmann distribution, and the Boltzmann distribution gives you softmax, simulated annealing and energy-based models. The partition function is not a bookkeeping constant, it is the object that contains every thermodynamic quantity, and it is intractable for exactly that reason.
Published:
Every sheaf model so far discretises the heat equation, which dissipates energy. Suk et al. try the wave equation instead, which conserves it, a one-line change of PDE with a clean theoretical motivation and a genuinely mixed empirical result.
Published:
A decision tree chops feature space into axis-aligned boxes and predicts one number per box, which explains why it needs no feature scaling, why it approximates a diagonal boundary as a staircase, and why it cannot extrapolate a single step beyond the training range.
Published:
Knowledge graphs encode human knowledge as typed entity-relation triples. GNNs enable structure-aware entity representation, multi-hop reasoning, knowledge base completion, and entity alignment, tasks that shallow embedding methods cannot fully solve.
Published:
R-GCN extends GCN to multi-relational graphs by learning a separate weight matrix for each relation type. It handles knowledge graphs with typed edges and powers both entity classification and link prediction tasks.
Published:
Thermodynamic entropy and Shannon entropy differ by a constant with units. Once you accept that, the ELBO stops being an inference trick and becomes a free energy, and diffusion models stop being a clever architecture and become a driven non-equilibrium process.
Published:
Minimising cross-entropy is minimising a KL divergence plus a constant you cannot change. Once that clicks, maximum likelihood, variational inference and the mode-collapse of a VAE all turn out to be the same argument run in different directions.
Published:
Energy is conserved because the laws of physics do not care what time it is. That single sentence is Noether’s theorem, and its machine learning descendant is the reason an equivariant network needs less data than one that must learn the symmetry from examples.
Published:
If rotating a molecule cannot change its energy, that is a fact about the target function you know before training starts. Encoding it in the architecture makes it true everywhere; learning it from augmented data makes it approximately true where you happened to have samples.
Published:
Sheaf networks move features through matrix-valued maps but ignore the symmetries of physical space; equivariant GNNs respect those symmetries but move vectors with scalars. ESNN does both, learned, directed, matrix-valued transport that is exactly E(n)-equivariant, and proves that when displacement is the only geometric input, the radial–tangential family is all the linear transport there is.
Published:
CNNs, GNNs, Transformers and sheaf models look like separate inventions. They are the same recipe applied to different domains: identify the symmetry of your data, then build layers that respect it. This book is that recipe, and the arguments that follow from it.
Published:
A dense layer from a 224-by-224 colour image to 1000 units holds 150.5 million weights; a 3-by-3, 64-filter convolution holds 1,792, and the two constraints that buy that factor of 84,000 are exactly the prior that makes it work on images.
Published:
Equivariance formalises the idea that a function should ‘commute with symmetry transformations.’ A rotation-equivariant model applied to rotated input gives the rotated output, no extra training needed. This is the foundation for geometric deep learning.
Published:
Laplacian eigenvectors are only defined up to sign: if u is an eigenvector, so is -u. This seemingly minor issue creates a fundamental problem for learning with LapPE. Here is the problem, its consequences, and how SignNet solves it.
Published:
Sheaves with orthogonal restriction maps define a connection on the graph, a parallel transport structure over edges. This connects sheaf GNNs to differential geometry and enables equivariant processing of data with local coordinate frames at each node.
Published:
SE(3)-Transformers extend self-attention to 3D point clouds and molecular graphs while maintaining SE(3) equivariance. Attention weights are learned between node pairs; values are equivariant features built from spherical harmonics.
Published:
EGNN achieves E(n)-equivariance with a simple update rule: positions updated via weighted sums of relative position vectors, features updated via invariant distances. No spherical harmonics needed.
Published:
Probability runs forwards: pick a model, predict the data. Statistics runs backwards, and backwards is harder, many models could have produced what you saw. Everything else in this book is machinery for handling that ambiguity honestly.
Published:
An estimator is a random variable, so it has a mean and a spread. Squared error splits exactly into those two pieces, and once you see the split, it becomes obvious that deliberately biasing an estimator can make it strictly better.
Published:
Newton says a particle moves because a force pushes it. Lagrange says it moves along the path that makes the action stationary. The second statement is harder to believe and far easier to use, and it is the one machine learning inherited.
Published:
Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first, which is faster, shorter, and free of race conditions.
Published:
Expectation adds up no matter how tangled the dependence. Variance does not, and the correction term, covariance, is where most of the interesting behaviour of ensembles, portfolios and minibatch gradients lives.
Published:
How powerful can a GNN be? Xu et al. (2019) answered with a theoretical bound, and GIN is the architecture that achieves it. The secret: use sum aggregation and an MLP, not mean or max.
Published:
A simplex has four kinds of neighbour rather than one, and separating them lets a network see the difference between a filled triangle and an empty one, a distinction no graph neural network can make.
Published:
The restriction maps in a cellular sheaf can be constrained to different matrix classes: scalars, diagonal matrices, orthogonal matrices, or general matrices. Each class offers a different trade-off between expressivity and computational cost.
Published:
Certain graph structures are invisible to message-passing GNNs, not because of bad training, but because of fundamental mathematical limits. Two structurally distinct graphs can produce identical embeddings in any MPNN.
Published:
The 1-WL graph isomorphism test provides the exact upper bound on message-passing GNN expressivity. GIN achieves this bound. Any pair of graphs that 1-WL cannot distinguish cannot be distinguished by any MPNN.
Published:
Strings cannot be modified, so every method that looks like it edits one actually builds another. That single fact explains the whole str API, and why growing a string inside a loop can quietly become quadratic.
Published:
Social networks are large sparse graphs with rich node features (user profiles) and heterogeneous edges (friendship, follow, retweet). GNNs predict user behaviour, detect communities, identify influential spreaders, and flag misinformation, tasks with significant real-world impact.
Published:
Better ODE solvers bottom out around ten evaluations because the trajectory is genuinely curved. To go lower you have to change the model, either teach a student to take two teacher steps at once, or train a network that jumps to the end of the trajectory from anywhere on it.
Published:
A graph is a set of nodes connected by edges, but the power of GNNs comes from the features attached to nodes and edges, and the labels we want to predict.
Published:
A perfectly straight generative trajectory is exactly solvable in a single Euler step. Every extra sampling step you pay for is buying back curvature, so rectified flow attacks the curvature itself rather than the solver.
with Is Not Optional Published:
A file object that falls out of scope without with does get closed eventually, but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.
Published:
Pick the parameter under which the data you actually observed would have been least surprising. That single sentence generates the Gaussian mean, cross-entropy loss, and, once you add a prior, L2 regularisation.
Published:
A Python variable is not a container that holds a value, it is a label stuck onto an object that lives somewhere else. Almost every early surprise, from shared lists to 0.1 + 0.2, follows from taking that sentence literally.
Published:
Flow matching is often presented as the successor to diffusion. It is more accurate, and more useful, to say that diffusion is one particular probability path inside the flow-matching framework, and not the straightest one available.
Published:
A perfectly straight generative trajectory is exactly solvable in a single Euler step. Every extra sampling step you pay for is buying back curvature, so rectified flow attacks the curvature itself rather than the solver.
Published:
Continuous normalising flows were elegant and nearly untrainable, every gradient step needed an ODE solve and a divergence estimate. Flow matching removes both by regressing a velocity field against a target you can write down in closed form, one example at a time.
Published:
Brain regions do not encode information in a shared feature space, which is exactly the assumption scalar message passing makes. BrainDyn puts learnable restriction maps between regions and integrates the result as a continuous-time system, the first pairing of cellular sheaves with neural ODEs.
Published:
A Langevin step is a gradient step plus noise. Change the potential to the negative log-density and it samples from that density instead of minimising it, which is the entire sampler behind score-based generative models.
Published:
Traffic prediction is a canonical spatio-temporal graph task: sensors on roads form a fixed graph, and speed/volume measurements evolve over time. GNNs capture spatial correlations between sensors; RNNs or convolutions capture temporal patterns. Together they achieve state-of-the-art traffic forecasting.
Published:
Spatio-temporal GNNs combine spatial message passing with temporal sequence modelling. They are the dominant approach for traffic forecasting, weather prediction, and any task where measurements at sensor nodes evolve over time on a fixed graph.
Published:
The forward process looks like the trivial half of diffusion, but every term in it is load-bearing. Drop the shrink factor and the variance diverges; pick the wrong schedule and a third of your timesteps are spent denoising static.
Published:
Almost every loss function in machine learning is a negative log-likelihood in disguise, and almost every model output is a distribution. This book rebuilds the probability you need to read those objects fluently, and flags the eight places interviewers know people slip.
Published:
The MPNN framework (Gilmer et al., 2017) unifies GCN, GAT, GIN, GraphSAGE, and almost all spatial GNNs under one abstraction: message functions, aggregation, and update. Understanding MPNN means understanding the whole GNN family.
Published:
Every GNN, GCN, GAT, GraphSAGE, GIN, is a special case of message passing. Learn the three-step loop that defines them all: compute messages, aggregate, update.
Published:
Thermodynamic entropy and Shannon entropy differ by a constant with units. Once you accept that, the ELBO stops being an inference trick and becomes a free energy, and diffusion models stop being a clever architecture and become a driven non-equilibrium process.
Published:
This paper does two things at once: it explains what RoPE is really doing inside a trained LLM, and it proposes p-RoPE, a partial rotary variant that drops the lowest frequencies to preserve stronger semantic channels.
Published:
One school says probability is a long-run frequency, so parameters cannot have probabilities. The other says probability is a degree of belief, so they can. Everything else, priors, credible intervals, the whole argument, follows from that one disagreement.
Published:
A def is executed, not declared: it builds a function object once, evaluating the default arguments there and then. Everything odd about mutable defaults and late-binding closures follows from taking that seriously.
pip install Published:
Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed, most of what people reach for a package to solve is one import away.
Published:
Before understanding GNNs, you need to understand how graphs are represented mathematically. The adjacency matrix is the foundation, a simple grid that tells you which nodes are connected.
Published:
Write self-attention in the message-passing template and you get the Graph Attention Network equations with one substitution: the neighbourhood becomes the whole input. The two architectures are not analogous, they are the same operator on different graphs.
Published:
GCN assigns the same (degree-based) weight to every neighbour. GAT learns which neighbours actually matter, using attention coefficients on edges. More expressive, more interpretable.
Published:
The previous chapter ended on an arbitrary choice that could not be eliminated. Gauge theory’s answer is to stop trying: keep every local frame, transport between them explicitly, and require the model to be indifferent to which frames you picked.
Published:
Sheaves with orthogonal restriction maps define a connection on the graph, a parallel transport structure over edges. This connects sheaf GNNs to differential geometry and enables equivariant processing of data with local coordinate frames at each node.
Published:
Ten families cover almost everything you will meet in a modelling interview. Nine of them fit in a table; the multivariate Gaussian deserves a page, because its covariance matrix is a statement about geometry.
Published:
k-means says a cluster is a ball around a centroid, DBSCAN says it is a connected dense region, and a Gaussian mixture says it is a bump in a density, pick the algorithm and you have already picked the answer.
Published:
GCN (Kipf & Welling, 2016) is the ‘hello world’ of GNNs. It simplifies spectral graph convolution into a single elegant layer: normalised neighbourhood averaging with a learned linear transformation.
Published:
Once ReLU became the default, researchers started asking a better question: can we keep the easy optimization while making the activation smoother, softer, and more expressive? This chapter covers the modern answers.
Published:
The restriction maps in a cellular sheaf can be constrained to different matrix classes: scalars, diagonal matrices, orthogonal matrices, or general matrices. Each class offers a different trade-off between expressivity and computational cost.
Published:
A model can fail for two opposite reasons: it is too rigid to represent the truth, or so flexible that it chases the noise. Squared error splits cleanly into exactly those two terms plus a floor you cannot beat.
Published:
Z-SASLM is a zero-shot, fine-tuning-free style blending pipeline that replaces linear latent interpolation with SLERP along the geodesic of the hypersphere, preserving latent manifold structure when blending multiple styles. Published at CVPR 2025 Workshop.
Published:
Continuous normalising flows were elegant and nearly untrainable, every gradient step needed an ODE solve and a divergence estimate. Flow matching removes both by regressing a velocity field against a target you can write down in closed form, one example at a time.
Published:
Destroying an image is easy and needs no learning at all. Diffusion models exploit that asymmetry: they define a trivial forward corruption, then train a network to walk it backwards one small step at a time.
Published:
A comprehension builds the whole result before you touch any of it. Change one pair of brackets to parentheses and nothing is built at all until you ask, the difference, on a million elements, is 40 MB against 432 bytes.
Published:
On a surface there is no global grid to slide a filter along, and no canonical direction to call ‘up’. What survives is distance, and building convolution out of distance alone exposes exactly one ambiguity, which is where the next chapter starts.
Published:
Put an inner product on every tangent space and let it vary smoothly: that single object determines lengths, angles, distances, straight lines and volume. It also explains why a hierarchy embeds badly in Euclidean space and beautifully in hyperbolic space, the volume is in the wrong place.
Published:
Sheaves with orthogonal restriction maps define a connection on the graph, a parallel transport structure over edges. This connects sheaf GNNs to differential geometry and enables equivariant processing of data with local coordinate frames at each node.
Published:
If rotating a molecule cannot change its energy, that is a fact about the target function you know before training starts. Encoding it in the architecture makes it true everywhere; learning it from augmented data makes it approximately true where you happened to have samples.
Published:
Sheaf networks move features through matrix-valued maps but ignore the symmetries of physical space; equivariant GNNs respect those symmetries but move vectors with scalars. ESNN does both, learned, directed, matrix-valued transport that is exactly E(n)-equivariant, and proves that when displacement is the only geometric input, the radial–tangential family is all the linear transport there is.
Published:
The previous chapter ended on an arbitrary choice that could not be eliminated. Gauge theory’s answer is to stop trying: keep every local frame, transport between them explicitly, and require the model to be indifferent to which frames you picked.
Published:
On a surface there is no global grid to slide a filter along, and no canonical direction to call ‘up’. What survives is distance, and building convolution out of distance alone exposes exactly one ambiguity, which is where the next chapter starts.
Published:
Translation is one group. Swap it for rotations, reflections, or the rigid motions of 3-D space and the same construction produces a different architecture, with the parameter count cut by exactly the size of the orbit.
Published:
Convolution is not a clever idea someone had about images. It is the only linear map that commutes with translation, a theorem, not a design choice, and one you can verify by exhaustion on a small enough case.
Published:
Message passing on a sparse graph does asymptotically less work than attention over every pair. It is still the slower one to train. This chapter is about why the architecture that wins is the one your hardware happens to like.
Published:
Write self-attention in the message-passing template and you get the Graph Attention Network equations with one substitution: the neighbourhood becomes the whole input. The two architectures are not analogous, they are the same operator on different graphs.
Published:
A graph’s nodes have no canonical order, so any model that reads one must give the same answer under relabelling. That single requirement forces the three-step message, aggregate, update template, it is not a design choice.
Published:
CNNs, GNNs, Transformers and sheaf models look like separate inventions. They are the same recipe applied to different domains: identify the symmetry of your data, then build layers that respect it. This book is that recipe, and the arguments that follow from it.
Published:
Tensor Field Networks (TFN) were the first architecture to achieve SE(3) equivariance using spherical harmonics and Clebsch-Gordan tensor products. They laid the theoretical foundation for NequIP and MACE, the current state-of-the-art in equivariant molecular force fields.
Published:
One bilinear form generates the whole of flat geometry: lengths, angles, orthogonality, projections and the distance from a point to a hyperplane. Get the projection formula and you get the SVM margin for free.
Published:
Every embedding you have ever trained lives in a metric space, every dataset you have ever fitted sits near a surface far thinner than its ambient dimension, and every architecture you trust encodes a symmetry. Geometry is not decoration on top of ML, it is what makes the problems tractable.
Published:
Many real-world graphs are embedded in 3D space, molecules, proteins, point clouds, crystal structures. Standard GNNs ignore coordinates and only use connectivity. Geometric GNNs incorporate spatial positions and must respect physical symmetries.
Published:
Python names are references, so passing a list to a function hands over the original. Python threads hold one interpreter lock, so two CPU-bound threads take as long as one after the other. Both facts explain bugs you have already had.
Published:
How powerful can a GNN be? Xu et al. (2019) answered with a theoretical bound, and GIN is the architecture that achieves it. The secret: use sum aggregation and an MLP, not mean or max.
Published:
Graph Transformers replace or augment local message passing with full pairwise attention, every node attends to every other node. This solves long-range dependencies and over-squashing at the cost of O(N²) computation.
Published:
To predict a property of an entire graph, node embeddings must be aggregated into a single vector. The choice of global pooling, mean, sum, or max, is not arbitrary: each has distinct expressive power and fits different tasks.
Published:
A sheaf is a mathematical object from algebraic topology that assigns vector spaces to cells and linear maps between them. On graphs, sheaves assign feature spaces to nodes and edges, with restriction maps encoding how node features relate across edges.
Published:
The last activation in your network is not a modelling preference, it is a contract with your loss function. Break it and training stops meaning anything. Here is the contract, and what changes when the activation itself becomes learned.
Published:
Write self-attention in the message-passing template and you get the Graph Attention Network equations with one substitution: the neighbourhood becomes the whole input. The two architectures are not analogous, they are the same operator on different graphs.
Published:
A graph’s nodes have no canonical order, so any model that reads one must give the same answer under relabelling. That single requirement forces the three-step message, aggregate, update template, it is not a design choice.
Published:
A decision tree chops feature space into axis-aligned boxes and predicts one number per box, which explains why it needs no feature scaling, why it approximates a diagonal boundary as a staircase, and why it cannot extrapolate a single step beyond the training range.
Published:
Convexity buys one enormous guarantee, every local minimum is global, and it says nothing at all about speed. Deep learning throws the guarantee away and still works, and it is worth being precise about how much of that we actually understand.
Published:
Training is one loop: measure the loss, ask backpropagation which way is downhill, take a small step. This chapter derives exactly how small that step has to be, why the answer is 2/a for a quadratic, and why reverse-mode differentiation gets you every gradient for roughly the price of one forward pass.
Published:
Without residual connections, training a 96-layer Transformer would be practically impossible. The skip connection is a simple addition that solves the vanishing gradient problem and enables arbitrary depth.
Published:
Most matrix-calculus mistakes are not mistakes about calculus. They are mistakes about which convention you were using, and they produce an answer that is correct up to a transpose, which is to say, wrong.
Published:
Treating the derivative as a slope stops working the moment there is more than one input. Treating it as the best linear approximation keeps working forever, and makes backpropagation an obvious consequence of the chain rule rather than an algorithm to memorise.
Published:
Dividing by √d_k is not just a trick, it prevents softmax from saturating and dying in high-dimensional spaces. Here’s the math and the intuition.
Published:
GCN assigns the same (degree-based) weight to every neighbour. GAT learns which neighbours actually matter, using attention coefficients on edges. More expressive, more interpretable.
Published:
The Graph Laplacian is L = D - A. Its eigenvectors reveal the graph’s community structure; its eigenvalues tell you how well-connected the graph is. It’s also the mathematical bridge from spectral theory to GNNs like GCN.
Published:
Before understanding GNNs, you need to understand how graphs are represented mathematically. The adjacency matrix is the foundation, a simple grid that tells you which nodes are connected.
Published:
Graphs are everywhere, molecules, social networks, road maps, knowledge bases. Graph Neural Networks learn from this relational structure by propagating information between connected nodes. Here’s the complete picture.
Published:
SheafPool solves a key missing piece in sheaf GNNs: graph-level pooling. Instead of averaging stalk vectors in arbitrary local bases, it aligns them into a shared canonical frame and builds a readout that is invariant to local basis changes.
Published:
Graph classification is the task of predicting a label for an entire graph. It requires composing message passing (node embeddings), readout (graph embedding), and a classifier, and all three choices interact to determine model expressiveness.
Published:
Mean and sum readout treat all nodes equally. Attention readout learns which nodes matter most for a given task. Set2Set goes further, it uses an LSTM to iteratively query the node set, producing richer graph representations than single-pass pooling.
Published:
Instead of soft cluster assignment (DiffPool), TopKPool and SAGPool select a subset of the most important nodes, producing a smaller but sparser graph at each level. Hard selection is scalable but requires careful score learning.
Published:
DiffPool learns to hierarchically cluster nodes into super-nodes across layers, like a convolutional pyramid for graphs. Unlike flat global pooling, it captures multi-scale graph structure by differentiably assigning nodes to clusters.
Published:
To predict a property of an entire graph, node embeddings must be aggregated into a single vector. The choice of global pooling, mean, sum, or max, is not arbitrary: each has distinct expressive power and fits different tasks.
Published:
GNNs can predict at three levels: properties of individual nodes, existence or type of edges, or properties of entire graphs. Each level requires a different output head and training setup.
Published:
SGC removes all nonlinearities between GCN layers and collapses the entire propagation into a single pre-computed matrix power. Surprisingly, it matches GCN on most benchmarks, revealing that nonlinearities between layers may be unnecessary.
Published:
| ChebNet avoids the expensive full eigendecomposition by approximating spectral filters with Chebyshev polynomials, achieving O( | E | ) computation and spatial locality without sacrificing expressiveness. |
Published:
Neural ODEs replace discrete layer-by-layer computation with continuous dynamics governed by a differential equation. Graph Neural ODEs apply this to graph data, treating node embeddings as a dynamical system evolving in continuous time.
Published:
The 1-WL graph isomorphism test provides the exact upper bound on message-passing GNN expressivity. GIN achieves this bound. Any pair of graphs that 1-WL cannot distinguish cannot be distinguished by any MPNN.
Published:
Standard GNNs assume neighbouring nodes should agree. Sheaf Neural Networks replace that assumption with a learned linear map on every edge, which turns heterophily, oversmoothing, and directional structure into one operator: the sheaf Laplacian.
Published:
Neural Sheaf Diffusion has a theoretical guarantee against representation collapse that does not survive contact with depth. DNSD diagnoses why, the Laplacian’s disagreement signal vanishes as diffusion succeeds, and replaces the operator rather than patching around it.
Published:
PolyNSD replaces the NSD propagation operator with a degree-K Chebyshev polynomial in the normalised sheaf Laplacian, achieving SOTA on homo- and heterophilic benchmarks with only diagonal restriction maps and dramatically lower memory usage.
Published:
HetSheaf encodes graph heterogeneity directly in the sheaf data structure, type-aware stalks and restriction maps conditioned on node and edge types, instead of specialised architectural components, achieving +2pp on HGB with 10× fewer parameters.
Published:
GNNs can predict at three levels: properties of individual nodes, existence or type of edges, or properties of entire graphs. Each level requires a different output head and training setup.
Published:
Oversquashing occurs when exponentially many node features must be compressed into a fixed-size embedding through a bottleneck edge. It is the reason GNNs struggle with long-range dependencies, not just oversmoothing.
Published:
The Graph Fourier Transform decomposes a signal on a graph into frequency components using the Laplacian’s eigenvectors. This spectral view is the mathematical foundation behind spectral GNNs like ChebNet and GCN.
Published:
A graph is a set of nodes connected by edges, but the power of GNNs comes from the features attached to nodes and edges, and the labels we want to predict.
Published:
The k smallest eigenvectors of the graph Laplacian form a natural positional embedding space, the graph’s own coordinate system. They capture global structure, symmetry, and community membership.
Published:
Graphormer encodes graph structure directly into Transformer attention via three biases: node centrality, spatial encoding (shortest paths), and edge encoding. It won the OGB-LSC 2021 competition on molecular property prediction.
Published:
Graph Transformers replace or augment local message passing with full pairwise attention, every node attends to every other node. This solves long-range dependencies and over-squashing at the cost of O(N²) computation.
Published:
Laplacian eigenvector positional encodings tell a node where it sits in the graph, but the graph Laplacian only knows adjacency, so two structurally identical nodes get identical encodings no matter how different their features are. Swap in the sheaf Laplacian and the features enter the spectrum.
Published:
Not all graphs are equal. Directed edges, edge weights, multiple node/edge types, each variant requires different GNN design choices.
Published:
Dijkstra is greedy: it finalises the closest unvisited vertex and never revisits it. That is valid only because every edge adds weight, put one negative edge in the graph and the algorithm returns a wrong answer without complaining.
Published:
GCN and GAT learn embeddings for fixed graphs, add a new node and you’re stuck. GraphSAGE (Hamilton et al., 2017) learns an aggregation function instead, so it can generate embeddings for entirely new nodes at inference time.
Published:
Convolution is not a clever idea someone had about images. It is the only linear map that commutes with translation, a theorem, not a design choice, and one you can verify by exhaustion on a small enough case.
Published:
Translation is one group. Swap it for rotations, reflections, or the rigid motions of 3-D space and the same construction produces a different architecture, with the parameter count cut by exactly the size of the orbit.
Published:
Equivariance formalises the idea that a function should ‘commute with symmetry transformations.’ A rotation-equivariant model applied to rotated input gives the rotated output, no extra training needed. This is the foundation for geometric deep learning.
Published:
If rotating a molecule cannot change its energy, that is a fact about the target function you know before training starts. Encoding it in the architecture makes it true everywhere; learning it from augmented data makes it approximately true where you happened to have samples.
Published:
A recurrent network shares weights across time exactly as a convolution shares them across space. The trouble is that gradients then travel through a product of Jacobians, and a product of a hundred numbers slightly below one is zero.
Published:
Conditioning a diffusion model on a prompt is easy; making it actually obey the prompt is not. Classifier-free guidance fixes that by training one network to do two jobs and then extrapolating between its own answers.
Published:
Hamiltonian Monte Carlo is not a heuristic that happens to move well. It is exact because Hamiltonian flow preserves phase-space volume and is reversible, which is what makes the Metropolis acceptance ratio collapse to a difference of energies.
Published:
Message passing on a sparse graph does asymptotically less work than attention over every pair. It is still the slower one to train. This chapter is about why the architecture that wins is the one your hardware happens to like.
Published:
A linked list has O(1) insertion and loses to an array anyway. Appending is O(1) but a single append can copy a million elements. Hash lookup is O(1) until every key lands in one bucket. All three claims are true, with the qualifier that got dropped.
Published:
A dict trades memory for the ability to skip the search entirely. Everything that follows, why keys must be hashable, why lists cannot be keys, and why 1, 1.0 and True collide, comes from that single trade.
Published:
A heap is a tree with no pointers, just an array and two index formulas, and building one from n items takes O(n), not O(n log n). The sum that proves it is the most quotable derivation in this book.
Published:
The Jacobian tells you how a map distorts volume, which is exactly the term normalising flows have to pay. The Hessian tells you the shape of the valley you are descending. Both are indispensable to reason with and, at a billion parameters, hopeless to form.
Published:
HAN combines meta-path decomposition with two levels of attention: node-level attention weights neighbours along a meta-path, and semantic-level attention weights different meta-paths. This lets the model learn which relationships matter most for a given task.
Published:
Not all graphs are equal. Directed edges, edge weights, multiple node/edge types, each variant requires different GNN design choices.
Published:
Most real-world graphs are heterogeneous, they contain multiple node types (users, items, tags) and edge types (clicks, rates, authors). Standard GNNs treat all nodes and edges identically, making them blind to this type structure.
Published:
SheafPool solves a key missing piece in sheaf GNNs: graph-level pooling. Instead of averaging stalk vectors in arbitrary local bases, it aligns them into a shared canonical frame and builds a readout that is invariant to local basis changes.
Published:
HetSheaf encodes graph heterogeneity directly in the sheaf data structure, type-aware stalks and restriction maps conditioned on node and edge types, instead of specialised architectural components, achieving +2pp on HGB with 10× fewer parameters.
Published:
Every sheaf network so far predicts restriction maps with an MLP on concatenated features, a universal approximator with no inductive bias for heterophily at all. Two alternatives drawn from opinion dynamics get the bias for free, and stop the parameter count scaling with the feature dimension.
Published:
Laplacian eigenvector positional encodings tell a node where it sits in the graph, but the graph Laplacian only knows adjacency, so two structurally identical nodes get identical encodings no matter how different their features are. Swap in the sheaf Laplacian and the features enter the spectrum.
Published:
Every sheaf model so far discretises the heat equation, which dissipates energy. Suk et al. try the wave equation instead, which conserves it, a one-line change of PDE with a clean theoretical motivation and a genuinely mixed empirical result.
Published:
Neural Sheaf Diffusion has a theoretical guarantee against representation collapse that does not survive contact with depth. DNSD diagnoses why, the Laplacian’s disagreement signal vanishes as diffusion succeeds, and replaces the operator rather than patching around it.
Published:
GNNs fail on heterophilic graphs and they oversmooth with depth. Bodnar et al. show these are one failure with one cause, the graph is implicitly equipped with a trivial sheaf, and prove a hierarchy of exactly what richer sheaves buy you.
Published:
Standard GNNs assume neighbouring nodes should agree. Sheaf Neural Networks replace that assumption with a learned linear map on every edge, which turns heterophily, oversmoothing, and directional structure into one operator: the sheaf Laplacian.
Published:
Sheaf GNNs are the principled solution to heterophily: by learning per-edge maps that transform features before comparison, they can perform diffusion that converges within classes and diverges across classes, the exact opposite of standard GCN’s collapse.
Published:
Neural Sheaf Diffusion (Bodnar et al., 2022) learns the sheaf restriction maps from data using a neural network, then performs diffusion with the learned Sheaf Laplacian. This gives a principled, topology-grounded GNN that handles heterophily without heuristic fixes.
Published:
Standard message passing aggregates neighbour features and averages. On heterophilic graphs (where neighbours often disagree), this is harmful. Cellular sheaves provide a mathematically principled framework to model per-edge relationships between node features, going beyond mere averaging.
Published:
Most GNNs assume nearby nodes are similar, the homophily assumption. When this breaks (heterophilic graphs), standard message passing hurts performance. Understanding this distinction is essential for modern GNN design.
Published:
k-means says a cluster is a ball around a centroid, DBSCAN says it is a connected dense region, and a Gaussian mixture says it is a bump in a density, pick the algorithm and you have already picked the answer.
Published:
DiffPool learns to hierarchically cluster nodes into super-nodes across layers, like a convolutional pyramid for graphs. Unlike flat global pooling, it captures multi-scale graph structure by differentiably assigning nodes to clusters.
Published:
A graph edge relates two things. A hyperedge relates any number of them, and hypergraph networks aggregate over it uniformly, every member contributes the same way. Attaching a sheaf gives each member its own linear map into the group, and turns forced consensus into apparent consensus.
Published:
A simplicial complex cannot hold a benzene ring as a single cell, filling the hexagon costs three edges between atoms that share no bond, and this one constraint is what cell and combinatorial complexes exist to remove.
Published:
TDA computes a topological descriptor and hands it to a model; TDL makes the topological object the domain the model runs on. Telling the two apart is the difference between a preprocessing step and an architecture.
Published:
Most knowledge graphs treat facts as timeless, but facts change. Barack Obama was president from 2009 to 2017. Temporal Knowledge Graphs add timestamps to triples, requiring models to reason about what was true when.
Published:
Every sheaf neural network is built on an operator defined in this 2019 paper. It is not a machine learning paper but a programme for lifting spectral graph theory to cellular sheaves, and it is unusually candid about which parts of the lift fail.
Published:
A simplex has four kinds of neighbour rather than one, and separating them lets a network see the difference between a filled triangle and an empty one, a distinction no graph neural network can make.
Published:
The previous chapter ended on an arbitrary choice that could not be eliminated. Gauge theory’s answer is to stop trying: keep every local frame, transport between them explicitly, and require the model to be indifferent to which frames you picked.
Published:
Most GNNs assume nearby nodes are similar, the homophily assumption. When this breaks (heterophilic graphs), standard message passing hurts performance. Understanding this distinction is essential for modern GNN design.
Published:
Put an inner product on every tangent space and let it vary smoothly: that single object determines lengths, angles, distances, straight lines and volume. It also explains why a hierarchy embeds badly in Euclidean space and beautifully in hyperbolic space, the volume is in the wrong place.
Published:
A graph edge relates two things. A hyperedge relates any number of them, and hypergraph networks aggregate over it uniformly, every member contributes the same way. Attaching a sheaf gives each member its own linear map into the group, and turns forced consensus into apparent consensus.
Published:
A simplicial complex cannot hold a benzene ring as a single cell, filling the hexagon costs three edges between atoms that share no bond, and this one constraint is what cell and combinatorial complexes exist to remove.
Published:
TDA computes a topological descriptor and hands it to a model; TDL makes the topological object the domain the model runs on. Telling the two apart is the difference between a preprocessing step and an architecture.
Published:
A p-value answers one narrow question: if nothing were going on, how often would data look at least this extreme? It says nothing about whether the effect is real, large, or worth shipping, and running twenty of them changes the meaning of all twenty.
Published:
Pythonic code and fast code are usually the same code, iterating over objects instead of indices is both clearer and quicker, but the two goals diverge exactly at the point where people start guessing instead of profiling.
Published:
An import is not a textual include, it executes a file once, caches the result, and binds a name. Almost every confusing import error, from circular imports to ‘attempted relative import with no known parent package’, follows directly from that one sentence.
Published:
A test catches 99% of a disease and false-alarms only 5% of the time. You test positive. The chance you are ill is one in six. Here is the computation, and why almost everyone guesses ten times too high.
Published:
Certain graph structures are invisible to message-passing GNNs, not because of bad training, but because of fundamental mathematical limits. Two structurally distinct graphs can produce identical embeddings in any MPNN.
Published:
GCN and GAT learn embeddings for fixed graphs, add a new node and you’re stuck. GraphSAGE (Hamilton et al., 2017) learns an aggregation function instead, so it can generate embeddings for entirely new nodes at inference time.
Published:
Every sheaf network so far predicts restriction maps with an MLP on concatenated features, a universal approximator with no inductive bias for heterophily at all. Two alternatives drawn from opinion dynamics get the bias for free, and stop the parameter count scaling with the feature dimension.
Published:
Probability runs forwards: pick a model, predict the data. Statistics runs backwards, and backwards is harder, many models could have produced what you saw. Everything else in this book is machinery for handling that ambiguity honestly.
Published:
Social networks are large sparse graphs with rich node features (user profiles) and heterogeneous edges (friendship, follow, retweet). GNNs predict user behaviour, detect communities, identify influential spreaders, and flag misinformation, tasks with significant real-world impact.
Published:
A Python class is a factory for namespaces, not a sealed blueprint. Once you see where an attribute actually lives, on the instance or on the class, the mutable-default trap, the point of self, and the reason composition usually beats inheritance all fall out of the same rule.
Published:
L1 regularisation produces exact zeros and L2 does not. The reason is not statistical, it is geometric: the L1 unit ball has corners on the axes, and corners are what optimisation solutions stick to.
Published:
Diffusion travels well between modalities, but not because it is generic. It fits wherever the answer is genuinely one-to-many, and the specific reason differs for a spectrogram, a protein backbone and a robot’s next ten actions.
Published:
NTK-Aware Scaling extends the context window of RoPE-based models by rescaling frequencies using Neural Tangent Kernel theory, with no fine-tuning required.
Published:
Almost every mathematical question asked in an ML interview reduces to two things: what a matrix does to space, and how to differentiate a composition. This book covers those two things properly and is honest about what you can safely forget.
Published:
An ML interview that asks you to invert a binary tree is not testing your tree knowledge, it is testing whether you can state a cost, defend it, and write a loop whose boundaries are right the first time. Here is the eight-post map of what that requires.
Published:
Almost every loss function in machine learning is a negative log-likelihood in disguise, and almost every model output is a distribution. This book rebuilds the probability you need to read those objects fluently, and flags the eight places interviewers know people slip.
Published:
Q, K, and V are not arbitrary labels. They map precisely onto search queries, database labels, and retrieved content, a framework you already understand.
Published:
If rotating a molecule cannot change its energy, that is a fact about the target function you know before training starts. Encoding it in the architecture makes it true everywhere; learning it from augmented data makes it approximately true where you happened to have samples.
Published:
Equivariance formalises the idea that a function should ‘commute with symmetry transformations.’ A rotation-equivariant model applied to rotated input gives the rotated output, no extra training needed. This is the foundation for geometric deep learning.
Published:
SE(3)-Transformers extend self-attention to 3D point clouds and molecular graphs while maintaining SE(3) equivariance. Attention weights are learned between node pairs; values are equivariant features built from spherical harmonics.
Published:
A comprehension builds the whole result before you touch any of it. Change one pair of brackets to parentheses and nothing is built at all until you ask, the difference, on a million elements, is 40 MB against 432 bytes.
pip install Published:
Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed, most of what people reach for a package to solve is one import away.
Published:
A comprehension builds the whole result before you touch any of it. Change one pair of brackets to parentheses and nothing is built at all until you ask, the difference, on a million elements, is 40 MB against 432 bytes.
Published:
The Jacobian tells you how a map distorts volume, which is exactly the term normalising flows have to pay. The Hessian tells you the shape of the valley you are descending. Both are indispensable to reason with and, at a billion parameters, hopeless to form.
with Is Not Optional Published:
A file object that falls out of scope without with does get closed eventually, but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.
Published:
k-means says a cluster is a ball around a centroid, DBSCAN says it is a connected dense region, and a Gaussian mixture says it is a bump in a density, pick the algorithm and you have already picked the answer.
Published:
Among all the hyperplanes that separate two classes, one sits furthest from both. Finding it turns out to depend on the data only through inner products, and that single fact is what lets you work in a space you never build.
Published:
The FFN block holds two-thirds of a Transformer’s parameters and does most of its factual recall. Yet it is almost always overlooked in introductions to attention.
Published:
Convexity buys one enormous guarantee, every local minimum is global, and it says nothing at all about speed. Deep learning throws the guarantee away and still works, and it is worth being precise about how much of that we actually understand.
Published:
Minimising cross-entropy is minimising a KL divergence plus a constant you cannot change. Once that clicks, maximum likelihood, variational inference and the mode-collapse of a VAE all turn out to be the same argument run in different directions.
Published:
Pick the parameter under which the data you actually observed would have been least surprising. That single sentence generates the Gaussian mean, cross-entropy loss, and, once you add a prior, L2 regularisation.
Published:
Knowledge graphs encode human knowledge as typed entity-relation triples. GNNs enable structure-aware entity representation, multi-hop reasoning, knowledge base completion, and entity alignment, tasks that shallow embedding methods cannot fully solve.
Published:
Knowledge graph completion can be solved with shallow KG embeddings (TransE, DistMult, ComplEx) or with structural GNNs (R-GCN, CompGCN). Each approach has different inductive biases and failure modes. Understanding when to use each is the central design decision for KG tasks.
Published:
R-GCN extends GCN to multi-relational graphs by learning a separate weight matrix for each relation type. It handles knowledge graphs with typed edges and powers both entity classification and link prediction tasks.
Published:
Most real-world graphs are heterogeneous, they contain multiple node types (users, items, tags) and edge types (clicks, rates, authors). Standard GNNs treat all nodes and edges identically, making them blind to this type structure.
Published:
A graph is a set of nodes connected by edges, but the power of GNNs comes from the features attached to nodes and edges, and the labels we want to predict.
Published:
Random walk positional encodings encode each node’s structural context by computing the probability of returning to it from itself in k steps, a computationally efficient alternative to Laplacian eigenvectors with no sign ambiguity.
Published:
A Langevin step is a gradient step plus noise. Change the potential to the negative log-density and it samples from that density instead of minimising it, which is the entire sampler behind score-based generative models.
Published:
On a surface there is no global grid to slide a filter along, and no canonical direction to call ‘up’. What survives is distance, and building convolution out of distance alone exposes exactly one ambiguity, which is where the next chapter starts.
Published:
The Graph Laplacian is L = D - A. Its eigenvectors reveal the graph’s community structure; its eigenvalues tell you how well-connected the graph is. It’s also the mathematical bridge from spectral theory to GNNs like GCN.
Published:
Fitting a line by least squares is not an optimisation trick, it is the orthogonal projection of the observation vector onto the column space of the design matrix, and once you see that, the normal equations, the failure modes, and the reason we solve by QR instead of inverting all follow from one picture.
Published:
Neural ODEs replace discrete layer-by-layer computation with continuous dynamics governed by a differential equation. Graph Neural ODEs apply this to graph data, treating node embeddings as a dynamical system evolving in continuous time.
Published:
Most of the bits in a photograph encode texture no one can see. Latent diffusion throws them away first with an autoencoder, then runs the entire diffusion process in a space roughly forty-eight times smaller, which is how Stable Diffusion fits on a consumer GPU.
Published:
DDPM’s objective never actually required the forward process to be Markov, only that its marginals be Gaussian. Dropping the Markov assumption exposes a whole family of samplers a trained model already supports, including a deterministic one that runs in 20 steps and gives an invertible latent space.
Published:
Z-SASLM is a zero-shot, fine-tuning-free style blending pipeline that replaces linear latent interpolation with SLERP along the geodesic of the hypersphere, preserving latent manifold structure when blending multiple styles. Published at CVPR 2025 Workshop.
Published:
A single Transformer block combines attention, residuals, layer norm, and an FFN into one reusable unit. Understanding this block is understanding the Transformer.
Published:
Layer norm is not optional plumbing. It determines training stability, gradient flow, and whether deep Transformers converge at all. Pre-LN vs Post-LN is not a detail, it changes training dynamics fundamentally.
Published:
Most matrix-calculus mistakes are not mistakes about calculus. They are mistakes about which convention you were using, and they produce an answer that is correct up to a transpose, which is to say, wrong.
Published:
Instead of a fixed formula, why not just train position embeddings from scratch, like word embeddings? That’s exactly what BERT and GPT-1 do. Here’s how and when it works.
Published:
GNNs fail on heterophilic graphs and they oversmooth with depth. Bodnar et al. show these are one failure with one cause, the graph is implicitly equipped with a trivial sheaf, and prove a hierarchy of exactly what richer sheaves buy you.
Published:
Neural Sheaf Diffusion (Bodnar et al., 2022) learns the sheaf restriction maps from data using a neural network, then performs diffusion with the learned Sheaf Laplacian. This gives a principled, topology-grounded GNN that handles heterophily without heuristic fixes.
Published:
Most matrix-calculus mistakes are not mistakes about calculus. They are mistakes about which convention you were using, and they produce an answer that is correct up to a transpose, which is to say, wrong.
Published:
Fitting a line by least squares is not an optimisation trick, it is the orthogonal projection of the observation vector onto the column space of the design matrix, and once you see that, the normal equations, the failure modes, and the reason we solve by QR instead of inverting all follow from one picture.
Published:
Standard message passing aggregates neighbour features and averages. On heterophilic graphs (where neighbours often disagree), this is harmful. Cellular sheaves provide a mathematically principled framework to model per-edge relationships between node features, going beyond mere averaging.
Published:
SGC removes all nonlinearities between GCN layers and collapses the entire propagation into a single pre-computed matrix power. Surprisingly, it matches GCN on most benchmarks, revealing that nonlinearities between layers may be unnecessary.
Published:
A matrix is not a table of numbers, it is a deformation of space. The SVD says every deformation is the same three moves in sequence: rotate, stretch along axes, rotate again.
Published:
One bilinear form generates the whole of flat geometry: lengths, angles, orthogonality, projections and the distance from a point to a hyperplane. Get the projection formula and you get the SVM margin for free.
Published:
A matrix is not a grid of numbers, it is a map. Once you read it that way, rank, column space, null space and rank–nullity stop being definitions to memorise and become one geometric statement about what the map keeps and what it destroys.
Published:
Almost every mathematical question asked in an ML interview reduces to two things: what a matrix does to space, and how to differentiate a composition. This book covers those two things properly and is honest about what you can safely forget.
Published:
Fitting a line by least squares is not an optimisation trick, it is the orthogonal projection of the observation vector onto the column space of the design matrix, and once you see that, the normal equations, the failure modes, and the reason we solve by QR instead of inverting all follow from one picture.
Published:
Social networks are large sparse graphs with rich node features (user profiles) and heterogeneous edges (friendship, follow, retweet). GNNs predict user behaviour, detect communities, identify influential spreaders, and flag misinformation, tasks with significant real-world impact.
Published:
TGN (Temporal Graph Network) is the leading framework for continuous-time dynamic graphs. It maintains a per-node memory that is updated upon each interaction, enabling efficient inductive link prediction on event streams.
Published:
Most knowledge graphs treat facts as timeless, but facts change. Barack Obama was president from 2009 to 2017. Temporal Knowledge Graphs add timestamps to triples, requiring models to reason about what was true when.
Published:
Knowledge graph completion can be solved with shallow KG embeddings (TransE, DistMult, ComplEx) or with structural GNNs (R-GCN, CompGCN). Each approach has different inductive biases and failure modes. Understanding when to use each is the central design decision for KG tasks.
Published:
R-GCN extends GCN to multi-relational graphs by learning a separate weight matrix for each relation type. It handles knowledge graphs with typed edges and powers both entity classification and link prediction tasks.
Published:
GNNs can predict at three levels: properties of individual nodes, existence or type of edges, or properties of entire graphs. Each level requires a different output head and training setup.
Published:
A Python list is a growable array of pointers, and that one implementation fact predicts every complexity in its API: appending is cheap, inserting at the front is not, and in costs a full scan.
Published:
The central limit theorem is an asymptotic statement about the centre of a distribution. It says nothing about your sample size, and nothing about the tails you actually care about, which is why concentration inequalities exist.
Published:
Neural Sheaf Diffusion learns the restriction maps by gradient descent. Conn-NSD computes them once, before training, by assuming the data lies on a manifold and optimally aligning neighbouring tangent spaces, matching the learned models on small graphs at roughly half the cost per epoch.
pip install Published:
Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed, most of what people reach for a package to solve is one import away.
Published:
Logistic regression is not a squashed linear regression, it is a straight line drawn in log-odds space, which is why one coefficient means one multiplication of the odds, and why perfectly separable data drives that coefficient to infinity.
Published:
FoPE rethinks long-context positional encoding from a frequency-domain perspective. Instead of only stretching RoPE heuristically, it explicitly improves attention’s periodic extension so Transformers generalize more gracefully to longer sequences.
Published:
Position Interpolation rescales positions before applying RoPE so a model trained on short contexts can be adapted to longer ones with surprisingly little fine-tuning. It became the reference baseline for long-context RoPE extension.
Published:
XPos modifies RoPE with a multiplicative decay that keeps relative rotations while stabilising magnitude at long distance. It is one of the cleanest attempts to make rotary embeddings extrapolate better.
Published:
This paper does two things at once: it explains what RoPE is really doing inside a trained LLM, and it proposes p-RoPE, a partial rotary variant that drops the lowest frequencies to preserve stronger semantic channels.
Published:
GAPE is a drop-in RoPE augmentation that adds content-aware attention logit biases: a query-gate suppresses irrelevant distant context while a key-gate preserves salient distant tokens. Provably sharper attention and improved long-context robustness, no architecture changes needed.
Published:
A sheaf gives every node a matrix-valued say in how its neighbours reach it, but not in whether they do. Set a node’s restriction maps to zero to stop it listening and you also stop it speaking. Fixing that needs sheaves on directed graphs, and the fix costs the Laplacian its positive semi-definiteness.
Published:
Oversquashing occurs when exponentially many node features must be compressed into a fixed-size embedding through a bottleneck edge. It is the reason GNNs struggle with long-range dependencies, not just oversmoothing.
Published:
Eigenvectors are the directions a matrix does not rotate, when they exist. The SVD asks a weaker question that always has an answer, and that is exactly why it, not the eigendecomposition, is the workhorse of applied linear algebra.
Published:
No comparison sort can beat n log n, that is a theorem with a two-line counting proof, not a limit of current cleverness. Counting sort beats it anyway, because it never compares two elements. Also: how to write a binary search whose boundaries are right.
Published:
A recurrent network shares weights across time exactly as a convolution shares them across space. The trouble is that gradients then travel through a product of Jacobians, and a product of a hundred numbers slightly below one is zero.
Published:
Every later book on this site assumes you already know what a loss is, why gradient descent works, and what a convolution buys you. This book supplies that, and follows one thread through it: how much structure you build in versus how much you let the data decide.
Published:
A manifold is a space that looks flat close up and can be curved or knotted globally. That gap between local and global is why charts exist, why a single autoencoder latent space sometimes cannot fit the data, and what dimensionality reduction is really exploiting.
Published:
A manifold is a space that looks flat close up and can be curved or knotted globally. That gap between local and global is why charts exist, why a single autoencoder latent space sometimes cannot fit the data, and what dimensionality reduction is really exploiting.
Published:
Every embedding you have ever trained lives in a metric space, every dataset you have ever fitted sits near a surface far thinner than its ambient dimension, and every architecture you trust encodes a symmetry. Geometry is not decoration on top of ML, it is what makes the problems tractable.
Published:
On a surface there is no global grid to slide a filter along, and no canonical direction to call ‘up’. What survives is distance, and building convolution out of distance alone exposes exactly one ambiguity, which is where the next chapter starts.
Published:
Robots interact with structured environments: objects have relationships, joints form kinematic chains, agents communicate through interaction graphs. GNNs encode these relational structures, enabling generalisation across object configurations, robot morphologies, and multi-agent scenarios.
Published:
Pick the parameter under which the data you actually observed would have been least surprising. That single sentence generates the Gaussian mean, cross-entropy loss, and, once you add a prior, L2 regularisation.
Published:
Among all the hyperplanes that separate two classes, one sits furthest from both. Finding it turns out to depend on the data only through inner products, and that single fact is what lets you work in a space you never build.
Published:
The difference between GPT, BERT, and T5 is largely a masking decision. Learn how causal, padding, and bidirectional masks shape what each token is allowed to see.
Published:
Big-O is an upper bound on growth, not a running time, and saying an algorithm is O(n²) is compatible with it being O(n). Here is what each symbol actually claims, how to read a cost off loops and recursions, and why the asymptotically better algorithm sometimes loses on real hardware.
is Is Not ==, and for Has an else Published:
== asks whether two objects are equal; is asks whether they are the same object. CPython’s caching of small integers makes the two agree just often enough to teach you the wrong lesson.
Published:
Most matrix-calculus mistakes are not mistakes about calculus. They are mistakes about which convention you were using, and they produce an answer that is correct up to a transpose, which is to say, wrong.
Published:
Pick the parameter under which the data you actually observed would have been least surprising. That single sentence generates the Gaussian mean, cross-entropy loss, and, once you add a prior, L2 regularisation.
Published:
Hamiltonian Monte Carlo is not a heuristic that happens to move well. It is exact because Hamiltonian flow preserves phase-space volume and is reversible, which is what makes the Metropolis acceptance ratio collapse to a difference of energies.
Published:
Probability is not defined as a long-run frequency or a degree of belief. It is defined as a function on sets that obeys three rules, and the third rule, countable additivity, is the one doing all the work.
Published:
Hamiltonian Monte Carlo is not a heuristic that happens to move well. It is exact because Hamiltonian flow preserves phase-space volume and is reversible, which is what makes the Metropolis acceptance ratio collapse to a difference of energies.
Published:
Newton says a particle moves because a force pushes it. Lagrange says it moves along the path that makes the action stationary. The second statement is harder to believe and far easier to use, and it is the one machine learning inherited.
Published:
Self-attention is the core of every Transformer. Learn how Query, Key, and Value vectors let every token directly attend to every other, and why that matters.
Published:
Dynamic programming is not a trick, it is a diagnosis: if a problem has optimal substructure and overlapping subproblems, exhaustive recursion is doing the same work exponentially often and a table fixes it. Here is the diagnosis, and edit distance worked out cell by cell.
Published:
Python names are references, so passing a list to a function hands over the original. Python threads hold one interpreter lock, so two CPU-bound threads take as long as one after the other. Both facts explain bugs you have already had.
Published:
TGN (Temporal Graph Network) is the leading framework for continuous-time dynamic graphs. It maintains a per-node memory that is updated upon each interaction, enabling efficient inductive link prediction on event streams.
Published:
On a surface there is no global grid to slide a filter along, and no canonical direction to call ‘up’. What survives is distance, and building convolution out of distance alone exposes exactly one ambiguity, which is where the next chapter starts.
Published:
A graph’s nodes have no canonical order, so any model that reads one must give the same answer under relabelling. That single requirement forces the three-step message, aggregate, update template, it is not a design choice.
Published:
A simplex has four kinds of neighbour rather than one, and separating them lets a network see the difference between a filled triangle and an empty one, a distinction no graph neural network can make.
Published:
Standard message passing aggregates neighbour features and averages. On heterophilic graphs (where neighbours often disagree), this is harmful. Cellular sheaves provide a mathematically principled framework to model per-edge relationships between node features, going beyond mere averaging.
Published:
The MPNN framework (Gilmer et al., 2017) unifies GCN, GAT, GIN, GraphSAGE, and almost all spatial GNNs under one abstraction: message functions, aggregation, and update. Understanding MPNN means understanding the whole GNN family.
Published:
Every GNN, GCN, GAT, GraphSAGE, GIN, is a special case of message passing. Learn the three-step loop that defines them all: compute messages, aggregate, update.
Published:
HAN combines meta-path decomposition with two levels of attention: node-level attention weights neighbours along a meta-path, and semantic-level attention weights different meta-paths. This lets the model learn which relationships matter most for a given task.
Published:
Most real-world graphs are heterogeneous, they contain multiple node types (users, items, tags) and edge types (clicks, rates, authors). Standard GNNs treat all nodes and edges identically, making them blind to this type structure.
Published:
Once ReLU became the default, researchers started asking a better question: can we keep the easy optimization while making the activation smoother, softer, and more expressive? This chapter covers the modern answers.
Published:
An import is not a textual include, it executes a file once, caches the result, and binds a name. Almost every confusing import error, from circular imports to ‘attempted relative import with no known parent package’, follows directly from that one sentence.
Published:
Molecules are graphs. Molecular GNNs predict chemical properties from structure. The best models use 3D coordinates and bond angles, not just connectivity.
Published:
EGNN achieves E(n)-equivariance with a simple update rule: positions updated via weighted sums of relative position vectors, features updated via invariant distances. No spherical harmonics needed.
Published:
Many real-world graphs are embedded in 3D space, molecules, proteins, point clouds, crystal structures. Standard GNNs ignore coordinates and only use connectivity. Geometric GNNs incorporate spatial positions and must respect physical symmetries.
Published:
Graphormer encodes graph structure directly into Transformer attention via three biases: node centrality, spatial encoding (shortest paths), and edge encoding. It won the OGB-LSC 2021 competition on molecular property prediction.
Published:
Graph neural networks are transforming computational drug discovery. Molecules are natural graphs, and GNNs learn molecular representations that predict toxicity, solubility, binding affinity, and synthesis feasibility, tasks that previously required expensive laboratory experiments.
Published:
Diffusion travels well between modalities, but not because it is generic. It fits wherever the answer is genuinely one-to-many, and the specific reason differs for a spectrogram, a protein backbone and a robot’s next ten actions.
Published:
Graph neural networks are transforming computational drug discovery. Molecules are natural graphs, and GNNs learn molecular representations that predict toxicity, solubility, binding affinity, and synthesis feasibility, tasks that previously required expensive laboratory experiments.
Published:
Every GNN, GCN, GAT, GraphSAGE, GIN, is a special case of message passing. Learn the three-step loop that defines them all: compute messages, aggregate, update.
Published:
Robots interact with structured environments: objects have relationships, joints form kinematic chains, agents communicate through interaction graphs. GNNs encode these relational structures, enabling generalisation across object configurations, robot morphologies, and multi-agent scenarios.
Published:
One attention head sees one relationship. Multiple heads running in parallel let the model capture syntax, semantics, and coreference simultaneously, here’s how.
Published:
Not all graphs are equal. Directed edges, edge weights, multiple node/edge types, each variant requires different GNN design choices.
Published:
Cross-attention lets one sequence query information from a completely different sequence. It is the bridge between encoder and decoder, and the core of multimodal AI.
Published:
A p-value answers one narrow question: if nothing were going on, how often would data look at least this extreme? It says nothing about whether the effect is real, large, or worth shipping, and running twenty of them changes the meaning of all twenty.
Published:
Minimising cross-entropy is minimising a KL divergence plus a constant you cannot change. Once that clicks, maximum likelihood, variational inference and the mode-collapse of a VAE all turn out to be the same argument run in different directions.
Published:
Neural ODEs replace discrete layer-by-layer computation with continuous dynamics governed by a differential equation. Graph Neural ODEs apply this to graph data, treating node embeddings as a dynamical system evolving in continuous time.
Published:
Graphs are everywhere, molecules, social networks, road maps, knowledge bases. Graph Neural Networks learn from this relational structure by propagating information between connected nodes. Here’s the complete picture.
Published:
Activation functions are the reason neural networks can model curved decision boundaries instead of collapsing into one giant linear map. This chapter builds the intuition first, then walks through the classical functions that shaped deep learning.
Published:
Brain regions do not encode information in a shared feature space, which is exactly the assumption scalar message passing makes. BrainDyn puts learnable restriction maps between regions and integrates the result as a continuous-time system, the first pairing of cellular sheaves with neural ODEs.
Published:
GNNs fail on heterophilic graphs and they oversmooth with depth. Bodnar et al. show these are one failure with one cause, the graph is implicitly equipped with a trivial sheaf, and prove a hierarchy of exactly what richer sheaves buy you.
Published:
Neural Sheaf Diffusion (Bodnar et al., 2022) learns the sheaf restriction maps from data using a neural network, then performs diffusion with the learned Sheaf Laplacian. This gives a principled, topology-grounded GNN that handles heterophily without heuristic fixes.
Published:
Brain regions do not encode information in a shared feature space, which is exactly the assumption scalar message passing makes. BrainDyn puts learnable restriction maps between regions and integrates the result as a continuous-time system, the first pairing of cellular sheaves with neural ODEs.
Published:
A self-contained guide to the Transformer, the engine behind GPT, BERT, and modern AI. Learn how attention replaces recurrence and why every major AI system uses it.
Published:
Sheaf GNNs are the principled solution to heterophily: by learning per-edge maps that transform features before comparison, they can perform diffusion that converges within classes and diverges across classes, the exact opposite of standard GCN’s collapse.
Published:
HAN combines meta-path decomposition with two levels of attention: node-level attention weights neighbours along a meta-path, and semantic-level attention weights different meta-paths. This lets the model learn which relationships matter most for a given task.
Published:
GNNs can predict at three levels: properties of individual nodes, existence or type of edges, or properties of entire graphs. Each level requires a different output head and training setup.
Published:
Instead of soft cluster assignment (DiffPool), TopKPool and SAGPool select a subset of the most important nodes, producing a smaller but sparser graph at each level. Hard selection is scalable but requires careful score learning.
Published:
A graph is a set of nodes connected by edges, but the power of GNNs comes from the features attached to nodes and edges, and the labels we want to predict.
Published:
Energy is conserved because the laws of physics do not care what time it is. That single sentence is Noether’s theorem, and its machine learning descendant is the reason an equivariant network needs less data than one that must learn the symmetry from examples.
Published:
The forward process looks like the trivial half of diffusion, but every term in it is load-bearing. Drop the shrink factor and the variance diverges; pick the wrong schedule and a third of your timesteps are spent denoising static.
Published:
L1 regularisation produces exact zeros and L2 does not. The reason is not statistical, it is geometric: the L1 unit ball has corners on the axes, and corners are what optimisation solutions stick to.
Published:
A matrix is not a grid of numbers, it is a map. Once you read it that way, rank, column space, null space and rank–nullity stop being definitions to memorise and become one geometric statement about what the map keeps and what it destroys.
Published:
Pythonic code and fast code are usually the same code, iterating over objects instead of indices is both clearer and quicker, but the two goals diverge exactly at the point where people start guessing instead of profiling.
Published:
Computer vision tasks increasingly require relational reasoning, understanding how objects relate to each other, not just what they are. Scene graph generation, visual question answering, action recognition from skeletons, and 3D point cloud processing all benefit from GNN-based relational modelling.
Published:
Once you see the reverse process as an ODE, the thousand-step sampling loop stops being a fact of life and becomes what it really is, a first-order solver with a terrible step size. Everything after that is numerical analysis.
Published:
Continuous normalising flows were elegant and nearly untrainable, every gradient step needed an ODE solve and a divergence estimate. Flow matching removes both by regressing a velocity field against a target you can write down in closed form, one example at a time.
Published:
A Python class is a factory for namespaces, not a sealed blueprint. Once you see where an attribute actually lives, on the instance or on the class, the mutable-default trap, the point of self, and the reason composition usually beats inheritance all fall out of the same rule.
Published:
Python has almost no interfaces to implement and no operators to declare. Instead, every piece of syntax, len(x), x[i], for y in x, a + b, with r as f, is a documented call to a method with a double-underscore name. Learn the mapping and your own types stop being second-class citizens.
is Is Not ==, and for Has an else Published:
== asks whether two objects are equal; is asks whether they are the same object. CPython’s caching of small integers makes the two agree just often enough to teach you the wrong lesson.
Published:
Every sheaf network so far predicts restriction maps with an MLP on concatenated features, a universal approximator with no inductive bias for heterophily at all. Two alternatives drawn from opinion dynamics get the bias for free, and stop the parameter count scaling with the feature dimension.
Published:
Almost every mathematical question asked in an ML interview reduces to two things: what a matrix does to space, and how to differentiate a composition. This book covers those two things properly and is honest about what you can safely forget.
Published:
Training is one loop: measure the loss, ask backpropagation which way is downhill, take a small step. This chapter derives exactly how small that step has to be, why the answer is 2/a for a quadratic, and why reverse-mode differentiation gets you every gradient for roughly the price of one forward pass.
Published:
The restriction maps in a cellular sheaf can be constrained to different matrix classes: scalars, diagonal matrices, orthogonal matrices, or general matrices. Each class offers a different trade-off between expressivity and computational cost.
Published:
GAT weights a neighbour by a scalar. SheafAN keeps the scalar and adds a learned orthogonal transport matrix alongside it, recovering GAT exactly at d = 1, and turning a model that goes numerically unstable past eight layers into one that runs to sixty-four.
Published:
APPNP decouples feature transformation from propagation. A neural network transforms features first; then Personalized PageRank propagates the result. This enables deep propagation without over-smoothing.
Published:
Most GNNs assume nearby nodes are similar, the homophily assumption. When this breaks (heterophilic graphs), standard message passing hurts performance. Understanding this distinction is essential for modern GNN design.
Published:
A model can fail for two opposite reasons: it is too rigid to represent the truth, or so flexible that it chases the noise. Squared error splits cleanly into exactly those two terms plus a floor you cannot beat.
Published:
Collaborative filtering represents every user and item as one static vector. Sheaf4Rec replaces each with a vector space, and reports consistent gains on ranking metrics, though the wins come from recall rather than precision, and the headline efficiency claim is hard to reconcile with the timing table.
Published:
A graph edge relates two things. A hyperedge relates any number of them, and hypergraph networks aggregate over it uniformly, every member contributes the same way. Attaching a sheaf gives each member its own linear map into the group, and turns forced consensus into apparent consensus.
Published:
Neural Sheaf Diffusion has a theoretical guarantee against representation collapse that does not survive contact with depth. DNSD diagnoses why, the Laplacian’s disagreement signal vanishes as diffusion succeeds, and replaces the operator rather than patching around it.
Published:
GNNs fail on heterophilic graphs and they oversmooth with depth. Bodnar et al. show these are one failure with one cause, the graph is implicitly equipped with a trivial sheaf, and prove a hierarchy of exactly what richer sheaves buy you.
Published:
Sheaf GNNs are the principled solution to heterophily: by learning per-edge maps that transform features before comparison, they can perform diffusion that converges within classes and diverges across classes, the exact opposite of standard GCN’s collapse.
Published:
In Transformers, depth = expressiveness. In GNNs, depth = both expressiveness AND over-smoothing. The optimal GNN depth is rarely more than 3-4 layers, fundamentally different from the hundreds of layers in modern LLMs.
Published:
Oversmoothing and oversquashing are both problems with deep GNNs, but they affect different nodes, have different causes, and require different fixes. Confusing them leads to applying the wrong solution.
Published:
Stack enough GNN layers and all node embeddings converge to the same vector, making the model useless. Oversmoothing is not a training problem; it is a mathematical inevitability of iterated averaging.
Published:
A sheaf gives every node a matrix-valued say in how its neighbours reach it, but not in whether they do. Set a node’s restriction maps to zero to stop it listening and you also stop it speaking. Fixing that needs sheaves on directed graphs, and the fix costs the Laplacian its positive semi-definiteness.
Published:
Oversmoothing and oversquashing are both problems with deep GNNs, but they affect different nodes, have different causes, and require different fixes. Confusing them leads to applying the wrong solution.
Published:
Oversquashing occurs when exponentially many node features must be compressed into a fixed-size embedding through a bottleneck edge. It is the reason GNNs struggle with long-range dependencies, not just oversmoothing.
Published:
Diffusion models, energy-based models and Hamiltonian Monte Carlo were not inspired by physics, they are physics, rewritten with a neural network in place of an analytic potential. Knowing which physics saves you from re-deriving it badly.
Published:
Every embedding you have ever trained lives in a metric space, every dataset you have ever fitted sits near a surface far thinner than its ambient dimension, and every architecture you trust encodes a symmetry. Geometry is not decoration on top of ML, it is what makes the problems tractable.
Published:
Almost every loss function in machine learning is a negative log-likelihood in disguise, and almost every model output is a distribution. This book rebuilds the probability you need to read those objects fluently, and flags the eight places interviewers know people slip.
Published:
Probability runs forwards: pick a model, predict the data. Statistics runs backwards, and backwards is harder, many models could have produced what you saw. Everything else in this book is machinery for handling that ambiguity honestly.
Published:
CNNs, GNNs, Transformers and sheaf models look like separate inventions. They are the same recipe applied to different domains: identify the symmetry of your data, then build layers that respect it. This book is that recipe, and the arguments that follow from it.
Published:
Destroying an image is easy and needs no learning at all. Diffusion models exploit that asymmetry: they define a trivial forward corruption, then train a network to walk it backwards one small step at a time.
Published:
Transformers see all tokens at once, which means without help they’d treat ‘cat ate mouse’ and ‘mouse ate cat’ the same. Positional encodings fix this. Here’s the full landscape.
Published:
Every later book on this site assumes you already know what a loss is, why gradient descent works, and what a convolution buys you. This book supplies that, and follows one thread through it: how much structure you build in versus how much you let the data decide.
Published:
Standard GNNs assume neighbouring nodes should agree. Sheaf Neural Networks replace that assumption with a learned linear map on every edge, which turns heterophily, oversmoothing, and directional structure into one operator: the sheaf Laplacian.
Published:
Graphs are everywhere, molecules, social networks, road maps, knowledge bases. Graph Neural Networks learn from this relational structure by propagating information between connected nodes. Here’s the complete picture.
Published:
This paper does two things at once: it explains what RoPE is really doing inside a trained LLM, and it proposes p-RoPE, a partial rotary variant that drops the lowest frequencies to preserve stronger semantic channels.
Published:
A p-value answers one narrow question: if nothing were going on, how often would data look at least this extreme? It says nothing about whether the effect is real, large, or worth shipping, and running twenty of them changes the meaning of all twenty.
Published:
An import is not a textual include, it executes a file once, caches the result, and binds a name. Almost every confusing import error, from circular imports to ‘attempted relative import with no known parent package’, follows directly from that one sentence.
Published:
The difference between GPT, BERT, and T5 is largely a masking decision. Learn how causal, padding, and bidirectional masks shape what each token is allowed to see.
Published:
Neural Sheaf Diffusion learns the restriction maps by gradient descent. Conn-NSD computes them once, before training, by assuming the data lies on a manifold and optimally aligning neighbouring tangent spaces, matching the learned models on small graphs at roughly half the cost per epoch.
Published:
The previous chapter ended on an arbitrary choice that could not be eliminated. Gauge theory’s answer is to stop trying: keep every local frame, transport between them explicitly, and require the model to be indifferent to which frames you picked.
Published:
Every sheaf network so far predicts restriction maps with an MLP on concatenated features, a universal approximator with no inductive bias for heterophily at all. Two alternatives drawn from opinion dynamics get the bias for free, and stop the parameter count scaling with the feature dimension.
with Is Not Optional Published:
A file object that falls out of scope without with does get closed eventually, but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.
Published:
PCA can be derived by asking for the directions of greatest spread, or by asking for the subspace that loses the least when you project onto it. The two questions look unrelated and have the same answer, which is the most useful thing to understand about it.
Published:
Every sheaf model so far discretises the heat equation, which dissipates energy. Suk et al. try the wave equation instead, which conserves it, a one-line change of PDE with a clean theoretical motivation and a genuinely mixed empirical result.
Published:
A probability density can be 2, or 200, and nothing is wrong. Getting clear on what a PDF actually is fixes half the confusion about continuous distributions, and explains the Jacobian term that makes normalising flows work.
Published:
Pythonic code and fast code are usually the same code, iterating over objects instead of indices is both clearer and quicker, but the two goals diverge exactly at the point where people start guessing instead of profiling.
Published:
A graph’s nodes have no canonical order, so any model that reads one must give the same answer under relabelling. That single requirement forces the three-step message, aggregate, update template, it is not a design choice.
Published:
If you cannot resample from the population, resample from your sample instead. That one substitution gives standard errors and intervals for statistics whose sampling distributions nobody can write down, and it fails in ways worth memorising.
Published:
This book has two halves. One computes topological summaries of data and feeds them to a model. The other makes the topology itself the domain the network runs on. This guide sets out both, and the pipeline that connects them.
Published:
Newton says a particle moves because a force pushes it. Lagrange says it moves along the path that makes the action stationary. The second statement is harder to believe and far easier to use, and it is the one machine learning inherited.
Published:
Diffusion models, energy-based models and Hamiltonian Monte Carlo were not inspired by physics, they are physics, rewritten with a neural network in place of an analytic potential. Knowing which physics saves you from re-deriving it badly.
Published:
Python is not interpreted line by line, and python is not the language. Knowing what CPython compiles your file into, and where it puts the packages you install, explains most of the confusion beginners hit in their first week.
Published:
Robots interact with structured environments: objects have relationships, joints form kinematic chains, agents communicate through interaction graphs. GNNs encode these relational structures, enabling generalisation across object configurations, robot morphologies, and multi-agent scenarios.
Published:
Computer vision tasks increasingly require relational reasoning, understanding how objects relate to each other, not just what they are. Scene graph generation, visual question answering, action recognition from skeletons, and 3D point cloud processing all benefit from GNN-based relational modelling.
Published:
Ten families cover almost everything you will meet in a modelling interview. Nine of them fit in a table; the multivariate Gaussian deserves a page, because its covariance matrix is a statement about geometry.
Published:
Polynomial Neural Sheaf Diffusion (PNSD) replaces the fixed diffusion operator (I - Δ_F) with a learnable polynomial of the Sheaf Laplacian. This gives the model spectral flexibility, it can learn to amplify or suppress different frequency components of the sheaf signal.
Published:
PolyNSD replaces the NSD propagation operator with a degree-K Chebyshev polynomial in the normalised sheaf Laplacian, achieving SOTA on homo- and heterophilic benchmarks with only diagonal restriction maps and dramatically lower memory usage.
Published:
Polynomial Neural Sheaf Diffusion (PNSD) replaces the fixed diffusion operator (I - Δ_F) with a learnable polynomial of the Sheaf Laplacian. This gives the model spectral flexibility, it can learn to amplify or suppress different frequency components of the sheaf signal.
Published:
SheafPool solves a key missing piece in sheaf GNNs: graph-level pooling. Instead of averaging stalk vectors in arbitrary local bases, it aligns them into a shared canonical frame and builds a readout that is invariant to local basis changes.
Published:
A dense layer from a 224-by-224 colour image to 1000 units holds 150.5 million weights; a 3-by-3, 64-filter convolution holds 1,792, and the two constraints that buy that factor of 84,000 are exactly the prior that makes it work on images.
Published:
To predict a property of an entire graph, node embeddings must be aggregated into a single vector. The choice of global pooling, mean, sum, or max, is not arbitrary: each has distinct expressive power and fits different tasks.
Published:
Positional encodings say where a node is in the graph. Structural encodings say what role it plays. They are complementary, and confusing them leads to poor design choices.
Published:
Position Interpolation rescales positions before applying RoPE so a model trained on short contexts can be adapted to longer ones with surprisingly little fine-tuning. It became the reference baseline for long-context RoPE extension.
Published:
FoPE rethinks long-context positional encoding from a frequency-domain perspective. Instead of only stretching RoPE heuristically, it explicitly improves attention’s periodic extension so Transformers generalize more gracefully to longer sequences.
Published:
Position Interpolation rescales positions before applying RoPE so a model trained on short contexts can be adapted to longer ones with surprisingly little fine-tuning. It became the reference baseline for long-context RoPE extension.
Published:
XPos modifies RoPE with a multiplicative decay that keeps relative rotations while stabilising magnitude at long distance. It is one of the cleanest attempts to make rotary embeddings extrapolate better.
Published:
This paper does two things at once: it explains what RoPE is really doing inside a trained LLM, and it proposes p-RoPE, a partial rotary variant that drops the lowest frequencies to preserve stronger semantic channels.
Published:
GAPE is a drop-in RoPE augmentation that adds content-aware attention logit biases: a query-gate suppresses irrelevant distant context while a key-gate preserves salient distant tokens. Provably sharper attention and improved long-context robustness, no architecture changes needed.
Published:
LongRoPE (Microsoft, 2024) pushes RoPE-based context to 2M tokens by searching for optimal per-dimension rescaling factors, far outperforming NTK or YaRN at extreme lengths.
Published:
YaRN combines NTK scaling for high-frequency dimensions with linear interpolation for low-frequency ones, plus a temperature correction, achieving better long-context performance with minimal fine-tuning.
Published:
NTK-Aware Scaling extends the context window of RoPE-based models by rescaling frequencies using Neural Tangent Kernel theory, with no fine-tuning required.
Published:
ALiBi skips traditional positional embeddings entirely and just subtracts a distance penalty from attention scores. Zero extra parameters, excellent extrapolation. Press et al., 2022.
Published:
RoPE encodes position by rotating query and key vectors by an angle proportional to position. The clever result: absolute encoding produces relative attention for free, and it’s now the dominant PE for large language models.
Published:
Instead of asking ‘where am I?’, relative PEs ask ‘how far are these two tokens apart?’ Shaw et al. and T5 both use this idea to build models that generalise better to variable-length inputs.
Published:
Instead of a fixed formula, why not just train position embeddings from scratch, like word embeddings? That’s exactly what BERT and GPT-1 do. Here’s how and when it works.
Published:
The PE method from the 2017 ‘Attention Is All You Need’ paper uses sine and cosine waves at different frequencies. Learn why this elegant choice encodes position without any training.
Published:
Transformers see all tokens at once, which means without help they’d treat ‘cat ate mouse’ and ‘mouse ate cat’ the same. Positional encodings fix this. Here’s the full landscape.
Published:
Positional encodings say where a node is in the graph. Structural encodings say what role it plays. They are complementary, and confusing them leads to poor design choices.
Published:
Random walk positional encodings encode each node’s structural context by computing the probability of returning to it from itself in k steps, a computationally efficient alternative to Laplacian eigenvectors with no sign ambiguity.
Published:
The k smallest eigenvectors of the graph Laplacian form a natural positional embedding space, the graph’s own coordinate system. They capture global structure, symmetry, and community membership.
Published:
Message-passing GNNs are permutation-equivariant by design, they cannot assign unique positions to nodes. Without positional encodings, symmetric nodes are indistinguishable. Here is why that matters and how to fix it.
Published:
Graph Transformers replace or augment local message passing with full pairwise attention, every node attends to every other node. This solves long-range dependencies and over-squashing at the cost of O(N²) computation.
Published:
Laplacian eigenvector positional encodings tell a node where it sits in the graph, but the graph Laplacian only knows adjacency, so two structurally identical nodes get identical encodings no matter how different their features are. Swap in the sheaf Laplacian and the features enter the spectrum.
Published:
Reversing a corruption is generally intractable. Diffusion escapes by taking steps small enough that the reverse conditional is itself Gaussian, so the network only has to output a mean. Which mean it outputs, though, turns out to matter a great deal.
Published:
A heap is a tree with no pointers, just an array and two index formulas, and building one from n items takes O(n), not O(n log n). The sum that proves it is the most quotable derivation in this book.
Published:
Probability is not defined as a long-run frequency or a degree of belief. It is defined as a function on sets that obeys three rules, and the third rule, countable additivity, is the one doing all the work.
Published:
Almost every loss function in machine learning is a negative log-likelihood in disguise, and almost every model output is a distribution. This book rebuilds the probability you need to read those objects fluently, and flags the eight places interviewers know people slip.
Published:
Noise prediction and score estimation are the same network in different units. Taking the step size to zero turns the whole method into a stochastic differential equation, and reveals a deterministic ODE with identical marginals hiding inside it.
Published:
Flow matching is often presented as the successor to diffusion. It is more accurate, and more useful, to say that diffusion is one particular probability path inside the flow-matching framework, and not the straightest one available.
Published:
Python names are references, so passing a list to a function hands over the original. Python threads hold one interpreter lock, so two CPU-bound threads take as long as one after the other. Both facts explain bugs you have already had.
Published:
Pythonic code and fast code are usually the same code, iterating over objects instead of indices is both clearer and quicker, but the two goals diverge exactly at the point where people start guessing instead of profiling.
Published:
One bilinear form generates the whole of flat geometry: lengths, angles, orthogonality, projections and the distance from a point to a hyperplane. Get the projection formula and you get the SVM margin for free.
Published:
APPNP decouples feature transformation from propagation. A neural network transforms features first; then Personalized PageRank propagates the result. This enables deep propagation without over-smoothing.
Published:
Python has almost no interfaces to implement and no operators to declare. Instead, every piece of syntax, len(x), x[i], for y in x, a + b, with r as f, is a documented call to a method with a double-underscore name. Learn the mapping and your own types stop being second-class citizens.
Published:
A comprehension builds the whole result before you touch any of it. Change one pair of brackets to parentheses and nothing is built at all until you ask, the difference, on a million elements, is 40 MB against 432 bytes.
Published:
A def is executed, not declared: it builds a function object once, evaluating the default arguments there and then. Everything odd about mutable defaults and late-binding closures follows from taking that seriously.
Published:
Strings cannot be modified, so every method that looks like it edits one actually builds another. That single fact explains the whole str API, and why growing a string inside a loop can quietly become quadratic.
Published:
A dict trades memory for the ability to skip the search entirely. Everything that follows, why keys must be hashable, why lists cannot be keys, and why 1, 1.0 and True collide, comes from that single trade.
Published:
A Python list is a growable array of pointers, and that one implementation fact predicts every complexity in its API: appending is cheap, inserting at the front is not, and in costs a full scan.
is Is Not ==, and for Has an else Published:
== asks whether two objects are equal; is asks whether they are the same object. CPython’s caching of small integers makes the two agree just often enough to teach you the wrong lesson.
Published:
A Python variable is not a container that holds a value, it is a label stuck onto an object that lives somewhere else. Almost every early surprise, from shared lists to 0.1 + 0.2, follows from taking that sentence literally.
Published:
Python is not interpreted line by line, and python is not the language. Knowing what CPython compiles your file into, and where it puts the packages you install, explains most of the confusion beginners hit in their first week.
Published:
A linked list has O(1) insertion and loses to an array anyway. Appending is O(1) but a single append can copy a million elements. Hash lookup is O(1) until every key lands in one bucket. All three claims are true, with the qualifier that got dropped.
Published:
A summary statistic is a lossy compression of a sample. Knowing which information each one discards, and why sample variance divides by n minus 1 rather than n, is most of what descriptive statistics has to teach.
Published:
A heap is a tree with no pointers, just an array and two index formulas, and building one from n items takes O(n), not O(n log n). The sum that proves it is the most quotable derivation in this book.
Published:
No comparison sort can beat n log n, that is a theorem with a two-line counting proof, not a limit of current cleverness. Counting sort beats it anyway, because it never compares two elements. Also: how to write a binary search whose boundaries are right.
Published:
Sheaf networks move features through matrix-valued maps but ignore the symmetries of physical space; equivariant GNNs respect those symmetries but move vectors with scalars. ESNN does both, learned, directed, matrix-valued transport that is exactly E(n)-equivariant, and proves that when displacement is the only geometric input, the radial–tangential family is all the linear transport there is.
Published:
A decision tree chops feature space into axis-aligned boxes and predicts one number per box, which explains why it needs no feature scaling, why it approximates a diagonal boundary as a staircase, and why it cannot extrapolate a single step beyond the training range.
Published:
A probability density can be 2, or 200, and nothing is wrong. Getting clear on what a PDF actually is fixes half the confusion about continuous distributions, and explains the Jacobian term that makes normalising flows work.
Published:
Random walk positional encodings encode each node’s structural context by computing the probability of returning to it from itself in k steps, a computationally efficient alternative to Laplacian eigenvectors with no sign ambiguity.
Published:
A matrix is not a grid of numbers, it is a map. Once you read it that way, rank, column space, null space and rank–nullity stop being definitions to memorise and become one geometric statement about what the map keeps and what it destroys.
Published:
Among all the hyperplanes that separate two classes, one sits furthest from both. Finding it turns out to depend on the data only through inner products, and that single fact is what lets you work in a space you never build.
Published:
Graph classification is the task of predicting a label for an entire graph. It requires composing message passing (node embeddings), readout (graph embedding), and a classifier, and all three choices interact to determine model expressiveness.
Published:
Mean and sum readout treat all nodes equally. Attention readout learns which nodes matter most for a given task. Set2Set goes further, it uses an LSTM to iteratively query the node set, producing richer graph representations than single-pass pooling.
Published:
To predict a property of an entire graph, node embeddings must be aggregated into a single vector. The choice of global pooling, mean, sum, or max, is not arbitrary: each has distinct expressive power and fits different tasks.
Published:
Knowledge graphs encode human knowledge as typed entity-relation triples. GNNs enable structure-aware entity representation, multi-hop reasoning, knowledge base completion, and entity alignment, tasks that shallow embedding methods cannot fully solve.
Published:
A dense layer from a 224-by-224 colour image to 1000 units holds 150.5 million weights; a 3-by-3, 64-filter convolution holds 1,792, and the two constraints that buy that factor of 84,000 are exactly the prior that makes it work on images.
Published:
In Transformers, depth = expressiveness. In GNNs, depth = both expressiveness AND over-smoothing. The optimal GNN depth is rarely more than 3-4 layers, fundamentally different from the hundreds of layers in modern LLMs.
Published:
Collaborative filtering represents every user and item as one static vector. Sheaf4Rec replaces each with a vector space, and reports consistent gains on ranking metrics, though the wins come from recall rather than precision, and the headline efficiency claim is hard to reconcile with the timing table.
Published:
Recommendation is naturally a graph problem: users and items are nodes, interactions are edges. GNNs on bipartite user-item graphs capture higher-order collaborative filtering signals, friends of friends liked this, that matrix factorisation cannot represent.
Published:
Flow matching is often presented as the successor to diffusion. It is more accurate, and more useful, to say that diffusion is one particular probability path inside the flow-matching framework, and not the straightest one available.
Published:
A perfectly straight generative trajectory is exactly solvable in a single Euler step. Every extra sampling step you pay for is buying back curvature, so rectified flow attacks the curvature itself rather than the solver.
Published:
Dynamic programming is not a trick, it is a diagnosis: if a problem has optimal substructure and overlapping subproblems, exhaustive recursion is doing the same work exponentially often and a table fixes it. Here is the diagnosis, and edit distance worked out cell by cell.
Published:
A perfectly straight generative trajectory is exactly solvable in a single Euler step. Every extra sampling step you pay for is buying back curvature, so rectified flow attacks the curvature itself rather than the solver.
Published:
Certain graph structures are invisible to message-passing GNNs, not because of bad training, but because of fundamental mathematical limits. Two structurally distinct graphs can produce identical embeddings in any MPNN.
Published:
Fitting a line by least squares is not an optimisation trick, it is the orthogonal projection of the observation vector onto the column space of the design matrix, and once you see that, the normal equations, the failure modes, and the reason we solve by QR instead of inverting all follow from one picture.
Published:
A model can fail for two opposite reasons: it is too rigid to represent the truth, or so flexible that it chases the noise. Squared error splits cleanly into exactly those two terms plus a floor you cannot beat.
Published:
R-GCN extends GCN to multi-relational graphs by learning a separate weight matrix for each relation type. It handles knowledge graphs with typed edges and powers both entity classification and link prediction tasks.
Published:
Most real-world graphs are heterogeneous, they contain multiple node types (users, items, tags) and edge types (clicks, rates, authors). Standard GNNs treat all nodes and edges identically, making them blind to this type structure.
Published:
Instead of asking ‘where am I?’, relative PEs ask ‘how far are these two tokens apart?’ Shaw et al. and T5 both use this idea to build models that generalise better to variable-length inputs.
Published:
Activation functions are the reason neural networks can model curved decision boundaries instead of collapsing into one giant linear map. This chapter builds the intuition first, then walks through the classical functions that shaped deep learning.
Published:
If you cannot resample from the population, resample from your sample instead. That one substitution gives standard errors and intervals for statistics whose sampling distributions nobody can write down, and it fails in ways worth memorising.
Published:
A single Transformer block combines attention, residuals, layer norm, and an FFN into one reusable unit. Understanding this block is understanding the Transformer.
Published:
Without residual connections, training a 96-layer Transformer would be practically impossible. The skip connection is a simple addition that solves the vanishing gradient problem and enables arbitrary depth.
Published:
A sheaf is a mathematical object from algebraic topology that assigns vector spaces to cells and linear maps between them. On graphs, sheaves assign feature spaces to nodes and edges, with restriction maps encoding how node features relate across edges.
Published:
Reversing a corruption is generally intractable. Diffusion escapes by taking steps small enough that the reverse conditional is itself Gaussian, so the network only has to output a mean. Which mean it outputs, though, turns out to matter a great deal.
Published:
An estimator is a random variable, so it has a mean and a spread. Squared error splits exactly into those two pieces, and once you see the split, it becomes obvious that deliberately biasing an estimator can make it strictly better.
Published:
Fitting a line by least squares is not an optimisation trick, it is the orthogonal projection of the observation vector onto the column space of the design matrix, and once you see that, the normal equations, the failure modes, and the reason we solve by QR instead of inverting all follow from one picture.
Published:
Put an inner product on every tangent space and let it vary smoothly: that single object determines lengths, angles, distances, straight lines and volume. It also explains why a hierarchy embeds badly in Euclidean space and beautifully in hyperbolic space, the volume is in the wrong place.
Published:
Neural Sheaf Diffusion learns the restriction maps by gradient descent. Conn-NSD computes them once, before training, by assuming the data lies on a manifold and optimally aligning neighbouring tangent spaces, matching the learned models on small graphs at roughly half the cost per epoch.
Published:
A recurrent network shares weights across time exactly as a convolution shares them across space. The trouble is that gradients then travel through a product of Jacobians, and a product of a hundred numbers slightly below one is zero.
Published:
Traffic prediction is a canonical spatio-temporal graph task: sensors on roads form a fixed graph, and speed/volume measurements evolve over time. GNNs capture spatial correlations between sensors; RNNs or convolutions capture temporal patterns. Together they achieve state-of-the-art traffic forecasting.
Published:
Diffusion travels well between modalities, but not because it is generic. It fits wherever the answer is genuinely one-to-many, and the specific reason differs for a spectrogram, a protein backbone and a robot’s next ten actions.
Published:
Robots interact with structured environments: objects have relationships, joints form kinematic chains, agents communicate through interaction graphs. GNNs encode these relational structures, enabling generalisation across object configurations, robot morphologies, and multi-agent scenarios.
Published:
A summary statistic is a lossy compression of a sample. Knowing which information each one discards, and why sample variance divides by n minus 1 rather than n, is most of what descriptive statistics has to teach.
Published:
Positional encodings say where a node is in the graph. Structural encodings say what role it plays. They are complementary, and confusing them leads to poor design choices.
Published:
GAPE is a drop-in RoPE augmentation that adds content-aware attention logit biases: a query-gate suppresses irrelevant distant context while a key-gate preserves salient distant tokens. Provably sharper attention and improved long-context robustness, no architecture changes needed.
Published:
RoPE encodes position by rotating query and key vectors by an angle proportional to position. The clever result: absolute encoding produces relative attention for free, and it’s now the dominant PE for large language models.
Published:
RoPE encodes position by rotating query and key vectors by an angle proportional to position. The clever result: absolute encoding produces relative attention for free, and it’s now the dominant PE for large language models.
Published:
Translation is one group. Swap it for rotations, reflections, or the rigid motions of 3-D space and the same construction produces a different architecture, with the parameter count cut by exactly the size of the orbit.
Published:
Every sheaf network so far predicts restriction maps with an MLP on concatenated features, a universal approximator with no inductive bias for heterophily at all. Two alternatives drawn from opinion dynamics get the bias for free, and stop the parameter count scaling with the feature dimension.
Published:
Instead of soft cluster assignment (DiffPool), TopKPool and SAGPool select a subset of the most important nodes, producing a smaller but sparser graph at each level. Hard selection is scalable but requires careful score learning.
Published:
Once you see the reverse process as an ODE, the thousand-step sampling loop stops being a fact of life and becomes what it really is, a first-order solver with a terrible step size. Everything after that is numerical analysis.
Published:
Conditioning a diffusion model on a prompt is easy; making it actually obey the prompt is not. Classifier-free guidance fixes that by training one network to do two jobs and then extrapolating between its own answers.
Published:
GCN and GAT learn embeddings for fixed graphs, add a new node and you’re stuck. GraphSAGE (Hamilton et al., 2017) learns an aggregation function instead, so it can generate embeddings for entirely new nodes at inference time.
Published:
A 95% confidence interval does not say the parameter is 95% likely to be inside it. It says the recipe that produced the interval succeeds 95% of the time. That distinction is the single most-failed question in statistics interviews.
Published:
The restriction maps in a cellular sheaf can be constrained to different matrix classes: scalars, diagonal matrices, orthogonal matrices, or general matrices. Each class offers a different trade-off between expressivity and computational cost.
Published:
Message passing on a sparse graph does asymptotically less work than attention over every pair. It is still the slower one to train. This chapter is about why the architecture that wins is the one your hardware happens to like.
Published:
Dividing by √d_k is not just a trick, it prevents softmax from saturating and dying in high-dimensional spaces. Here’s the math and the intuition.
Published:
Computer vision tasks increasingly require relational reasoning, understanding how objects relate to each other, not just what they are. Scene graph generation, visual question answering, action recognition from skeletons, and 3D point cloud processing all benefit from GNN-based relational modelling.
Published:
A def is executed, not declared: it builds a function object once, evaluating the default arguments there and then. Everything odd about mutable defaults and late-binding closures follows from taking that seriously.
Published:
A Langevin step is a gradient step plus noise. Change the potential to the negative log-density and it samples from that density instead of minimising it, which is the entire sampler behind score-based generative models.
Published:
Noise prediction and score estimation are the same network in different units. Taking the step size to zero turns the whole method into a stochastic differential equation, and reveals a deterministic ODE with identical marginals hiding inside it.
Published:
Noise prediction and score estimation are the same network in different units. Taking the step size to zero turns the whole method into a stochastic differential equation, and reveals a deterministic ODE with identical marginals hiding inside it.
Published:
The Jacobian tells you how a map distorts volume, which is exactly the term normalising flows have to pay. The Hessian tells you the shape of the valley you are descending. Both are indispensable to reason with and, at a billion parameters, hopeless to form.
Published:
A recurrent network shares weights across time exactly as a convolution shares them across space. The trouble is that gradients then travel through a product of Jacobians, and a product of a hundred numbers slightly below one is zero.
Published:
Mean and sum readout treat all nodes equally. Attention readout learns which nodes matter most for a given task. Set2Set goes further, it uses an LSTM to iteratively query the node set, producing richer graph representations than single-pass pooling.
Published:
A dict trades memory for the ability to skip the search entirely. Everything that follows, why keys must be hashable, why lists cannot be keys, and why 1, 1.0 and True collide, comes from that single trade.
Published:
Training is one loop: measure the loss, ask backpropagation which way is downhill, take a small step. This chapter derives exactly how small that step has to be, why the answer is 2/a for a quadratic, and why reverse-mode differentiation gets you every gradient for roughly the price of one forward pass.
Published:
Standard GNNs assume neighbouring nodes should agree. Sheaf Neural Networks replace that assumption with a learned linear map on every edge, which turns heterophily, oversmoothing, and directional structure into one operator: the sheaf Laplacian.
Published:
Polynomial Neural Sheaf Diffusion (PNSD) replaces the fixed diffusion operator (I - Δ_F) with a learnable polynomial of the Sheaf Laplacian. This gives the model spectral flexibility, it can learn to amplify or suppress different frequency components of the sheaf signal.
Published:
Sheaves with orthogonal restriction maps define a connection on the graph, a parallel transport structure over edges. This connects sheaf GNNs to differential geometry and enables equivariant processing of data with local coordinate frames at each node.
Published:
Sheaf GNNs are the principled solution to heterophily: by learning per-edge maps that transform features before comparison, they can perform diffusion that converges within classes and diverges across classes, the exact opposite of standard GCN’s collapse.
Published:
A sheaf is a mathematical object from algebraic topology that assigns vector spaces to cells and linear maps between them. On graphs, sheaves assign feature spaces to nodes and edges, with restriction maps encoding how node features relate across edges.
Published:
Standard message passing aggregates neighbour features and averages. On heterophilic graphs (where neighbours often disagree), this is harmful. Cellular sheaves provide a mathematically principled framework to model per-edge relationships between node features, going beyond mere averaging.
Published:
GAT weights a neighbour by a scalar. SheafAN keeps the scalar and adds a learned orthogonal transport matrix alongside it, recovering GAT exactly at d = 1, and turning a model that goes numerically unstable past eight layers into one that runs to sixty-four.
Published:
Every sheaf neural network is built on an operator defined in this 2019 paper. It is not a machine learning paper but a programme for lifting spectral graph theory to cellular sheaves, and it is unusually candid about which parts of the lift fail.
Published:
The Sheaf Laplacian generalises the graph Laplacian by incorporating per-edge restriction maps. Its spectrum reveals how consistent data is under the sheaf. Sheaf diffusion with this Laplacian generalises GCN to handle heterophilic graphs.
Published:
The restriction maps in a cellular sheaf can be constrained to different matrix classes: scalars, diagonal matrices, orthogonal matrices, or general matrices. Each class offers a different trade-off between expressivity and computational cost.
Published:
Sheaf networks move features through matrix-valued maps but ignore the symmetries of physical space; equivariant GNNs respect those symmetries but move vectors with scalars. ESNN does both, learned, directed, matrix-valued transport that is exactly E(n)-equivariant, and proves that when displacement is the only geometric input, the radial–tangential family is all the linear transport there is.
Published:
Collaborative filtering represents every user and item as one static vector. Sheaf4Rec replaces each with a vector space, and reports consistent gains on ranking metrics, though the wins come from recall rather than precision, and the headline efficiency claim is hard to reconcile with the timing table.
Published:
A sheaf gives every node a matrix-valued say in how its neighbours reach it, but not in whether they do. Set a node’s restriction maps to zero to stop it listening and you also stop it speaking. Fixing that needs sheaves on directed graphs, and the fix costs the Laplacian its positive semi-definiteness.
Published:
A graph edge relates two things. A hyperedge relates any number of them, and hypergraph networks aggregate over it uniformly, every member contributes the same way. Attaching a sheaf gives each member its own linear map into the group, and turns forced consensus into apparent consensus.
Published:
If a sheaf neural network learns its geometry from data, it can learn the wrong geometry and have no way of knowing. Treating the sheaf Laplacian as a latent random variable fixes that, but requires a reparameterisable distribution on SO(n) with a tractable density, which did not exist.
Published:
Every sheaf network so far predicts restriction maps with an MLP on concatenated features, a universal approximator with no inductive bias for heterophily at all. Two alternatives drawn from opinion dynamics get the bias for free, and stop the parameter count scaling with the feature dimension.
Published:
Every sheaf model so far discretises the heat equation, which dissipates energy. Suk et al. try the wave equation instead, which conserves it, a one-line change of PDE with a clean theoretical motivation and a genuinely mixed empirical result.
Published:
Neural Sheaf Diffusion learns the restriction maps by gradient descent. Conn-NSD computes them once, before training, by assuming the data lies on a manifold and optimally aligning neighbouring tangent spaces, matching the learned models on small graphs at roughly half the cost per epoch.
Published:
Brain regions do not encode information in a shared feature space, which is exactly the assumption scalar message passing makes. BrainDyn puts learnable restriction maps between regions and integrates the result as a continuous-time system, the first pairing of cellular sheaves with neural ODEs.
Published:
Neural Sheaf Diffusion has a theoretical guarantee against representation collapse that does not survive contact with depth. DNSD diagnoses why, the Laplacian’s disagreement signal vanishes as diffusion succeeds, and replaces the operator rather than patching around it.
Published:
SheafPool solves a key missing piece in sheaf GNNs: graph-level pooling. Instead of averaging stalk vectors in arbitrary local bases, it aligns them into a shared canonical frame and builds a readout that is invariant to local basis changes.
Published:
PolyNSD replaces the NSD propagation operator with a degree-K Chebyshev polynomial in the normalised sheaf Laplacian, achieving SOTA on homo- and heterophilic benchmarks with only diagonal restriction maps and dramatically lower memory usage.
Published:
HetSheaf encodes graph heterogeneity directly in the sheaf data structure, type-aware stalks and restriction maps conditioned on node and edge types, instead of specialised architectural components, achieving +2pp on HGB with 10× fewer parameters.
Published:
SheafPool solves a key missing piece in sheaf GNNs: graph-level pooling. Instead of averaging stalk vectors in arbitrary local bases, it aligns them into a shared canonical frame and builds a readout that is invariant to local basis changes.
Published:
The previous chapter ended on an arbitrary choice that could not be eliminated. Gauge theory’s answer is to stop trying: keep every local frame, transport between them explicitly, and require the model to be indifferent to which frames you picked.
Published:
Shortest-path distances between nodes can be encoded as attention biases or node features, directly informing the model about graph proximity without requiring message passing.
Published:
An estimator is a random variable, so it has a mean and a spread. Squared error splits exactly into those two pieces, and once you see the split, it becomes obvious that deliberately biasing an estimator can make it strictly better.
Published:
Activation functions are the reason neural networks can model curved decision boundaries instead of collapsing into one giant linear map. This chapter builds the intuition first, then walks through the classical functions that shaped deep learning.
Published:
Logistic regression is not a squashed linear regression, it is a straight line drawn in log-odds space, which is why one coefficient means one multiplication of the odds, and why perfectly separable data drives that coefficient to infinity.
Published:
Laplacian eigenvectors are only defined up to sign: if u is an eigenvector, so is -u. This seemingly minor issue creates a fundamental problem for learning with LapPE. Here is the problem, its consequences, and how SignNet solves it.
Published:
Once ReLU became the default, researchers started asking a better question: can we keep the easy optimization while making the activation smoother, softer, and more expressive? This chapter covers the modern answers.
Published:
SGC removes all nonlinearities between GCN layers and collapses the entire propagation into a single pre-computed matrix power. Surprisingly, it matches GCN on most benchmarks, revealing that nonlinearities between layers may be unnecessary.
Published:
A simplex has four kinds of neighbour rather than one, and separating them lets a network see the difference between a filled triangle and an empty one, a distinction no graph neural network can make.
Published:
TDA computes a topological descriptor and hands it to a model; TDL makes the topological object the domain the model runs on. Telling the two apart is the difference between a preprocessing step and an architecture.
Published:
This book has two halves. One computes topological summaries of data and feeds them to a model. The other makes the topology itself the domain the network runs on. This guide sets out both, and the pipeline that connects them.
Published:
Robots interact with structured environments: objects have relationships, joints form kinematic chains, agents communicate through interaction graphs. GNNs encode these relational structures, enabling generalisation across object configurations, robot morphologies, and multi-agent scenarios.
Published:
The PE method from the 2017 ‘Attention Is All You Need’ paper uses sine and cosine waves at different frequencies. Learn why this elegant choice encodes position without any training.
Published:
The last activation in your network is not a modelling preference, it is a contract with your loss function. Break it and training stops meaning anything. Here is the contract, and what changes when the activation itself becomes learned.
Published:
Computer vision tasks increasingly require relational reasoning, understanding how objects relate to each other, not just what they are. Scene graph generation, visual question answering, action recognition from skeletons, and 3D point cloud processing all benefit from GNN-based relational modelling.
Published:
Without residual connections, training a 96-layer Transformer would be practically impossible. The skip connection is a simple addition that solves the vanishing gradient problem and enables arbitrary depth.
Published:
Put an inner product on every tangent space and let it vary smoothly: that single object determines lengths, angles, distances, straight lines and volume. It also explains why a hierarchy embeds badly in Euclidean space and beautifully in hyperbolic space, the volume is in the wrong place.
Published:
A Python list is a growable array of pointers, and that one implementation fact predicts every complexity in its API: appending is cheap, inserting at the front is not, and in costs a full scan.
Published:
Most GNN research assumes a fixed graph. Real graphs evolve: edges appear and disappear, node features drift, new nodes arrive. Dynamic graph learning addresses how to model and predict on graphs whose structure changes over time.
Published:
Social networks are large sparse graphs with rich node features (user profiles) and heterogeneous edges (friendship, follow, retweet). GNNs predict user behaviour, detect communities, identify influential spreaders, and flag misinformation, tasks with significant real-world impact.
Published:
DiffPool learns to hierarchically cluster nodes into super-nodes across layers, like a convolutional pyramid for graphs. Unlike flat global pooling, it captures multi-scale graph structure by differentiably assigning nodes to clusters.
Published:
The last activation in your network is not a modelling preference, it is a contract with your loss function. Break it and training stops meaning anything. Here is the contract, and what changes when the activation itself becomes learned.
Published:
Dividing by √d_k is not just a trick, it prevents softmax from saturating and dying in high-dimensional spaces. Here’s the math and the intuition.
Published:
Logistic regression is not a squashed linear regression, it is a straight line drawn in log-odds space, which is why one coefficient means one multiplication of the odds, and why perfectly separable data drives that coefficient to infinity.
Published:
No comparison sort can beat n log n, that is a theorem with a two-line counting proof, not a limit of current cleverness. Counting sort beats it anyway, because it never compares two elements. Also: how to write a binary search whose boundaries are right.
Published:
Message passing on a sparse graph does asymptotically less work than attention over every pair. It is still the slower one to train. This chapter is about why the architecture that wins is the one your hardware happens to like.
Published:
Instead of soft cluster assignment (DiffPool), TopKPool and SAGPool select a subset of the most important nodes, producing a smaller but sparser graph at each level. Hard selection is scalable but requires careful score learning.
Published:
The last activation in your network is not a modelling preference, it is a contract with your loss function. Break it and training stops meaning anything. Here is the contract, and what changes when the activation itself becomes learned.
Published:
Every sheaf neural network is built on an operator defined in this 2019 paper. It is not a machine learning paper but a programme for lifting spectral graph theory to cellular sheaves, and it is unusually candid about which parts of the lift fail.
Published:
L1 regularisation produces exact zeros and L2 does not. The reason is not statistical, it is geometric: the L1 unit ball has corners on the axes, and corners are what optimisation solutions stick to.
Published:
Traffic prediction is a canonical spatio-temporal graph task: sensors on roads form a fixed graph, and speed/volume measurements evolve over time. GNNs capture spatial correlations between sensors; RNNs or convolutions capture temporal patterns. Together they achieve state-of-the-art traffic forecasting.
Published:
Spatio-temporal GNNs combine spatial message passing with temporal sequence modelling. They are the dominant approach for traffic forecasting, weather prediction, and any task where measurements at sensor nodes evolve over time on a fixed graph.
Published:
Polynomial Neural Sheaf Diffusion (PNSD) replaces the fixed diffusion operator (I - Δ_F) with a learnable polynomial of the Sheaf Laplacian. This gives the model spectral flexibility, it can learn to amplify or suppress different frequency components of the sheaf signal.
Published:
The Sheaf Laplacian generalises the graph Laplacian by incorporating per-edge restriction maps. Its spectrum reveals how consistent data is under the sheaf. Sheaf diffusion with this Laplacian generalises GCN to handle heterophilic graphs.
Published:
| ChebNet avoids the expensive full eigendecomposition by approximating spectral filters with Chebyshev polynomials, achieving O( | E | ) computation and spatial locality without sacrificing expressiveness. |
Published:
The Graph Fourier Transform decomposes a signal on a graph into frequency components using the Laplacian’s eigenvectors. This spectral view is the mathematical foundation behind spectral GNNs like ChebNet and GCN.
Published:
GCN (Kipf & Welling, 2016) is the ‘hello world’ of GNNs. It simplifies spectral graph convolution into a single elegant layer: normalised neighbourhood averaging with a learned linear transformation.
Published:
The Graph Laplacian is L = D - A. Its eigenvectors reveal the graph’s community structure; its eigenvalues tell you how well-connected the graph is. It’s also the mathematical bridge from spectral theory to GNNs like GCN.
Published:
PolyNSD replaces the NSD propagation operator with a degree-K Chebyshev polynomial in the normalised sheaf Laplacian, achieving SOTA on homo- and heterophilic benchmarks with only diagonal restriction maps and dramatically lower memory usage.
Published:
Laplacian eigenvector positional encodings tell a node where it sits in the graph, but the graph Laplacian only knows adjacency, so two structurally identical nodes get identical encodings no matter how different their features are. Swap in the sheaf Laplacian and the features enter the spectrum.
Published:
Every sheaf neural network is built on an operator defined in this 2019 paper. It is not a machine learning paper but a programme for lifting spectral graph theory to cellular sheaves, and it is unusually candid about which parts of the lift fail.
Published:
Eigenvectors are the directions a matrix does not rotate, when they exist. The SVD asks a weaker question that always has an answer, and that is exactly why it, not the eigendecomposition, is the workhorse of applied linear algebra.
Published:
SE(3)-Transformers extend self-attention to 3D point clouds and molecular graphs while maintaining SE(3) equivariance. Attention weights are learned between node pairs; values are equivariant features built from spherical harmonics.
Published:
Most of the bits in a photograph encode texture no one can see. Latent diffusion throws them away first with an autoencoder, then runs the entire diffusion process in a space roughly forty-eight times smaller, which is how Stable Diffusion fits on a consumer GPU.
Published:
A heap is a tree with no pointers, just an array and two index formulas, and building one from n items takes O(n), not O(n log n). The sum that proves it is the most quotable derivation in this book.
Published:
A sheaf is a mathematical object from algebraic topology that assigns vector spaces to cells and linear maps between them. On graphs, sheaves assign feature spaces to nodes and edges, with restriction maps encoding how node features relate across edges.
Published:
A 95% confidence interval does not say the parameter is 95% likely to be inside it. It says the recipe that produced the interval succeeds 95% of the time. That distinction is the single most-failed question in statistics interviews.
Published:
Counting microstates gives you the Boltzmann distribution, and the Boltzmann distribution gives you softmax, simulated annealing and energy-based models. The partition function is not a bookkeeping constant, it is the object that contains every thermodynamic quantity, and it is intractable for exactly that reason.
Published:
Diffusion models, energy-based models and Hamiltonian Monte Carlo were not inspired by physics, they are physics, rewritten with a neural network in place of an analytic potential. Knowing which physics saves you from re-deriving it badly.
Published:
A p-value answers one narrow question: if nothing were going on, how often would data look at least this extreme? It says nothing about whether the effect is real, large, or worth shipping, and running twenty of them changes the meaning of all twenty.
Published:
Probability runs forwards: pick a model, predict the data. Statistics runs backwards, and backwards is harder, many models could have produced what you saw. Everything else in this book is machinery for handling that ambiguity honestly.
Published:
Translation is one group. Swap it for rotations, reflections, or the rigid motions of 3-D space and the same construction produces a different architecture, with the parameter count cut by exactly the size of the orbit.
Published:
Most GNN research assumes a fixed graph. Real graphs evolve: edges appear and disappear, node features drift, new nodes arrive. Dynamic graph learning addresses how to model and predict on graphs whose structure changes over time.
Published:
Strings cannot be modified, so every method that looks like it edits one actually builds another. That single fact explains the whole str API, and why growing a string inside a loop can quietly become quadratic.
Published:
Positional encodings say where a node is in the graph. Structural encodings say what role it plays. They are complementary, and confusing them leads to poor design choices.
Published:
Message-passing GNNs are permutation-equivariant by design, they cannot assign unique positions to nodes. Without positional encodings, symmetric nodes are indistinguishable. Here is why that matters and how to fix it.
Published:
Graphormer encodes graph structure directly into Transformer attention via three biases: node centrality, spatial encoding (shortest paths), and edge encoding. It won the OGB-LSC 2021 competition on molecular property prediction.
Published:
Z-SASLM is a zero-shot, fine-tuning-free style blending pipeline that replaces linear latent interpolation with SLERP along the geodesic of the hypersphere, preserving latent manifold structure when blending multiple styles. Published at CVPR 2025 Workshop.
Published:
Every later book on this site assumes you already know what a loss is, why gradient descent works, and what a convolution buys you. This book supplies that, and follows one thread through it: how much structure you build in versus how much you let the data decide.
Published:
Curvature at a point is one over the radius of the circle that best hugs the curve there. Push that idea up to surfaces and Gauss’s Theorema Egregium falls out: some curvature is visible from inside the surface, which is why no map projection can ever get distances right.
Published:
A matrix is not a table of numbers, it is a deformation of space. The SVD says every deformation is the same three moves in sequence: rotate, stretch along axes, rotate again.
Published:
Eigenvectors are the directions a matrix does not rotate, when they exist. The SVD asks a weaker question that always has an answer, and that is exactly why it, not the eigendecomposition, is the workhorse of applied linear algebra.
Published:
PCA can be derived by asking for the directions of greatest spread, or by asking for the subspace that loses the least when you project onto it. The two questions look unrelated and have the same answer, which is the most useful thing to understand about it.
Published:
One bilinear form generates the whole of flat geometry: lengths, angles, orthogonality, projections and the distance from a point to a hyperplane. Get the projection formula and you get the SVM margin for free.
Published:
Among all the hyperplanes that separate two classes, one sits furthest from both. Finding it turns out to depend on the data only through inner products, and that single fact is what lets you work in a space you never build.
Published:
The last activation in your network is not a modelling preference, it is a contract with your loss function. Break it and training stops meaning anything. Here is the contract, and what changes when the activation itself becomes learned.
Published:
Once ReLU became the default, researchers started asking a better question: can we keep the easy optimization while making the activation smoother, softer, and more expressive? This chapter covers the modern answers.
Published:
Energy is conserved because the laws of physics do not care what time it is. That single sentence is Noether’s theorem, and its machine learning descendant is the reason an equivariant network needs less data than one that must learn the symmetry from examples.
Published:
Every embedding you have ever trained lives in a metric space, every dataset you have ever fitted sits near a surface far thinner than its ambient dimension, and every architecture you trust encodes a symmetry. Geometry is not decoration on top of ML, it is what makes the problems tractable.
Published:
A graph’s nodes have no canonical order, so any model that reads one must give the same answer under relabelling. That single requirement forces the three-step message, aggregate, update template, it is not a design choice.
Published:
CNNs, GNNs, Transformers and sheaf models look like separate inventions. They are the same recipe applied to different domains: identify the symmetry of your data, then build layers that respect it. This book is that recipe, and the arguments that follow from it.
Published:
Equivariance formalises the idea that a function should ‘commute with symmetry transformations.’ A rotation-equivariant model applied to rotated input gives the rotated output, no extra training needed. This is the foundation for geometric deep learning.
Published:
Many real-world graphs are embedded in 3D space, molecules, proteins, point clouds, crystal structures. Standard GNNs ignore coordinates and only use connectivity. Geometric GNNs incorporate spatial positions and must respect physical symmetries.
Published:
Message-passing GNNs are permutation-equivariant by design, they cannot assign unique positions to nodes. Without positional encodings, symmetric nodes are indistinguishable. Here is why that matters and how to fix it.
Published:
Sheaf networks move features through matrix-valued maps but ignore the symmetries of physical space; equivariant GNNs respect those symmetries but move vectors with scalars. ESNN does both, learned, directed, matrix-valued transport that is exactly E(n)-equivariant, and proves that when displacement is the only geometric input, the radial–tangential family is all the linear transport there is.
Published:
Hamiltonian Monte Carlo is not a heuristic that happens to move well. It is exact because Hamiltonian flow preserves phase-space volume and is reversible, which is what makes the Metropolis acceptance ratio collapse to a difference of energies.
Published:
A 95% confidence interval does not say the parameter is 95% likely to be inside it. It says the recipe that produced the interval succeeds 95% of the time. That distinction is the single most-failed question in statistics interviews.
Published:
A manifold is a space that looks flat close up and can be curved or knotted globally. That gap between local and global is why charts exist, why a single autoencoder latent space sometimes cannot fit the data, and what dimensionality reduction is really exploiting.
Published:
Activation functions are the reason neural networks can model curved decision boundaries instead of collapsing into one giant linear map. This chapter builds the intuition first, then walks through the classical functions that shaped deep learning.
Published:
TDA computes a topological descriptor and hands it to a model; TDL makes the topological object the domain the model runs on. Telling the two apart is the difference between a preprocessing step and an architecture.
Published:
Counting microstates gives you the Boltzmann distribution, and the Boltzmann distribution gives you softmax, simulated annealing and energy-based models. The partition function is not a bookkeeping constant, it is the object that contains every thermodynamic quantity, and it is intractable for exactly that reason.
Published:
TGN (Temporal Graph Network) is the leading framework for continuous-time dynamic graphs. It maintains a per-node memory that is updated upon each interaction, enabling efficient inductive link prediction on event streams.
Published:
Most GNN research assumes a fixed graph. Real graphs evolve: edges appear and disappear, node features drift, new nodes arrive. Dynamic graph learning addresses how to model and predict on graphs whose structure changes over time.
Published:
Most knowledge graphs treat facts as timeless, but facts change. Barack Obama was president from 2009 to 2017. Temporal Knowledge Graphs add timestamps to triples, requiring models to reason about what was true when.
Published:
Tensor Field Networks (TFN) were the first architecture to achieve SE(3) equivariance using spherical harmonics and Clebsch-Gordan tensor products. They laid the theoretical foundation for NequIP and MACE, the current state-of-the-art in equivariant molecular force fields.
Published:
Curvature at a point is one over the radius of the circle that best hugs the curve there. Push that idea up to surfaces and Gauss’s Theorema Egregium falls out: some curvature is visible from inside the surface, which is why no map projection can ever get distances right.
Published:
Thermodynamic entropy and Shannon entropy differ by a constant with units. Once you accept that, the ELBO stops being an inference trick and becomes a free energy, and diffusion models stop being a clever architecture and become a driven non-equilibrium process.
Published:
Diffusion models, energy-based models and Hamiltonian Monte Carlo were not inspired by physics, they are physics, rewritten with a neural network in place of an analytic potential. Knowing which physics saves you from re-deriving it badly.
Published:
Python names are references, so passing a list to a function hands over the original. Python threads hold one interpreter lock, so two CPU-bound threads take as long as one after the other. Both facts explain bugs you have already had.
Published:
Most knowledge graphs treat facts as timeless, but facts change. Barack Obama was president from 2009 to 2017. Temporal Knowledge Graphs add timestamps to triples, requiring models to reason about what was true when.
Published:
Instead of soft cluster assignment (DiffPool), TopKPool and SAGPool select a subset of the most important nodes, producing a smaller but sparser graph at each level. Hard selection is scalable but requires careful score learning.
Published:
A simplicial complex cannot hold a benzene ring as a single cell, filling the hexagon costs three edges between atoms that share no bond, and this one constraint is what cell and combinatorial complexes exist to remove.
Published:
A simplex has four kinds of neighbour rather than one, and separating them lets a network see the difference between a filled triangle and an empty one, a distinction no graph neural network can make.
Published:
TDA computes a topological descriptor and hands it to a model; TDL makes the topological object the domain the model runs on. Telling the two apart is the difference between a preprocessing step and an architecture.
Published:
This book has two halves. One computes topological summaries of data and feeds them to a model. The other makes the topology itself the domain the network runs on. This guide sets out both, and the pipeline that connects them.
Published:
This book has two halves. One computes topological summaries of data and feeds them to a model. The other makes the topology itself the domain the network runs on. This guide sets out both, and the pipeline that connects them.
Published:
A graph edge relates two things. A hyperedge relates any number of them, and hypergraph networks aggregate over it uniformly, every member contributes the same way. Attaching a sheaf gives each member its own linear map into the group, and turns forced consensus into apparent consensus.
Published:
Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first, which is faster, shorter, and free of race conditions.
Published:
Traffic prediction is a canonical spatio-temporal graph task: sensors on roads form a fixed graph, and speed/volume measurements evolve over time. GNNs capture spatial correlations between sensors; RNNs or convolutions capture temporal patterns. Together they achieve state-of-the-art traffic forecasting.
Published:
Spatio-temporal GNNs combine spatial message passing with temporal sequence modelling. They are the dominant approach for traffic forecasting, weather prediction, and any task where measurements at sensor nodes evolve over time on a fixed graph.
Published:
The DDPM objective starts as a variational bound with T+1 KL terms and ends as a plain mean-squared error on noise. Following the collapse shows why the discarded weighting term is not an approximation you tolerate but a reweighting that improves samples.
Published:
Layer norm is not optional plumbing. It determines training stability, gradient flow, and whether deep Transformers converge at all. Pre-LN vs Post-LN is not a detail, it changes training dynamics fundamentally.
Published:
APPNP decouples feature transformation from propagation. A neural network transforms features first; then Personalized PageRank propagates the result. This enables deep propagation without over-smoothing.
Published:
A matrix is not a table of numbers, it is a deformation of space. The SVD says every deformation is the same three moves in sequence: rotate, stretch along axes, rotate again.
Published:
A single Transformer block combines attention, residuals, layer norm, and an FFN into one reusable unit. Understanding this block is understanding the Transformer.
Published:
Write self-attention in the message-passing template and you get the Graph Attention Network equations with one substitution: the neighbourhood becomes the whole input. The two architectures are not analogous, they are the same operator on different graphs.
Published:
GAPE is a drop-in RoPE augmentation that adds content-aware attention logit biases: a query-gate suppresses irrelevant distant context while a key-gate preserves salient distant tokens. Provably sharper attention and improved long-context robustness, no architecture changes needed.
Published:
Convolution is not a clever idea someone had about images. It is the only linear map that commutes with translation, a theorem, not a design choice, and one you can verify by exhaustion on a small enough case.
Published:
A binary search tree gives O(log n) search only if it is balanced, and inserting sorted keys, the most natural thing a caller ever does, makes it a linked list with extra pointers. Here is what each traversal is for, and what AVL and red-black trees actually guarantee.
Published:
A binary search tree gives O(log n) search only if it is balanced, and inserting sorted keys, the most natural thing a caller ever does, makes it a linked list with extra pointers. Here is what each traversal is for, and what AVL and red-black trees actually guarantee.
Published:
A binary search tree gives O(log n) search only if it is balanced, and inserting sorted keys, the most natural thing a caller ever does, makes it a linked list with extra pointers. Here is what each traversal is for, and what AVL and red-black trees actually guarantee.
Published:
PCA can be derived by asking for the directions of greatest spread, or by asking for the subspace that loses the least when you project onto it. The two questions look unrelated and have the same answer, which is the most useful thing to understand about it.
Published:
A Python list is a growable array of pointers, and that one implementation fact predicts every complexity in its API: appending is cheap, inserting at the front is not, and in costs a full scan.
Published:
A def is executed, not declared: it builds a function object once, evaluating the default arguments there and then. Everything odd about mutable defaults and late-binding closures follows from taking that seriously.
Published:
A Python variable is not a container that holds a value, it is a label stuck onto an object that lives somewhere else. Almost every early surprise, from shared lists to 0.1 + 0.2, follows from taking that sentence literally.
pip install Published:
Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed, most of what people reach for a package to solve is one import away.
Published:
If a sheaf neural network learns its geometry from data, it can learn the wrong geometry and have no way of knowing. Treating the sheaf Laplacian as a latent random variable fixes that, but requires a reparameterisable distribution on SO(n) with a tractable density, which did not exist.
Published:
Strings cannot be modified, so every method that looks like it edits one actually builds another. That single fact explains the whole str API, and why growing a string inside a loop can quietly become quadratic.
Published:
The MPNN framework (Gilmer et al., 2017) unifies GCN, GAT, GIN, GraphSAGE, and almost all spatial GNNs under one abstraction: message functions, aggregation, and update. Understanding MPNN means understanding the whole GNN family.
Published:
Dijkstra is greedy: it finalises the closest unvisited vertex and never revisits it. That is valid only because every edge adds weight, put one negative edge in the graph and the algorithm returns a wrong answer without complaining.
Published:
Every later book on this site assumes you already know what a loss is, why gradient descent works, and what a convolution buys you. This book supplies that, and follows one thread through it: how much structure you build in versus how much you let the data decide.
Published:
Reversing a corruption is generally intractable. Diffusion escapes by taking steps small enough that the reverse conditional is itself Gaussian, so the network only has to output a mean. Which mean it outputs, though, turns out to matter a great deal.
Published:
A recurrent network shares weights across time exactly as a convolution shares them across space. The trouble is that gradients then travel through a product of Jacobians, and a product of a hundred numbers slightly below one is zero.
Published:
A Python variable is not a container that holds a value, it is a label stuck onto an object that lives somewhere else. Almost every early surprise, from shared lists to 0.1 + 0.2, follows from taking that sentence literally.
Published:
Expectation adds up no matter how tangled the dependence. Variance does not, and the correction term, covariance, is where most of the interesting behaviour of ensembles, portfolios and minibatch gradients lives.
Published:
A summary statistic is a lossy compression of a sample. Knowing which information each one discards, and why sample variance divides by n minus 1 rather than n, is most of what descriptive statistics has to teach.
Published:
If a sheaf neural network learns its geometry from data, it can learn the wrong geometry and have no way of knowing. Treating the sheaf Laplacian as a latent random variable fixes that, but requires a reparameterisable distribution on SO(n) with a tractable density, which did not exist.
Published:
Newton says a particle moves because a force pushes it. Lagrange says it moves along the path that makes the action stationary. The second statement is harder to believe and far easier to use, and it is the one machine learning inherited.
Published:
Diffusion travels well between modalities, but not because it is generic. It fits wherever the answer is genuinely one-to-many, and the specific reason differs for a spectrogram, a protein backbone and a robot’s next ten actions.
Published:
An import is not a textual include, it executes a file once, caches the result, and binds a name. Almost every confusing import error, from circular imports to ‘attempted relative import with no known parent package’, follows directly from that one sentence.
Published:
Python is not interpreted line by line, and python is not the language. Knowing what CPython compiles your file into, and where it puts the packages you install, explains most of the confusion beginners hit in their first week.
Published:
Computer vision tasks increasingly require relational reasoning, understanding how objects relate to each other, not just what they are. Scene graph generation, visual question answering, action recognition from skeletons, and 3D point cloud processing all benefit from GNN-based relational modelling.
Published:
Every sheaf model so far discretises the heat equation, which dissipates energy. Suk et al. try the wave equation instead, which conserves it, a one-line change of PDE with a clean theoretical motivation and a genuinely mixed empirical result.
Published:
Not all graphs are equal. Directed edges, edge weights, multiple node/edge types, each variant requires different GNN design choices.
Published:
How powerful can a GNN be? Xu et al. (2019) answered with a theoretical bound, and GIN is the architecture that achieves it. The secret: use sum aggregation and an MLP, not mean or max.