ML Blog
Welcome to my research blog โ structured like a library of books. Each book covers a major AI topic; every chapter is a short, self-contained post you can read in 3โ5 minutes. Start with the Start Here overview of any book, then dive into whichever chapters interest you most.
Activation Functions in Neural Networks: Why Non-Linearity Matters
Activation functions are the reason neural networks can model curved decision boundaries instead of collapsing into one giant linear map. This chapter builds the intuition first, then walks through the class...
Output, Gated, and Special Activations: Softmax, GLU, SIREN, and More
Not every activation is a hidden-layer curve. Some produce probabilities, some implement learned gates...
Modern Activation Functions: GELU, SiLU, Mish, and Smooth Gating
Once ReLU became the default, researchers started asking a better question: can we keep the easy optim...
Activation Functions in Neural Networks: Why Non-Linearity Matters
Activation functions are the reason neural networks can model curved decision boundaries instead of co...
Transformers: The Architecture That Changed AI
A self-contained guide to the Transformer โ the engine behind GPT, BERT, and modern AI. Learn how attention replaces recurrence and why every major AI system uses it.
The Transformer Block: Putting It All Together
A single Transformer block combines attention, residuals, layer norm, and an FFN into one reusable uni...
Feed-Forward Networks: The Forgotten Half of Transformers
The FFN block holds two-thirds of a Transformer's parameters and does most of its factual recall. Yet ...
Residual Connections: Why Transformers Can Be Deep
Without residual connections, training a 96-layer Transformer would be practically impossible. The ski...
Layer Normalization in Transformers
Layer norm is not optional plumbing. It determines training stability, gradient flow, and whether deep...
Encoder vs Decoder vs Encoder-Decoder Transformers
BERT, GPT, and T5 are all Transformers โ but their architectures are fundamentally different. One comp...
Cross-Attention: How Models Attend to Another Sequence
Cross-attention lets one sequence query information from a completely different sequence. It is the br...
Attention Masks: Causal, Padding, and Bidirectional
The difference between GPT, BERT, and T5 is largely a masking decision. Learn how causal, padding, and...
Query, Key, Value: The Intuition Behind QKV
Q, K, and V are not arbitrary labels. They map precisely onto search queries, database labels, and ret...
Scaled Dot-Product Attention: Why the โd Matters
Dividing by โd_k is not just a trick โ it prevents softmax from saturating and dying in high-dimension...
Multi-Head Attention: Many Eyes on the Data
One attention head sees one relationship. Multiple heads running in parallel let the model capture syn...
Self-Attention: Teaching Machines to Focus
Self-attention is the core of every Transformer. Learn how Query, Key, and Value vectors let every tok...
GAPE: Remember to Forget โ Gated Adaptive Positional Encoding
GAPE is a drop-in RoPE augmentation that adds content-aware attention logit biases: a query-gate suppr...
LongRoPE: Extending Context to 2 Million Tokens
LongRoPE (Microsoft, 2024) pushes RoPE-based context to 2M tokens by searching for optimal per-dimensi...
YaRN: Yet Another RoPE Extensionn Method
YaRN combines NTK scaling for high-frequency dimensions with linear interpolation for low-frequency on...
NTK-Aware Scaling: Extending Context Without Fine-Tuning
NTK-Aware Scaling extends the context window of RoPE-based models by rescaling frequencies using Neura...
ALiBi: Attention with Linear Biases
ALiBi skips traditional positional embeddings entirely and just subtracts a distance penalty from atte...
RoPE: Rotary Position Embeddings
RoPE encodes position by rotating query and key vectors by an angle proportional to position. The clev...
Relative Positional Encodings: It's All About Distance
Instead of asking 'where am I?', relative PEs ask 'how far are these two tokens apart?' Shaw et al. an...
Learned Positional Encodings: Data-Driven Position
Instead of a fixed formula, why not just train position embeddings from scratch โ like word embeddings...
Sinusoidal Positional Encodings: The Original Solution
The PE method from the 2017 'Attention Is All You Need' paper uses sine and cosine waves at different ...
Positional Encodings: Why Position Matters
Transformers see all tokens at once โ which means without help they'd treat 'cat ate mouse' and 'mouse...
FoPE: Fourier Position Embedding for Length Generalization
FoPE rethinks long-context positional encoding from a frequency-domain perspective. Instead of only st...
Position Interpolation: Extending RoPE with Minimal Fine-Tuning
Position Interpolation rescales positions before applying RoPE so a model trained on short contexts ca...
XPos: Length-Extrapolatable Rotary Embeddings
XPos modifies RoPE with a multiplicative decay that keeps relative rotations while stabilising magnitu...
p-RoPE: What Makes Rotary Positional Encodings Useful?
This paper does two things at once: it explains what RoPE is really doing inside a trained LLM, and it...
Graph Neural Networks: Learning on Graphs
Graphs are everywhere โ molecules, social networks, road maps, knowledge bases. Graph Neural Networks learn from this relational structure by propagating information between connected nodes. Here's the compl...
The Graph Adjacency Matrix: A Graph in Matrix Form
Before understanding GNNs, you need to understand how graphs are represented mathematically. The adjac...
The Graph Laplacian: Spectral Graph Theory Explained Simply
The Graph Laplacian is L = D - A. Its eigenvectors reveal the graph's community structure; its eigenva...
What Is a Graph? Nodes, Edges, Features, and Labels
A graph is a set of nodes connected by edges โ but the power of GNNs comes from the features attached ...
Directed, Undirected, Weighted, and Heterogeneous Graphs
Not all graphs are equal. Directed edges, edge weights, multiple node/edge types โ each variant requir...
Homophily vs Heterophily: When Neighbours Are Similar or Different
Most GNNs assume nearby nodes are similar โ the homophily assumption. When this breaks (heterophilic g...
Graph Tasks: Node, Edge, and Graph-Level Prediction
GNNs can predict at three levels: properties of individual nodes, existence or type of edges, or prope...
Graph Fourier Transform: The Spectral View of Graphs
The Graph Fourier Transform decomposes a signal on a graph into frequency components using the Laplaci...
Message Passing: The Universal GNN Framework
Every GNN โ GCN, GAT, GraphSAGE, GIN โ is a special case of message passing. Learn the three-step loop...
GCN: Graph Convolutional Networks
GCN (Kipf & Welling, 2016) is the 'hello world' of GNNs. It simplifies spectral graph convolution into...
GAT: Graph Attention Networks
GCN assigns the same (degree-based) weight to every neighbour. GAT learns which neighbours actually ma...
GraphSAGE: Inductive Learning on Large Graphs
GCN and GAT learn embeddings for fixed graphs โ add a new node and you're stuck. GraphSAGE (Hamilton e...
GIN: Graph Isomorphism Network โ The Most Expressive GNN
How powerful can a GNN be? Xu et al. (2019) answered with a theoretical bound โ and GIN is the archite...
ChebNet: Spectral Graph Convolutions via Chebyshev Polynomials
ChebNet avoids the expensive full eigendecomposition by approximating spectral filters with Chebyshev ...
SGC: Simple Graph Convolution
SGC removes all nonlinearities between GCN layers and collapses the entire propagation into a single p...
APPNP: Personalized PageRank Meets Graph Neural Networks
APPNP decouples feature transformation from propagation. A neural network transforms features first; t...
Graph Transformers: Bringing Attention to Graphs
Graph Transformers replace or augment local message passing with full pairwise attention โ every node ...
Graphormer: Transformers with Structural Biases for Graphs
Graphormer encodes graph structure directly into Transformer attention via three biases: node centrali...
MPNN: The General Message Passing Neural Network Framework
The MPNN framework (Gilmer et al., 2017) unifies GCN, GAT, GIN, GraphSAGE, and almost all spatial GNNs...
The Weisfeiler-Lehman Test: How Powerful Are GNNs?
The 1-WL graph isomorphism test provides the exact upper bound on message-passing GNN expressivity. GI...
Oversmoothing: When All Node Embeddings Become the Same
Stack enough GNN layers and all node embeddings converge to the same vector โ making the model useless...
Oversquashing: When Too Much Information Passes Through Bottlenecks
Oversquashing occurs when exponentially many node features must be compressed into a fixed-size embedd...
Over-smoothing vs Over-squashing: The Difference
Oversmoothing and oversquashing are both problems with deep GNNs, but they affect different nodes, hav...
Depth in GNNs: Why Deeper Is Not Always Better
In Transformers, depth = expressiveness. In GNNs, depth = both expressiveness AND over-smoothing. The ...
Why GNNs Need Positional Encodings
Message-passing GNNs are permutation-equivariant by design โ they cannot assign unique positions to no...
Laplacian Eigenvectors as Graph Positional Encodings
The k smallest eigenvectors of the graph Laplacian form a natural positional embedding space โ the gra...
Random Walk Positional Encodings
Random walk positional encodings encode each node's structural context by computing the probability of...
Shortest-Path Encodings for Graph Transformers
Shortest-path distances between nodes can be encoded as attention biases or node features โ directly i...
Structural vs Positional Encodings in Graphs
Positional encodings say where a node is in the graph. Structural encodings say what role it plays. Th...
Sign Ambiguity in Laplacian Eigenvectors
Laplacian eigenvectors are only defined up to sign: if u is an eigenvector, so is -u. This seemingly m...
Global Pooling in GNNs: Mean, Sum, and Max
To predict a property of an entire graph, node embeddings must be aggregated into a single vector. The...
DiffPool: Learning Hierarchical Graph Pooling
DiffPool learns to hierarchically cluster nodes into super-nodes across layers โ like a convolutional ...
TopKPool and SAGPool: Sparse Graph Pooling
Instead of soft cluster assignment (DiffPool), TopKPool and SAGPool select a subset of the most import...
Set2Set and Attention Readout: Order-Invariant Graph Summaries
Mean and sum readout treat all nodes equally. Attention readout learns which nodes matter most for a g...
Graph Classification: From Node Embeddings to Graph Embeddings
Graph classification is the task of predicting a label for an entire graph. It requires composing mess...
Heterogeneous Graphs: When Nodes and Edges Have Types
Most real-world graphs are heterogeneous โ they contain multiple node types (users, items, tags) and e...
R-GCN: Relational Graph Convolutional Networks
R-GCN extends GCN to multi-relational graphs by learning a separate weight matrix for each relation ty...
HAN: Heterogeneous Graph Attention Networks
HAN combines meta-path decomposition with two levels of attention: node-level attention weights neighb...
Knowledge Graph Embeddings vs GNNs
Knowledge graph completion can be solved with shallow KG embeddings (TransE, DistMult, ComplEx) or wit...
Temporal Knowledge Graphs: Facts That Change Over Time
Most knowledge graphs treat facts as timeless โ but facts change. Barack Obama was president from 2009...
Static vs Dynamic Graphs: When Structure Changes Over Time
Most GNN research assumes a fixed graph. Real graphs evolve: edges appear and disappear, node features...
Temporal Graph Networks: Learning from Events
TGN (Temporal Graph Network) is the leading framework for continuous-time dynamic graphs. It maintains...
Graph Neural ODEs: Continuous-Time Graph Dynamics
Neural ODEs replace discrete layer-by-layer computation with continuous dynamics governed by a differe...
Spatio-Temporal GNNs: Learning on Graphs Through Time
Spatio-temporal GNNs combine spatial message passing with temporal sequence modelling. They are the do...
Why Geometry Matters in Graph Neural Networks
Many real-world graphs are embedded in 3D space โ molecules, proteins, point clouds, crystal structure...
Equivariance: What It Means and Why It Matters
Equivariance formalises the idea that a function should 'commute with symmetry transformations.' A rot...
EGNN: E(n)-Equivariant Graph Neural Networks
EGNN achieves E(n)-equivariance with a simple update rule: positions updated via weighted sums of rela...
SE(3)-Transformers: Attention with 3D Symmetry
SE(3)-Transformers extend self-attention to 3D point clouds and molecular graphs while maintaining SE(...
Tensor Field Networks and Geometric Deep Learning
Tensor Field Networks (TFN) were the first architecture to achieve SE(3) equivariance using spherical ...
Molecular GNNs: Learning on Atoms and Bonds
Molecules are graphs. Molecular GNNs predict chemical properties from structure. The best models use 3...
GNNs for Molecules: Drug Discovery and Material Design
Graph neural networks are transforming computational drug discovery. Molecules are natural graphs, and...
GNNs for Recommender Systems
Recommendation is naturally a graph problem: users and items are nodes, interactions are edges. GNNs o...
GNNs for Social Networks: Influence, Communities, and Misinformation
Social networks are large sparse graphs with rich node features (user profiles) and heterogeneous edge...
GNNs for Traffic Forecasting
Traffic prediction is a canonical spatio-temporal graph task: sensors on roads form a fixed graph, and...
GNNs for Knowledge Graphs: Reasoning and Completion
Knowledge graphs encode human knowledge as typed entity-relation triples. GNNs enable structure-aware ...
GNNs for Robotics: Planning, Manipulation, and Multi-Agent Systems
Robots interact with structured environments: objects have relationships, joints form kinematic chains...
GNNs for Computer Vision: Scene Graphs and Beyond
Computer vision tasks increasingly require relational reasoning โ understanding how objects relate to ...
HetSheaf: Heterogeneous Graphs Meet Cellular Sheaves
HetSheaf encodes graph heterogeneity directly in the sheaf data structure โ type-aware stalks and rest...
SheafPool: Basis-Invariant Graph Readout for Sheaf Neural Networks
SheafPool solves a key missing piece in sheaf GNNs: graph-level pooling. Instead of averaging stalk ve...