Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI 👨🏻‍💻🤖

GAT: Graph Attention Networks

5 minute read

Published:

TL;DR: GAT (Veličković et al., 2018) replaces GCN's fixed degree-normalised weights with learned attention coefficients \(\alpha_{ij}\) on each edge. Each node can learn to attend more strongly to certain neighbours — adaptive, task-specific, interpretable.
GAT attention visualisation
Graph Attention Network: t-SNE visualisation of learned representations (Veličković et al., 2018)

The Problem with Fixed Weights

In GCN, the aggregation weight for neighbour \(u\) contributing to node \(v\) is fixed at \(1/\sqrt{\tilde{d}_u \tilde{d}_v}\), where \(\tilde{d}_u\) is the degree of \(u\) counting its self-loop. This depends only on node degrees — the model can’t learn that some neighbours are more important than others for the task at hand.

For a citation network: when predicting paper topic, citing a paper on the exact same topic should count more than citing a survey that covers dozens of topics. GCN can’t express this.

GAT solves it by learning attention weights from features.

The Attention Mechanism

For each directed edge \(j \to i\), GAT computes an attention coefficient \(\alpha_{ij}\) — how much node \(i\) attends to neighbour \(j\). Throughout, \(h_i \in \mathbb{R}^{F}\) is the input representation of node \(i\), \(W \in \mathbb{R}^{F' \times F}\) is a weight matrix shared by all nodes, and \(\mathcal{N}(i)\) is the neighbourhood of \(i\) (in the paper this includes \(i\) itself, so a node also attends to its own features).

Step 1: Linear transform. Apply the shared weight matrix \(W\) to both node features:

\[ z_i = W h_i, \qquad z_j = W h_j \]

Step 2: Concatenate and score. Compute a raw attention score with a learned vector \(\mathbf{a} \in \mathbb{R}^{2F'}\):

\[ e_{ij} = \operatorname{LeakyReLU}\big( \mathbf{a}^{\top} [\, z_i \,\Vert\, z_j \,] \big) \]

Here \(\Vert\) is concatenation and the LeakyReLU uses a negative slope of \(0.2\).

Step 3: Softmax over neighbours. Normalise the scores across all neighbours of \(i\), so that they form a distribution:

\[ \alpha_{ij} = \operatorname{softmax}_{j \in \mathcal{N}(i)}\big(e_{ij}\big) = \frac{\exp(e_{ij})}{\sum_{k \in \mathcal{N}(i)} \exp(e_{ik})}, \qquad \sum_{j \in \mathcal{N}(i)} \alpha_{ij} = 1 \]

Step 4: Weighted aggregate.

\[ h_i' = \sigma\Big( \sum_{j \in \mathcal{N}(i)} \alpha_{ij} \, W h_j \Big) \]

Note that \(\alpha_{ij} \ne \alpha_{ji}\) in general: the normalisation is per receiving node, so attention is asymmetric even on an undirected graph.

i target A α = 0.65 (high) B α = 0.25 (medium) C α = 0.10 (low) α_iA + α_iB + α_iC = 0.65 + 0.25 + 0.10 = 1.0 (softmax normalised) Multi-head: K heads each with different α → concat or average
Figure 1: GAT learns an attention coefficient \(\alpha_{ij}\) for each edge \(j \to i\). Neighbour A gets high attention (0.65), B medium (0.25), C low (0.10). Because of the softmax, \(\sum_j \alpha_{ij} = 1\), and these coefficients weight the neighbourhood aggregation. (This example omits the self-loop for clarity.)

Concrete Numerical Example: Computing One Attention Coefficient

Suppose \(W = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}\) (the identity, for clarity), and the attention vector \(\mathbf{a} = [1, 1, 1, 1]\) (4-dimensional, matching the concatenation of two 2-dimensional vectors).

Node \(i\) has \(h_i = [1, 0]\) and neighbour \(j\) has \(h_j = [0, 1]\).

Step 1 — Linear transform:

\[ z_i = W h_i = [1, 0], \qquad z_j = W h_j = [0, 1] \]

Step 2 — Concatenate and score:

\[ [\, z_i \Vert z_j \,] = [1, 0, 0, 1], \qquad e_{ij} = \operatorname{LeakyReLU}\big( \mathbf{a}^{\top} [1,0,0,1] \big) = \operatorname{LeakyReLU}(2) = 2 \]

Step 3 — Softmax over neighbours. Suppose \(i\) has one other neighbour \(k\), with raw score \(e_{ik} = 1\):

\[ \alpha_{ij} = \frac{e^{2}}{e^{2} + e^{1}} = \frac{7.39}{7.39 + 2.72} \approx 0.73, \qquad \alpha_{ik} = \frac{e^{1}}{e^{2} + e^{1}} \approx 0.27 \]

Step 4 — Weighted aggregate:

\[ h_i' = \sigma\big( 0.73 \, W h_j + 0.27 \, W h_k \big) \]

Neighbour \(j\) gets roughly \(2.7\times\) the weight of \(k\).

Key Insight: The attention coefficient \(\alpha_{ij}\) is learned from feature content, not just graph topology. A neighbour whose transformed features score highly under \(\mathbf{a}^{\top}[z_i \Vert z_j]\) gets a large weight. This means the same graph structure can produce different attention patterns depending on the node features — making GAT adaptive to the task in a way GCN's fixed degree normalisation cannot be.

Multi-Head GAT

Just like Multi-Head Attention in Transformers, GAT can run \(K\) independent attention heads, each with its own \(W^k\) and \(\mathbf{a}^k\) (and therefore its own coefficients \(\alpha^k_{ij}\)):

  • For intermediate layers, concatenate the \(K\) head outputs — this multiplies the feature dimension by \(K\):
\[ h_i' = \big\Vert_{k=1}^{K} \; \sigma\Big( \sum_{j \in \mathcal{N}(i)} \alpha^{k}_{ij} \, W^{k} h_j \Big) \]
  • For the final layer, average the \(K\) head outputs instead, which keeps the output dimension:
\[ h_i' = \sigma\Big( \frac{1}{K} \sum_{k=1}^{K} \sum_{j \in \mathcal{N}(i)} \alpha^{k}_{ij} \, W^{k} h_j \Big) \]

Each head can specialise in different types of relationships, exactly as in Transformer multi-head attention.

GAT v2 (2022)

The original GAT has a subtle expressiveness issue. Because \(\mathbf{a}\) is applied after the concatenation but before the non-linearity, the score decomposes as \(e_{ij} = \operatorname{LeakyReLU}(\mathbf{a}_1^{\top} z_i + \mathbf{a}_2^{\top} z_j)\), and the term \(\mathbf{a}_1^{\top} z_i\) is the same for every \(j\). Since LeakyReLU is monotonic, the ranking of neighbours by attention score is therefore identical for every query node \(i\) — attention that is static rather than query-dependent. Brody et al. (2022) named and proved this.

GAT v2 fixes it with a small change — applying the non-linearity before the dot product with \(\mathbf{a}\):

\[ e_{ij} = \mathbf{a}^{\top} \operatorname{LeakyReLU}\big( W_{\ell} h_i + W_{r} h_j \big) \]

Now the non-linearity mixes the two nodes before \(\mathbf{a}\) scores them, so the attention is dynamic: a genuine function of the pair \((i, j)\), and each node can induce its own ranking over its neighbours.

When to Use GAT over GCN?

  • When neighbour importance varies and you want the model to learn which neighbours matter.
  • When interpretability is important — the \(\alpha_{ij}\) values can be visualised as edge importance scores (though attention weights should be read as a hint, not a faithful explanation).
  • When edge features are available (they can be concatenated into the attention score).

✅ Key Takeaways

  • GAT replaces GCN's fixed degree weights with learned attention coefficients \(\alpha_{ij}\) on each edge, normalised by a softmax over \(\mathcal{N}(i)\).
  • Attention is computed from both endpoints' features — adaptive to the task, not just graph topology — and is asymmetric: \(\alpha_{ij} \ne \alpha_{ji}\).
  • Multi-head GAT runs \(K\) attention heads in parallel: concatenated in intermediate layers, averaged in the final one.
  • GAT v2 fixes the original's static attention — a neighbour ranking shared by every query node — by applying the non-linearity before the scoring vector \(\mathbf{a}\).

You May Also Enjoy

The Standard Library Modules Worth Knowing Before You pip install

8 minute read

Published:

Python ships with a counter, a queue, a memoiser, a CLI parser, a test framework’s smaller cousin, and a JSON-safe date library already installed — most of what people reach for a package to solve is one import away.

Files and Context Managers: Why with Is Not Optional

9 minute read

Published:

A file object that falls out of scope without with does get closed eventually — but ‘eventually’ means whenever the garbage collector gets around to it, which is not a promise any program handling more than a handful of files can live with.

Errors and Exceptions: EAFP, the Hierarchy, and Reading a Traceback

9 minute read

Published:

Exceptions in Python are not exceptional. They are a normal control-flow mechanism that the language leans on so heavily that the idiomatic style is to try the operation and handle the failure, rather than check first — which is faster, shorter, and free of race conditions.