Alessio Borgi

Alessio Borgi

PhD in Graph Neural Networks and Generative AI ๐Ÿ‘จ๐Ÿปโ€๐Ÿ’ป๐Ÿค–

Depth in GNNs: Why Deeper Is Not Always Better

5 minute read

Published:

TL;DR: Each GNN layer expands the receptive field by one hop โ€” seemingly beneficial. But more layers also mean more averaging and oversmoothing. On most real graphs, the optimal depth is 2-4 layers. Deep GNNs (8+ layers) require special architectural tricks (GCNII, JK-Net, APPNP) to avoid performance collapse.

Depth in GNNs Is Different

In Transformers, adding layers increases representational depth โ€” each layer refines the global representation with no structural penalty. Transformers scale to 96+ layers with consistent improvement.

In GNNs, each layer has a dual effect:

  1. Positive: expands the receptive field by one hop (more context)
  2. Negative: smooths features by averaging neighbours (more oversmoothing)

These effects fight each other. The optimal depth depends on which dominates.

What Depth Buys: Receptive Field

With K GNN layers, node v aggregates information from its K-hop neighbourhood. For node classification, the useful depth K* is the number of hops that contains task-relevant information.

For homophilic datasets (Cora, CiteSeer): most task-relevant context is at 1-2 hops. Beyond that, the neighbourhood is dominated by same-label nodes that add little information while increasing oversmoothing.

For long-range tasks (predicting molecular properties from whole-molecule context): you need K โ‰ฅ graph diameter โ€” which is 10+ for medium-sized molecules. This immediately conflicts with oversmoothing.

The Empirical Depth Cliff

On standard GNN benchmarks, accuracy as a function of depth follows a characteristic pattern:

Layers:   1     2     3     4     8     16    32
GCN:      75%   82%   80%   76%   58%   42%   25%
GAT:      76%   83%   81%   78%   61%   44%   28%

(Illustrative values on Cora-style datasets.) Performance peaks at 2-3 layers, then drops dramatically. At 32 layers, the model fails catastrophically โ€” oversmoothing has collapsed all node distinctions.

The Depth Dilemma

This creates an uncomfortable trade-off:

  • Task requires long-range context โ†’ need many layers
  • Many layers โ†’ oversmoothing โ†’ performance collapse
  • Few layers โ†’ undershooting the diameter โ†’ missing distant context

For graphs with large diameter (long molecules, social networks, knowledge graphs), standard GNNs are caught in this dilemma with no good resolution.

Why CNNs don't have this problem: In image CNNs, each layer also expands the receptive field. But the operation is convolution (pattern detection), not averaging. A 32-layer CNN does not collapse pixel values to a uniform grey โ€” it detects increasingly abstract patterns. GNN averaging is fundamentally different: it destroys information rather than abstracting it.

Architectural Solutions for Deep GNNs

GCNII (Chen et al., 2020)

GCNII adds two modifications to enable 64-layer GCNs:

  1. Initial residual: skip connection to the initial features X at every layer:
  2. Identity mapping: identity initialisation for weight matrices
H^{(k+1)} = ฯƒ( ((1โˆ’ฮฑ) Sฬƒ H^{(k)} + ฮฑ H^{(0)}) ((1โˆ’ฮฒ) I + ฮฒ W^{(k)}) )

ฮฑ controls how much initial feature signal is retained; ฮฒ controls how close the weight matrix stays to identity. Together, they prevent both oversmoothing and gradient vanishing.

GCNII achieves competitive accuracy with 64 layers on Cora โ€” a previously impossible depth.

JK-Net (Jumping Knowledge, Xu et al., 2018)

JK-Net uses all intermediate representations, not just the last layer:

h_v = AGG( h^{(1)}_v, h^{(2)}_v, ..., h^{(K)}_v )

Where AGG is concatenation, max-pooling, or LSTM. Each nodeโ€™s final embedding includes information from all receptive field sizes simultaneously. Oversmoothing in deep layers is offset by the sharp early-layer representations.

APPNP

As discussed in the APPNP post: separate transformation from propagation. The teleport probability ฮฑ keeps each node anchored to its own features even after 20 propagation steps โ€” preventing oversmoothing.

DropEdge

Randomly drop a fraction of edges during each training step. This reduces the averaging effect per layer, slowing oversmoothing. Analogous to Dropout for edges.

The Depth vs Width Trade-off

An alternative to depth: make each layer wider (more hidden dimensions). In practice, going from d=64 to d=256 with 2-3 layers often outperforms using d=64 with 8-16 layers โ€” wider layers better capture local structure without oversmoothing.

The GNN community is moving toward:

  • Shallow local GNNs (2-4 layers) for node-level tasks
  • Aggregation-then-transform designs (APPNP, SGC) for medium range
  • Graph Transformers or GPS (Transformer + local MPNN) for long-range tasks

Summary

DepthEffectRecommendation
1-2 layersMinimal smoothing, limited contextGood baseline
3-4 layersOptimal for most homophilic benchmarksDefault choice
8-16 layersOversmoothing dominates; needs residualsUse GCNII, JK-Net
32+ layersNear-impossible without special designGCNII or abandon local MPNN

GNN depth scaling does not follow the same scaling laws as Transformer depth. Understanding this โ€” and which architectural tricks restore the benefit of depth โ€” is central to modern GNN design.

References

You May Also Enjoy

Open Problems and Future Directions in Sheaf Neural Networks

5 minute read

Published:

Sheaf Neural Networks are a young field with many open research problems. This post surveys the most important unsolved questions: scalability to million-node graphs, stochastic sheaves for uncertainty quantification, sheaf autoencoders for generative models, connections to persistent homology, and unification with other topological deep learning frameworks.

Sheaf GNNs for Molecular Property Prediction

7 minute read

Published:

Molecular graphs are naturally heterophilic โ€” atoms of different types bonded together โ€” and have rich geometric structure (bond angles, torsion angles). Sheaf GNNs can encode this structure via restriction maps that represent the relational geometry between bonded atoms. This post explores how sheaf theory applies to molecular property prediction.

Sheaf GNNs for Heterophilic Node Classification: Benchmarks and Results

6 minute read

Published:

A systematic comparison of sheaf GNNs against baselines on standard heterophilic node classification benchmarks. Which datasets benefit most from sheaf structure? How does stalk dimension d affect performance? When do sheaves win โ€” and when donโ€™t they?

Sheaves Meet Attention: Transformer-Inspired Sheaf Models

6 minute read

Published:

Graph Transformers use global attention to capture long-range dependencies that local message passing misses. Sheaf Transformers combine sheaf-based local aggregation (with learnable relational geometry) with global attention (for long-range dependencies). This post surveys the design space of sheaf-augmented attention models.