Graph Neural ODEs: Continuous-Time Graph Dynamics

Neural ODEs: A Quick Refresher

A residual block computes \(H^{(k+1)} = H^{(k)} + f_\theta(H^{(k)})\). Read the layer index as time with unit step size and this is exactly the forward-Euler discretisation, with step \(h = 1\), of

That is the whole idea behind Neural ODEs (Chen et al., 2018): keep the ODE and drop the fixed step size. Parameterise the derivative with a neural network and hand the equation to any numerical integrator (RK4, dopri5). The solution \(H(T)\) is the output.

Gradients can be obtained by backpropagating through the solver’s operations, which costs memory proportional to the number of steps taken, or by the adjoint method: solve a second ODE backwards in time for \(a(t) = \partial L / \partial H(t)\),

and accumulate \(\partial L / \partial \theta\) along the way. Memory is then constant in the number of solver steps, because intermediate states are re-derived rather than stored. The trade-off is real: the adjoint costs a second (backward) integration, and the reconstructed forward trajectory is only as accurate as the solver tolerance, so gradients can be noisier than with direct backpropagation.

Graph Neural ODEs

Extend the dynamics to incorporate graph structure:

where \(A\) is the graph adjacency (fixed or time-varying) and \(f_\theta\) is a GNN layer. Each function evaluation inside the solver is one round of graph-aware message passing, so the amount of propagation is set by the integration horizon \(T\) rather than by a layer count.

Continuous GCN dynamics

This is the continuous analogue of a GCN layer. Two things are worth separating carefully here. The integration horizon \(T\) controls how far information propagates and is the real analogue of depth. The number of solver steps is a numerical-accuracy decision: taking more steps to integrate to the same \(T\) approximates the same function more precisely — it does not make the model more expressive. Claims of the form “\(K\) solver steps equal a \(K\)-layer GCN” conflate the two.

Latent Graph ODE

For trajectory prediction:

1. Encoder: observe partial trajectories \(\{x_i(t)\}\) for \(t \in [t_0, t_{\text{obs}}]\); encode to an initial latent state \(z_0\)
2. GNN-ODE dynamics: \(\frac{dz}{dt} = \mathrm{GNN}(z, A)\) — latent dynamics coupled by graph structure
3. Decoder: decode \(z(t)\) for \(t > t_{\text{obs}}\) to predict future trajectories

This models physically-coupled systems (particle dynamics, multi-agent trajectories) where entities interact through the graph structure.

Worked Example: Graph Neural ODE vs Discrete GCN

Consider a path graph with 3 nodes, A — B — C, each with a scalar feature, and \(H(0) = [1, 0, 0]^{\!\top}\) (only A is active). Degrees are \(1, 2, 1\), so the symmetrically normalised adjacency \(\hat{A} = D^{-1/2} A D^{-1/2}\) has \(\hat{A}_{AB} = \hat{A}_{BC} = 1/\sqrt{2} \approx 0.71\), with eigenvalues \(\{1, 0, -1\}\).

Discrete propagation, \(H^{(k+1)} = \hat{A} H^{(k)}\):

• Layer 1: \(H^{(1)} = [0,\ 0.71,\ 0]^{\!\top}\) — the signal has reached B and left A entirely
• Layer 2: \(H^{(2)} = [0.5,\ 0,\ 0.5]^{\!\top}\) — it reaches C, and bounces back to A

The signal reaches C exactly at layer 2. To reach further you must add more layers — the depth is a hard hyperparameter.

Graph Neural ODE (integrate from \(t = 0\) to \(T\)):

The right continuous analogue is diffusion, driven by the negative normalised Laplacian \(-\hat{L} = \hat{A} - I\), not by \(\hat{A}\) alone:

The distinction matters. \(\hat{A}\) has a \(+1\) eigenvalue, so \(e^{\hat{A}t}\) grows without bound and the “continuous GCN” would diverge; \(-\hat{L}\) has eigenvalues \(\{0, -1, -2\}\), all \(\le 0\), so the solution stays bounded and settles. Expanding \(H(0)\) in the eigenbasis gives a closed form for node C:

• At \(t = 0.5\): \(h_C \approx 0.04\) — signal just starting to reach C
• At \(t = 1.0\): \(h_C \approx 0.10\)
• At \(t = 2.0\): \(h_C \approx 0.19\)
• As \(t \to \infty\): \(h_C \to 0.25\), and the whole state converges to the \(\hat{L}\)-null eigenvector \(\propto (1, \sqrt{2}, 1)\)

That limit is oversmoothing, stated exactly: integrate too far and every node converges to the same degree-scaled constant, and the initial condition is forgotten. Choosing \(T\) is choosing how much smoothing you want — the continuous version of choosing depth, but on a real-valued dial.

Continuous-Time Graph Learning (CTDG Perspective)

For continuous-time dynamic graphs where events arrive at irregular times, Graph Neural ODEs offer a natural framework:

1. Between events: node states evolve according to \(\frac{dh_v}{dt} = f(h_v)\)
2. At event \((u, v, t)\): apply a discrete jump to \(h_u\) and \(h_v\) based on the interaction

This “flow, then jump” pattern is the ODE counterpart of TGN’s memory: the ODE handles smooth evolution between events, the jump handles the discrete update. It respects causality for free — the state at \(t\) is an integral over \([0, t]\), so future events cannot enter.

Advantages of the ODE Formulation

Continuous time: make predictions at any real-valued time \(t\), not just at integer layer depths. This is the genuine advantage, and it is what makes ODEs a natural fit for irregularly sampled data.

Solver-chosen work: adaptive solvers spend more function evaluations where the trajectory is hard to integrate. Note what this does and does not buy: it adapts numerical effort, not model capacity. The function being computed is fixed by \(f_\theta\) and \(T\).

Physical interpretability: ODE dynamics have clear physical analogues — diffusion, oscillation, predator-prey dynamics.

Memory efficiency: the adjoint method computes gradients with memory constant in the number of solver steps, against \(O(K)\) for storing \(K\) layers’ activations — at the cost of a second backward integration and some gradient error.

Limitations

Speed: numerical ODE solvers are slower than fixed matrix multiplications, and the cost of a forward pass varies with the input because the solver’s step count does.

Stiffness: some graph dynamics are “stiff” — components evolving on very different timescales — forcing explicit solvers into very small step sizes.

Expressiveness: the continuous dynamics \(f\) must be chosen carefully. A GCN-like \(f(H, A)\) integrated for time \(T\) is not more expressive than message passing; it is the same function class reached by a different route. There is also a structural constraint: for an autonomous ODE with a unique solution, trajectories cannot cross, so the flow map \(H(0) \mapsto H(T)\) is a homeomorphism. Functions that must “fold” the input space are therefore not representable without augmenting the state.

Applications

• Particle physics: learn interaction dynamics from trajectory data
• Traffic flow: model road network congestion as a PDE
• Epidemic modelling: SIR dynamics on contact graphs
• Multi-agent systems: robots, pedestrians interacting through proximity graphs
• Time-series prediction on graphs: predicting future states of coupled systems

Summary

Graph Neural ODEs are not universally better than discrete GNNs — they are a more natural fit for physical and temporal systems where dynamics are inherently continuous. For standard graph classification or node classification on static graphs, discrete GNNs remain preferred.

References

• Chen, R. T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D. (2018). Neural Ordinary Differential Equations. NeurIPS 2018 (Neural ODEs: replacing discrete residual layers with continuous ODE solvers via the adjoint method).
• Poli, M., Massaroli, S., Park, J., Yamashita, A., Asama, H., & Park, J. (2019). Graph Neural Ordinary Differential Equations. arXiv 2019 (Graph Neural ODEs: combining ODE dynamics with GNN spatial aggregation for continuous-time graphs).
• Rubanova, Y., Chen, R. T. Q., & Duvenaud, D. (2019). Latent ODEs for Irregularly-Sampled Time Series. NeurIPS 2019 (Latent ODEs handling irregular observation times — foundational for temporal graph ODEs).