Graph Neural ODEs: Continuous-Time Graph Dynamics

Neural ODEs: A Quick Refresher

Standard residual network: H^{(k+1)} = H^{(k)} + f_θ(H^{(k)}). This is the Euler discretisation of the ODE:

Neural ODE (Chen et al., 2018): parameterise the derivative dH/dt with a neural network, and solve the ODE with any numerical integrator (RK4, dopri5). The solution H(T) is the output. Backpropagation through the integrator uses the adjoint method — O(1) memory regardless of integration steps.

Graph Neural ODEs

Extend the dynamics to incorporate graph structure:

Where A is the graph adjacency (fixed or time-varying) and f_θ is a GNN layer. Each “step” of the ODE solver corresponds to one round of graph-aware message passing. The number of effective layers is determined by the integration horizon T and step size — not a fixed hyperparameter.

CGODE (Continuous Graph Neural ODE)

This is the continuous analogue of a GCN layer. The solution H(T) has the same expressive power as a K-layer GCN, where K corresponds to the integration steps — but K can be non-integer and is adaptive.

Latent Graph ODE

For trajectory prediction:

1. Encoder: observe partial trajectories {x_i(t)} for t ∈ [t_0, t_obs]; encode to initial latent state z_0
2. GNN-ODE dynamics: dz/dt = GNN(z, A) — latent dynamics coupled by graph structure
3. Decoder: decode z(t) for t > t_obs to predict future trajectories

This models physically-coupled systems (particle dynamics, multi-agent trajectories) where entities interact through the graph structure.

Worked Example: Graph Neural ODE vs Discrete GCN

Consider a path graph with 3 nodes: A — B — C, each with scalar feature h(0) = [1, 0, 0] (only A is active). Adjacency after normalisation: A_hat has 1/sqrt(deg) weights.

Discrete GCN (2 layers):

• Layer 1: h_B gets contribution from A and C. h_B^(1) ≈ 0.5 (half of A’s signal)
• Layer 2: h_C gets contribution from B. h_C^(2) ≈ 0.25

The signal reaches C exactly at layer 2. To reach further you must add more layers — the depth is a hard hyperparameter.

Graph Neural ODE (integrate from t=0 to T):

The ODE dH/dt = A_hat · H diffuses the signal continuously. At time t, the solution is:

H(t) = exp(A_hat · t) · H(0)

• At t=0.5: h_C(0.5) ≈ 0.06 (signal just starting to reach C)
• At t=1.0: h_C(1.0) ≈ 0.18 (more signal)
• At t=2.0: h_C(2.0) ≈ 0.30 (stronger, oversmoothed if too large)

You can choose T to match the task’s natural scale — no need to count layers. The ODE solver also automatically refines its steps near t=0 where the gradient is largest.

Continuous-Time Graph Learning (CTDG Perspective)

For continuous-time dynamic graphs where events arrive at irregular times, Graph Neural ODEs offer a natural framework:

1. Between events: node states evolve according to dh_v/dt = f(h_v)
2. At event (u, v, t): update h_u and h_v based on the interaction

This is the approach taken by NDCG (Neural Dynamics on Complex Graphs) and similar models. The ODE handles smooth evolution between events; the event mechanism handles discrete updates.

Advantages of the ODE Formulation

Adaptive depth: ODE solvers automatically use more steps where the dynamics are complex. This is analogous to “use more layers where needed” — impossible in fixed discrete architectures.

Continuous time: make predictions at any continuous time t, not just at integer layer depths.

Physical interpretability: ODE dynamics have clear physical analogues — diffusion, oscillation, predator-prey dynamics.

Memory efficiency: adjoint method computes gradients with O(1) memory regardless of integration steps (vs O(K) for K-layer backprop).

Limitations

Speed: numerical ODE solvers are slower than fixed matrix multiplications. Adaptive step-size solvers can be unpredictable.

Stiffness: some graph dynamics are “stiff” — small perturbations cause rapid changes — requiring very small step sizes and slow integration.

Expressiveness: the continuous dynamics f must be chosen carefully. A simple GCN-like f(H, A) is no more expressive than discrete GCN.

Applications

• Particle physics: learn interaction dynamics from trajectory data
• Traffic flow: model road network congestion as a PDE
• Epidemic modelling: SIR dynamics on contact graphs
• Multi-agent systems: robots, pedestrians interacting through proximity graphs
• Time-series prediction on graphs: predicting future states of coupled systems

Summary

Graph Neural ODEs are not universally better than discrete GNNs — they are a more natural fit for physical and temporal systems where dynamics are inherently continuous. For standard graph classification or node classification on static graphs, discrete GNNs remain preferred.

References

• Chen, R. T. Q., Rubanova, Y., Bettencourt, J., & Duvenaud, D. (2018). Neural Ordinary Differential Equations. NeurIPS 2018 (Neural ODEs: replacing discrete residual layers with continuous ODE solvers via the adjoint method).
• Poli, M., Massaroli, S., Park, J., Yamashita, A., Asama, H., & Park, J. (2019). Graph Neural Ordinary Differential Equations. arXiv 2019 (Graph Neural ODEs: combining ODE dynamics with GNN spatial aggregation for continuous-time graphs).
• Rubanova, Y., Chen, R. T. Q., & Duvenaud, D. (2019). Latent ODEs for Irregularly-Sampled Time Series. NeurIPS 2019 (Latent ODEs handling irregular observation times — foundational for temporal graph ODEs).