Neural Ode
Neural ODEs are deep learning models that define the hidden state dynamics as a continuous ordinary differential equation rather than discrete layers — replacing the sequence of finite transformation layers with a continuous-time flow $dh/dt = f_\theta(h(t), t)$ solved by numerical ODE integrators, enabling adaptive computation depth, memory-efficient training, and principled modeling of continuous-time processes.
From ResNets to Neural ODEs
- ResNet: $h_{t+1} = h_t + f_\theta(h_t)$ — discrete step, fixed number of layers.
- Neural ODE: $\frac{dh}{dt} = f_\theta(h(t), t)$ — continuous transformation, solved from t=0 to t=1.
- ResNet layers are Euler discretizations of the underlying ODE.
- Neural ODE makes this connection explicit → can use sophisticated ODE solvers.
Forward Pass
1. Start with initial condition h(0) = input features.
2. Define dynamics function: $f_\theta(h, t)$ — a neural network.
3. Solve ODE from t=0 to t=T using numerical solver: `h(T) = ODESolve(f_θ, h(0), 0, T)`.
4. h(T) is the output representation.
Backward Pass (Adjoint Method)
- Naive approach: Backprop through ODE solver steps → O(L) memory (like a deep ResNet).
- Adjoint method: Solve a second ODE backwards in time to compute gradients.
- Memory cost: O(1) — constant regardless of number of solver steps.
- Trade-off: Recomputes forward trajectory during backward pass → slightly slower but dramatically less memory.
ODE Solvers Used
| Solver | Order | Steps | Adaptive | Use Case |
|---|---|---|---|---|
| Euler | 1 | Fixed | No | Fast, low accuracy |
| RK4 (Runge-Kutta) | 4 | Fixed | No | Good accuracy |
| Dopri5 (RK45) | 5(4) | Adaptive | Yes | Default choice |
| Adams (multistep) | Variable | Adaptive | Yes | Stiff systems |
Adaptive Computation
- Adaptive solvers take more steps where dynamics are complex, fewer where simple.
- Result: Model automatically allocates more computation to harder inputs.
- During inference: "Easy" inputs processed with fewer function evaluations → faster.
Applications
- Time-Series Modeling: Irregularly-sampled data (medical records, sensor logs) — ODE naturally handles variable time gaps.
- Continuous Normalizing Flows: Invertible generative models with exact log-likelihood.
- Physics-Informed ML: Model physical systems (fluid dynamics, molecular dynamics) with neural ODEs that respect continuous dynamics.
Implementation: torchdiffeq
from torchdiffeq import odeint
h_T = odeint(dynamics_func, h_0, t_span, method='dopri5')Neural ODEs are a foundational bridge between deep learning and dynamical systems theory — their continuous formulation provides principled tools for modeling temporal processes, enabling adaptive computation, and connecting modern machine learning with centuries of mathematical theory about differential equations.