Adjoint Sensitivity Method

Adjoint Sensitivity Method is the memory-efficient technique for computing gradients through ODE solvers — instead of storing all intermediate states (backpropagation), it solves an adjoint ODE backward in time, reducing memory from $O(L)$ (number of steps) to $O(1)$.

How the Adjoint Method Works

• Forward Pass: Solve the ODE $dz/dt = f_\theta(z, t)$ from $t_0$ to $t_1$, storing only the final state.
• Adjoint ODE: Define $a(t) = dL/dz(t)$ (the adjoint). It satisfies $da/dt = -a^T \partial f/\partial z$.
• Backward Pass: Solve the adjoint ODE backward from $t_1$ to $t_0$, simultaneously computing parameter gradients.
• Constant Memory: Only stores the current state and adjoint — no checkpointing needed.

Why It Matters

• Memory Efficiency: Enables Neural ODEs with very deep (continuous) dynamics without memory blow-up.
• Scalability: Train models with millions of time steps that would be impossible with standard backpropagation.
• Trade-Off: Adjoint method requires solving an additional ODE backward — trades memory for compute.

Adjoint Sensitivity is backpropagation without storing intermediates — solving an ODE backward to compute gradients with constant memory.