Trust Region Policy Optimization

TRPO (Trust Region Policy Optimization) is a policy gradient RL algorithm that constrains policy updates to a trust region — ensuring that each update doesn't change the policy too much, providing theoretical monotonic improvement guarantees.

TRPO Algorithm

• Constraint: Limit the KL divergence between old and new policies: $D_{KL}(pi_{old} | pi_{new}) \leq delta$.
• Optimization: $\max_\theta \mathbb{E}[\frac{pi_\theta(a|s)}{pi_{old}(a|s)} A(s,a)]$ subject to the KL constraint.
• Solving: Uses conjugate gradient + line search to approximately solve the constrained optimization.
• Natural Gradient: TRPO is equivalent to a natural gradient step — accounts for the policy's geometry.

Why It Matters

• Monotonic Improvement: Each TRPO update is guaranteed to improve (or not decrease) the expected return.
• Stability: KL constraint prevents destructive large policy updates — stable training.
• Foundation: TRPO laid the theoretical foundation for PPO — PPO simplifies TRPO's constrained optimization.

TRPO is safe policy updates — constraining each step to a trust region for guaranteed monotonic improvement in reinforcement learning.