Proximal Policy Optimization

PPO (Proximal Policy Optimization) is the most widely used policy gradient RL algorithm — simplifying TRPO's constrained optimization into a clipped surrogate objective that achieves similar stability with much simpler implementation and better empirical performance.

PPO Clipped Objective

• Ratio: $r_t(\theta) = \frac{pi_\theta(a_t|s_t)}{pi_{old}(a_t|s_t)}$ — probability ratio between new and old policy.
• Clipped: $L^{CLIP} = min(r_t A_t, \text{clip}(r_t, 1-\epsilon, 1+\epsilon) A_t)$ — clip the ratio to $[1-epsilon, 1+epsilon]$.
• $epsilon$ Parameter: Typically 0.1-0.2 — controls how much the policy can change per update.
• Mini-Batch: Multiple optimization epochs per data collection — more sample efficient than vanilla policy gradient.

Why It Matters

• Simplicity: Much simpler than TRPO — no conjugate gradient, no KL constraint, just clipping.
• RLHF: PPO is the standard algorithm for RLHF (Reinforcement Learning from Human Feedback) in LLMs.
• Versatility: Works for discrete and continuous actions, single and multi-agent, games and robotics.

PPO is the workhorse of modern RL — simple, stable, and effective policy optimization through clipped surrogate objectives.