Reinforcement Learning Policy Gradient

Policy Gradient and Actor-Critic Methods are reinforcement learning algorithms that directly optimize the policy function — learning to select actions by computing gradients of expected cumulative reward with respect to policy parameters.

Policy Gradient Theorem

• Policy $\pi_\theta(a|s)$: Probability of action $a$ in state $s$, parameterized by $\theta$.
• Objective: Maximize $J(\theta) = E_{\tau \sim \pi_\theta}[\sum_t r_t]$.
• Gradient: $\nabla_\theta J(\theta) = E[\nabla_\theta \log \pi_\theta(a|s) \cdot Q(s,a)]$
• Key insight: Weight log-probability of actions by their value — increase probability of good actions.

REINFORCE (Williams, 1992)

• Monte Carlo estimate: $g = \sum_t \nabla \log \pi_\theta(a_t|s_t) \cdot G_t$
• $G_t$: Return from step t onward.
• High variance — slow to converge.

Actor-Critic

• Actor: Policy network $\pi_\theta$ — selects actions.
• Critic: Value network $V_\phi$ — estimates $V(s)$ to reduce variance.
• Advantage: $A(s,a) = Q(s,a) - V(s)$ — how much better action $a$ is vs. average.
• Lower variance than REINFORCE; less biased than pure value methods.

A3C (Asynchronous Advantage Actor-Critic)

• Multiple parallel workers independently explore + compute gradients.
• Asynchronous updates to shared global network.
• Better sample diversity than single-agent training.

PPO (Proximal Policy Optimization)

• Clip objective: $L^{CLIP} = E[\min(r_t A_t, clip(r_t, 1-\epsilon, 1+\epsilon) A_t)]$
• $r_t = \pi_\theta / \pi_{old}$: Probability ratio.
• Prevents too-large policy updates — stable training.
• Default RL algorithm for RLHF, robotics, game playing.

Reward Shaping

• Sparse rewards: Difficult to learn (reward only at goal).
• Reward shaping: Add auxiliary rewards for intermediate progress.
• Potential-based shaping: $F(s,a,s') = \gamma\Phi(s') - \Phi(s)$ — provably doesn't change optimal policy.

Policy gradient methods are the core of modern RL — PPO specifically powers RLHF for LLM alignment and robotic manipulation, making it one of the most practically important algorithms in current AI research.