Reinforcement Learning Policy Gradient

Deep Reinforcement Learning is the artificial intelligence paradigm where agents learn optimal behavior through trial-and-error interaction with environments — combining deep neural networks as function approximators with reinforcement learning algorithms to handle high-dimensional state spaces, enabling mastery of games, robotic control, and complex decision-making tasks.

Value-Based Methods:
- Q-Learning: learns action-value function Q(s,a) estimating expected cumulative reward for taking action a in state s — agent selects action with highest Q-value; tabular Q-learning works for small state spaces
- Deep Q-Network (DQN): neural network approximates Q-function for high-dimensional states (e.g., raw pixels) — key innovations: experience replay (randomly sample past transitions), target network (slowly updated copy for stable targets), and ε-greedy exploration
- Double DQN: addresses Q-value overestimation by using online network for action selection and target network for value estimation — reduces positive bias that causes suboptimal policies in standard DQN
- Dueling DQN: separates Q(s,a) into state value V(s) and advantage A(s,a) streams — V(s) estimates how good a state is regardless of action; A(s,a) estimates relative advantage of each action; improves learning for states where action choice matters less

Policy Gradient Methods:
- REINFORCE: directly optimizes policy π(a|s;θ) by gradient ascent on expected reward — ∇J(θ) = E[∇log π(a|s;θ) × R]; high variance requires baseline subtraction (typically V(s)) for practical convergence
- Actor-Critic: actor (policy network) selects actions, critic (value network) estimates expected return — advantage A(s,a) = Q(s,a) - V(s) reduces variance compared to pure policy gradient; TD error provides online update signal
- PPO (Proximal Policy Optimization): clips policy ratio to prevent destructively large updates — L^CLIP = min(r_t(θ)A_t, clip(r_t(θ), 1-ε, 1+ε)A_t) where r_t is probability ratio of new/old policy; stable training without careful learning rate tuning
- SAC (Soft Actor-Critic): maximizes reward plus entropy bonus — encourages exploration by penalizing deterministic policies; achieves robust performance across continuous control tasks; automatic temperature adjustment

Exploration vs. Exploitation:
- ε-Greedy: with probability ε take random action, otherwise take greedy action — simple but uniform random exploration is inefficient in large action spaces
- Intrinsic Motivation: reward agent for visiting novel states — curiosity-driven exploration using prediction error of learned world model; count-based exploration bonuses for rarely visited states
- Reward Shaping: engineer intermediate rewards to guide learning toward distant goals — must preserve optimal policy (potential-based shaping); helps bridge sparse reward signals in long-horizon tasks

Deep reinforcement learning has achieved superhuman performance in Atari games (DQN), Go (AlphaGo/AlphaZero), StarCraft II (AlphaStar), and robotic manipulation — representing the frontier of AI systems that learn complex behaviors through environmental interaction rather than supervised data.