Distributional Rl

Distributional RL is a reinforcement learning framework that learns the full distribution of returns (not just the expected value) — instead of estimating $Q(s,a) = \mathbb{E}[R]$, distributional RL estimates the random variable $Z(s,a)$ such that $Q(s,a) = \mathbb{E}[Z(s,a)]$.

Distributional RL Methods

• C51: Represent the distribution as a categorical distribution over 51 atoms (fixed support).
• QR-DQN: Learn quantiles of the return distribution — flexible, non-parametric.
• IQN: Implicit Quantile Network — sample any quantile at inference time.
• Bellman Update: $Z(s,a) overset{D}{=} R + \gamma Z(s', a')$ — distribute the Bellman equation over the full distribution.

Why It Matters

• Richer Information: The full distribution captures risk, multimodality, and uncertainty — not just the mean.
• Better Optimization: Learning distributions provides richer gradient signals — improves optimization.
• Risk-Sensitive: Enables risk-sensitive policies — optimize for different quantiles (worst-case, median, etc.).

Distributional RL is learning the entire story of returns — capturing the full distribution of outcomes for richer, risk-aware reinforcement learning.