Quantile Regression Dqn

QR-DQN (Quantile Regression DQN) is a distributional RL algorithm that learns quantiles of the return distribution — instead of fixed atoms (like C51), QR-DQN directly learns the values at fixed quantile levels using quantile regression, providing a flexible, non-parametric representation.

QR-DQN Algorithm

• Quantiles: Learn $N$ quantile values $\theta_i(s,a)$ at fixed quantile levels $\tau_i = (2i-1)/(2N)$ for $i = 1,...,N$.
• Loss: Quantile Huber loss — asymmetric loss that penalizes over/under-estimation differently for each quantile.
• No Projection: Unlike C51, no need to project distributions onto a fixed support — quantiles are free-form.
• Q-Value: $Q(s,a) = \frac{1}{N}\sum_i \theta_i(s,a)$ — the mean of the quantile values.

Why It Matters

• Flexible: Quantiles can represent any distribution shape — not limited to a fixed support like C51.
• Simpler: No distribution projection needed — cleaner algorithm than C51.
• Risk: Different quantiles enable risk-sensitive policies — optimize for extreme quantiles (CVaR).

QR-DQN is learning the quantiles of returns — a flexible, projection-free distributional RL method using quantile regression.