C51

C51 (Categorical 51-Atom) is the first practical distributional RL algorithm — representing the return distribution as a categorical distribution over 51 equally-spaced atoms, learning the probability of each atom to capture the full distribution of future returns.

C51 Algorithm

• Atoms: 51 fixed values $z_i$ equally spaced in $[V_{min}, V_{max}]$ — the support of the distribution.
• Probabilities: Neural network outputs $p_i(s,a)$ — probability that the return falls in each atom's bin.
• Projection: After Bellman update, project the shifted distribution back onto the fixed support.
• Loss: Cross-entropy between the projected target distribution and the predicted distribution.

Why It Matters

• Breakthrough: C51 (Bellemare et al., 2017) showed distributional RL works better than expected — not just theoretically interesting.
• Performance: C51 significantly outperforms standard DQN on Atari — richer gradient signal.
• Foundation: C51 spawned QR-DQN, IQN, and the distributional RL revolution.

C51 is the 51-bin histogram of returns — discretizing the return distribution into 51 atoms for practical distributional reinforcement learning.