Double Dqn

Double DQN is an improvement to DQN that addresses the overestimation bias in Q-learning — using the online network to select the best action and the target network to evaluate it, decoupling action selection from evaluation to reduce systematic overestimation.

Double DQN Fix

• DQN Problem: $y = r + \gamma max_{a'} Q_{target}(s', a')$ — the same network both selects and evaluates, causing overestimation.
• Double DQN: $y = r + \gamma Q_{target}(s', argmax_{a'} Q_\theta(s', a'))$ — online network selects, target network evaluates.
• Decoupling: Separating selection and evaluation eliminates the positive bias.
• Simple: Just one line of code difference from DQN — use online network for argmax.

Why It Matters

• Overestimation: DQN's max operator systematically overestimates Q-values — Double DQN eliminates this.
• Better Performance: Double DQN consistently improves upon DQN across Atari games.
• No Extra Cost: Same computational cost as DQN — the target network already exists.

Double DQN is the overestimation fix — decoupling action selection from evaluation for more accurate Q-value estimates.