Dueling Dqn

Dueling DQN is a DQN architecture that separates the Q-function into a state value function and an advantage function — $Q(s,a) = V(s) + A(s,a) - \text{mean}(A(s,cdot))$, allowing the network to independently learn the value of being in a state and the relative advantage of each action.

Dueling Architecture

• Shared Backbone: Convolutional layers shared for feature extraction.
• Value Stream: Fully connected layers outputting $V(s)$ — scalar value of the state.
• Advantage Stream: Fully connected layers outputting $A(s,a)$ — advantage of each action relative to average.
• Combination: $Q(s,a) = V(s) + A(s,a) - \frac{1}{|A|}\sum_{a'} A(s,a')$ — centering for identifiability.

Why It Matters

• State vs. Action: Many states have similar value regardless of action — dueling architecture captures this.
• Sample Efficiency: The value stream updates for every action — more efficient learning of state values.
• Complements: Combines well with Double DQN and Prioritized Experience Replay.

Dueling DQN is separating what matters from what to do — independently learning state value and action advantages for more efficient Q-learning.