Naf

NAF (Normalized Advantage Functions) is a continuous control RL algorithm that represents the Q-function as a quadratic function of actions — $Q(s,a) = V(s) + A(s,a)$ where the advantage is a negative-definite quadratic: $A(s,a) = -\frac{1}{2}(a-\mu(s))^T P(s)(a-\mu(s))$.

NAF Architecture

• Value: Neural network outputs $V(s)$ — state value.
• Action: Neural network outputs $\mu(s)$ — optimal action (the quadratic peak).
• Advantage Matrix: Neural network outputs lower-triangular $L(s)$ — $P(s) = L(s)L(s)^T$ ensures positive definiteness.
• Closed-Form Max: $argmax_a Q(s,a) = \mu(s)$ — no separate actor network needed.

Why It Matters

• No Actor: The optimal action is computed analytically — no separate actor network or actor optimization.
• Simple: Single network outputs value, action, and advantage matrix — cleaner than DDPG.
• Limitation: The quadratic assumption limits expressiveness — can't represent complex, multi-modal Q-functions.

NAF is Q-learning with a quadratic shortcut — using a quadratic advantage function for closed-form continuous action optimization.