Td3

TD3 (Twin Delayed DDPG) is an improvement to DDPG for continuous control that addresses overestimation bias — using twin critics, delayed policy updates, and target policy smoothing for stable, high-performance actor-critic learning.

TD3 Innovations

• Twin Critics: Two Q-networks — use the minimum $min(Q_1, Q_2)$ to compute targets, reducing overestimation.
• Delayed Policy Updates: Update the actor less frequently than the critics (every 2 critic updates) — more stable learning.
• Target Smoothing: Add noise to the target action: $a' = pi_{target}(s') + \text{clip}(N(0, \sigma), -c, c)$ — regularizes the value function.
• Deterministic Policy: The actor outputs deterministic actions — exploration via added Gaussian noise.

Why It Matters

• Overestimation Fix: Twin critics + target smoothing dramatically reduce Q-value overestimation.
• Stability: Delayed updates prevent the policy from exploiting estimation errors in the Q-function.
• Performance: TD3 significantly outperforms DDPG on MuJoCo continuous control benchmarks.

TD3 is DDPG done right — fixing overestimation and instability with twin critics, delayed updates, and target smoothing.