Ddpg

DDPG (Deep Deterministic Policy Gradient) is an off-policy actor-critic algorithm for continuous action spaces — extending DQN's ideas (replay buffer, target network) to continuous control by learning a deterministic policy that directly outputs continuous actions.

DDPG Components

• Actor: Deterministic policy $\mu_\theta(s)$ — outputs a continuous action.
• Critic: Q-network $Q_\phi(s, a)$ — estimates the value of state-action pairs.
• Replay Buffer: Store and replay transitions for off-policy learning — sample efficiency.
• Target Networks: Soft-updated copies — $\theta' \leftarrow \tau\theta + (1-\tau)\theta'$ for stable targets.

Why It Matters

• Continuous Actions: DQN can't handle continuous actions (can't enumerate them) — DDPG solves this.
• Off-Policy: Replay buffer enables sample-efficient, off-policy learning in continuous spaces.
• Foundation: DDPG is the foundation for TD3 and SAC — the family of continuous control algorithms.

DDPG is DQN for continuous actions — combining a deterministic policy with Q-learning for off-policy continuous control.