A3c

A3C (Asynchronous Advantage Actor-Critic) is an asynchronous RL algorithm that runs multiple parallel agent instances to explore different parts of the environment simultaneously — each instance independently computes gradients and asynchronously updates a shared global model.

A3C Architecture

• Parallel Workers: Multiple CPU workers each run their own copy of the environment.
• Async Updates: Each worker computes gradients locally and asynchronously updates the shared global model.
• No Replay Buffer: On-policy — no experience replay needed because parallel workers provide decorrelated data.
• Exploration: Different workers explore different regions of the state space — diverse experience.

Why It Matters

• CPU Efficiency: A3C runs on CPUs — no GPU needed — workers parallelize across CPU cores.
• Decorrelation: Parallel environments naturally decorrelate the experience stream — stabilizes training.
• Historical: A3C was a breakthrough (Mnih et al., 2016) — but has been largely superseded by A2C and PPO.

A3C is parallel exploration — running many agents asynchronously for efficient, decorrelated reinforcement learning.