Curiosity-Driven Learning

Curiosity-Driven Learning is a specific form of intrinsic motivation where the agent is rewarded for encountering situations that are difficult to predict — the agent's curiosity reward is the prediction error of a forward dynamics model, driving it toward novel, surprising states.

ICM (Intrinsic Curiosity Module)

• Forward Model: Predicts next state features: $\hat{\phi}(s_{t+1}) = f(\phi(s_t), a_t)$.
• Curiosity Reward: $r_i = |\hat{\phi}(s_{t+1}) - \phi(s_{t+1})|^2$ — prediction error = surprise.
• Feature Space: Predict in a learned feature space, not raw pixels — avoids the "noisy TV" problem.
• Inverse Model: Predict action from consecutive states — ensures the feature space captures actionable information.

Why It Matters

• No Reward Needed: The agent explores effectively driven purely by curiosity — no external reward required.
• Game Playing: Curiosity-driven agents learn to play Atari games with zero external reward — remarkable emergent behavior.
• Transfer: Curiosity-learned representations transfer to downstream tasks.

Curiosity-Driven Learning is exploring the unpredictable — rewarding the agent for encountering states it cannot yet predict.