Information Gain Exploration

Information Gain Exploration is an exploration strategy that rewards actions that maximize the information gained about the environment — the agent seeks states and actions that reduce its uncertainty about the transition dynamics, reward function, or other aspects of the MDP.

Information Gain Formulations

• Bayesian: Information gain = reduction in posterior uncertainty over model parameters: $I(a; \theta | s, D)$.
• VIME: Variational Information Maximizing Exploration — reward = KL divergence between prior and posterior dynamics.
• Prediction Gain: Improvement in world model prediction accuracy after experiencing a transition.
• Empowerment: Information gain about the relationship between actions and future states.

Why It Matters

• Principled: Information gain is a theoretically grounded exploration objective — Bayesian optimal design.
• Efficient: Targets exploration toward states that are most informative — avoids wasting time on irrelevant novelty.
• Model Learning: Naturally improves the world model — exploration and model learning are synergistic.

Information Gain Exploration is seeking the most informative experiences — exploring where uncertainty is highest to learn the environment fastest.