Pseudo-Count Methods

Pseudo-Count Methods are exploration techniques that extend count-based exploration to high-dimensional state spaces — using density models to estimate pseudo-counts $\hat{N}(s)$ that approximate traditional visit counts, enabling count-based exploration bonuses for complex observations like images.

Pseudo-Count from Density

• Density Model: Train a density model $\rho(s)$ on visited states.
• Pseudo-Count: $\hat{N}(s) = \frac{\rho(s)(1 - \rho'(s))}{\rho'(s) - \rho(s)}$ where $\rho'$ is the density after one additional visit.
• Bonus: $r_{bonus} = \beta / \sqrt{\hat{N}(s)}$ — same form as tabular count bonus.
• Models: PixelCNN, context tree switching, or other generative models for density estimation.

Why It Matters

• High-Dimensional: Extends count-based exploration to pixel observations — where tabular counts are infeasible.
• Theory Meets Practice: Bridges the theoretical elegance of count-based exploration with practical deep RL.
• Montezuma: Pseudo-counts enabled early progress on hard-exploration Atari games.

Pseudo-Count is counting in pixel space — using density models to approximate visit counts for scalable count-based exploration.