Never Give Up

NGU (Never Give Up) is an exploration algorithm that combines episodic novelty with life-long novelty for persistent exploration — using both a within-episode novelty signal (encourage visiting new states within the current episode) and a between-episode signal (encourage visiting states not seen in previous episodes).

NGU Components

• Episodic Novelty: K-nearest neighbor in an episodic memory of embeddings — reward decreases as similar states accumulate within the episode.
• Life-Long Novelty: RND-based — detects states novel across all episodes.
• Combined: $r_i = r_{episodic} \cdot \min(\max(r_{lifelong}, 1), L)$ — multiplicative combination.
• Multiple Policies: Train a family of policies with different exploration-exploitation trade-offs.

Why It Matters

• Persistent Exploration: Unlike pure curiosity (which fades), NGU's episodic component ensures continued exploration.
• State-of-Art: NGU set new records on hard-exploration Atari games (Montezuma's Revenge, Pitfall).
• Multi-Scale: Captures novelty at both short-term (episode) and long-term (lifetime) scales.

NGU is curiosity that never fades — combining episodic and life-long novelty for relentless, multi-scale exploration.