Universal Value Function Approximators

UVFA (Universal Value Function Approximators) is a framework for generalizing value functions across goals — extending standard value functions $V(s)$ to $V(s, g)$ that estimate the expected return from state $s$ when pursuing goal $g$, enabling a single learned function to evaluate any state-goal pair.

UVFA Architecture

• Input: State $s$ and goal $g$ — both encoded and combined as input to the value network.
• Generalization: The network learns to generalize across goals — can predict values for unseen goals.
• Factored: State and goal can be processed by separate embeddings before being combined.
• Training: Train on multiple goals simultaneously — Horde architecture for parallel goal learning.

Why It Matters

• Multi-Goal: One value function serves all goals — no need to learn separate value functions for each goal.
• Transfer: Knowledge transfers across goals — similar goals yield similar value estimates.
• Foundation: UVFAs are the value-function counterpart to goal-conditioned policies — enabling flexible multi-goal RL.

UVFA is one value function for all goals — generalizing value estimation across the entire goal space for efficient multi-goal learning.