Feudal Networks

Feudal Networks (FuN) is a hierarchical RL architecture inspired by feudalism — a Manager network sets abstract goals in a learned latent space, and a Worker network executes primitive actions to achieve those goals, creating a two-level hierarchy of decision-making.

FuN Architecture

• Manager: Operates at a slower timescale — sets a goal direction $g_t$ in a learned embedding space every $c$ steps.
• Worker: Operates at every timestep — policy is conditioned on the manager's goal: $pi_{worker}(a|s, g_t)$.
• Goal Embedding: Goals are direction vectors in a learned state representation space — the worker should move in that direction.
• Transition Policy Gradient: Manager is trained to set goals that lead to higher returns.

Why It Matters

• Automatic Subgoals: The manager learns to set meaningful subgoals — no manual subtask definition.
• Temporal Abstraction: Manager operates at coarser timescale — handles long-horizon planning.
• State-of-Art: FuN enabled progress on hard exploration tasks (Montezuma's Revenge) with learned hierarchies.

Feudal Networks is the lord-and-serf architecture — a manager sets abstract goals, a worker executes them for flexible hierarchical RL.