Ham

HAM (Hierarchies of Abstract Machines) is a hierarchical RL framework that constrains the agent's policy space using partial programs — defining the high-level task structure as a set of abstract machines (finite state controllers) that specify the skeleton of behavior, with choice points where RL selects among alternatives.

HAM Components

• Abstract Machines: Finite state machines that define the structure of behavior for each subtask.
• Choice Points: States in the abstract machine where RL must decide which sub-machine to call or which action to take.
• Call Stack: HAMs can call other HAMs — creating a hierarchical call structure (like function calls).
• Constrained MDP: The HAM reduces the original MDP to a constrained SMDP over just the choice points.

Why It Matters

• Domain Knowledge: HAMs encode domain knowledge as program structure — RL only fills in the decisions.
• Reduced Search: By constraining the policy space, HAMs dramatically reduce the RL search problem.
• Composable: HAMs compose hierarchically — complex behaviors emerge from combining simple machines.

HAM is programming the structure, learning the decisions — using abstract machines to constrain hierarchical RL with domain knowledge.