Options Framework

The Options Framework is the foundational formalism for hierarchical RL — defining options as temporally extended actions (macro-actions) with three components: an initiation set (where the option can start), an option policy (how it acts), and a termination condition (when it finishes).

Options Formalism

• Option $o$: $o = (I_o, pi_o, \beta_o)$ — initiation set, policy, and termination probability.
• Initiation Set $I_o$: The set of states where option $o$ can be initiated.
• Policy $pi_o(a|s)$: The action selection policy while option $o$ is active.
• Termination $\beta_o(s)$: Probability of terminating the option upon reaching state $s$.

Why It Matters

• Temporal Abstraction: Options abstract away sequences of primitive actions — enabling planning at a higher level.
• SMDP: Options induce a Semi-Markov Decision Process (SMDP) at the higher level.
• Option-Critic: The Option-Critic architecture learns options end-to-end using policy gradient — no manual definition needed.

The Options Framework is the grammar of hierarchical RL — formalizing macro-actions as reusable, temporally extended building blocks.