Maxq

MAXQ is a hierarchical RL value decomposition method that breaks down the overall value function into a sum of subtask completion rewards and subtask values — each node in the task hierarchy contributes to the overall value, enabling modular learning and state abstraction.

MAXQ Decomposition

• Task Graph: Define a directed acyclic graph of subtasks — leaf nodes are primitive actions, internal nodes are composite tasks.
• Decomposed Value: $Q(s, a) = V(s, a) + C(s, a)$ where $V$ is the subtask's own reward and $C$ is the completion function (reward after subtask finishes).
• Recursive: Each subtask's value decomposes further — the entire tree contributes to the root value.
• State Abstraction: Each subtask can use its own state abstraction — only relevant features needed.

Why It Matters

• Modularity: Each subtask learns independently — modular, reusable value functions.
• State Abstraction: Different subtasks can ignore irrelevant state features — faster learning.
• Interpretable: The decomposed value function shows exactly how each subtask contributes to overall value.

MAXQ is value decomposition for hierarchical RL — breaking the overall value into modular subtask contributions for efficient, interpretable learning.