Reward Learning

Reward Learning is the general problem of learning a reward function from various forms of human feedback — including demonstrations, preferences, corrections, natural language instructions, and other signals, to define the objective for RL without manual reward engineering.

Forms of Reward Feedback

• Demonstrations: Expert trajectories (IRL) — infer the reward from observed expert behavior.
• Preferences: Pairwise comparisons — learn which behaviors are preferred.
• Corrections: Physical corrections or partial feedback — the human adjusts the agent's behavior.
• Language: Natural language descriptions of desired behavior — ground language to reward.

Why It Matters

• Reward Design is Hard: Manual reward engineering leads to reward hacking — reward learning avoids this.
• Alignment: Learning rewards from human feedback is the core of AI alignment — ensure AI optimizes for human values.
• Specification: Reward learning turns informal human preferences into formal optimization objectives.

Reward Learning is letting humans define the objective — automatically constructing reward functions from diverse forms of human feedback.