Preference-Based Rl

Preference-Based RL is a reinforcement learning paradigm where the reward signal comes from human preferences over trajectory pairs — instead of numeric rewards, a human evaluator compares two behaviors and indicates which is preferred, and a reward model is learned from these comparisons.

Preference Learning Pipeline

• Query: Present the human with two trajectory segments $(\sigma_1, \sigma_2)$.
• Label: Human indicates preference: $\sigma_1 succ \sigma_2$, $\sigma_2 succ \sigma_1$, or roughly equal.
• Reward Model: Train $R(s,a)$ such that $P(\sigma_1 succ \sigma_2) = \frac{\exp(\sum R(\sigma_1))}{\exp(\sum R(\sigma_1)) + \exp(\sum R(\sigma_2))}$ (Bradley-Terry model).
• RL: Optimize policy using the learned reward model via standard RL (PPO, SAC, etc.).

Why It Matters

• Reward-Free: No need to hand-craft reward functions — preferences define the objective implicitly.
• Scalable: Preferences are faster to provide than demonstrations — binary comparison is cognitively easy.
• Active Queries: Active learning selects the most informative trajectory pairs to query — minimizes human effort.

Preference-Based RL is learning rewards from comparisons — using human preferences over behaviors to automatically derive reward functions.