RLHF

RLHF (Reinforcement Learning from Human Feedback) is a training methodology that aligns LLMs with human preferences by training a reward model on human comparisons and optimizing the LLM policy with RL — the technique behind ChatGPT and most deployed aligned models.

RLHF Pipeline

Phase 1 — Supervised Fine-Tuning (SFT):
- Fine-tune the pretrained LLM on high-quality human-written demonstrations.
- Creates a reasonable starting point for preference learning.

Phase 2 — Reward Model Training:
- Collect preference data: Show human raters two LLM responses to the same prompt.
- Raters choose which response is better (helpful, harmless, honest).
- Train a reward model $r_\phi$ to predict which response humans prefer.
- Reward model: Same LLM backbone + regression head.

Phase 3 — RL Optimization (PPO):
- Use PPO to update the LLM policy to maximize $r_\phi$ score.
- KL penalty: $r_{\text{total}} = r_\phi(x,y) - \beta \cdot KL(\pi_\theta \pi_{SFT})$
- KL term prevents the model from drifting too far from SFT behavior ("reward hacking").

Why RLHF Works

• Human preferences capture things hard to specify as a loss: helpfulness, tone, safety, nuance.
• Enables models to learn "be helpful but not harmful" holistically.
• InstructGPT (RLHF) dramatically outperformed 100x larger GPT-3 on human preference evaluations.

Challenges

• Expensive: Requires large-scale human annotation.
• Reward hacking: Models find ways to score high without being genuinely helpful.
• PPO instability: Training is sensitive to hyperparameters.
• Preference noise: Human raters disagree, labels are noisy.

RLHF is the alignment technique that made LLMs genuinely useful and safe for broad deployment — it transformed raw language models into helpful assistants.