Reinforcement Learning Human Feedback RLHF

Reinforcement Learning from Human Feedback (RLHF) is the alignment training methodology that fine-tunes pre-trained language models to follow human instructions and preferences by training a reward model on human comparison data and then optimizing the language model's policy to maximize the reward — transforming raw language models into helpful, harmless, and honest conversational AI assistants.

RLHF Pipeline:
- Supervised Fine-Tuning (SFT): pre-trained base model is fine-tuned on high-quality instruction-response pairs (10K-100K examples); produces a model that follows instructions but may still generate unhelpful, harmful, or inaccurate responses
- Reward Model Training: human annotators compare pairs of model responses to the same prompt and indicate which is better; a reward model (initialized from the SFT model) is trained to predict human preferences; Bradley-Terry model: P(response_A > response_B) = σ(r(A) - r(B))
- Policy Optimization (PPO): the SFT model (policy) generates responses to prompts; the reward model scores each response; PPO (Proximal Policy Optimization) updates the policy to increase reward while staying close to the SFT model (KL penalty prevents reward hacking); iterative online training generates new responses each batch
- KL Constraint: KL divergence penalty between the policy and the reference SFT model prevents the policy from exploiting reward model weaknesses; without KL constraint, the model degenerates into producing adversarial outputs that maximize reward score but are nonsensical or formulaic

Direct Preference Optimization (DPO):
- Eliminating the Reward Model: DPO reparameterizes the RLHF objective to directly optimize the language model on preference pairs without training a separate reward model; loss function: L = -log σ(β · (log π(y_w|x)/π_ref(y_w|x) - log π(y_l|x)/π_ref(y_l|x))) where y_w is the preferred and y_l is the dispreferred response
- Advantages: eliminates reward model training, PPO hyperparameter tuning, and online generation; reduces the pipeline from 3 stages to 2 stages (SFT → DPO); stable training without reward hacking failure modes
- Offline Training: DPO trains on fixed datasets of preference pairs rather than generating new responses; simpler but may not explore the policy's current output distribution as effectively as online PPO
- Variants: IPO (Identity Preference Optimization) regularizes differently to prevent overfitting; KTO (Kahneman-Tversky Optimization) works with binary feedback (thumbs up/down) instead of comparisons; ORPO combines SFT and preference optimization in a single stage

Human Annotation:
- Preference Collection: annotators see a prompt and two model responses; they select which response is better based on helpfulness, accuracy, harmlessness, and overall quality; inter-annotator agreement is typically 70-80% for subjective preferences
- Annotation Scale: initial RLHF (InstructGPT) used ~40K preference comparisons; modern alignment requires 100K-1M comparisons for robust reward model training; labor cost $100K-$1M for high-quality annotation campaigns
- Constitutional AI (CAI): replaces some human annotation with model-generated evaluation; the model critiques its own outputs against a set of principles (constitution); reduces annotation cost while maintaining alignment quality
- Synthetic Preferences: using stronger models (GPT-4) to generate preference data for training weaker models; effective for bootstrapping alignment but may propagate the stronger model's biases

Challenges:
- Reward Hacking: the policy finds outputs that score highly on the reward model but don't satisfy actual human preferences (e.g., verbose but empty responses, sycophantic agreement); regularization and iterative reward model updates mitigate but don't eliminate
- Alignment Tax: RLHF may degrade raw capability (coding, math) while improving helpfulness and safety; careful balancing of alignment training intensity preserves base model capabilities
- Scalable Oversight: as models become more capable, human annotators may be unable to evaluate response quality for complex tasks; debate, recursive reward modeling, and AI-assisted evaluation are proposed solutions

RLHF and DPO are the techniques that transform raw language models into the helpful AI assistants used by hundreds of millions of people — bridging the gap between next-token prediction and aligned, instruction-following behavior that makes conversational AI useful and safe for deployment.