Learning from Human Feedback

RLHF (Reinforcement Learning from Human Feedback) is the technique of training AI models using human preferences as the reward signal — instead of a hand-crafted reward function, humans compare model outputs, these preferences train a reward model, and the reward model guides RL-based policy optimization.

RLHF Pipeline

• SFT: Supervised Fine-Tuning on curated demonstrations — baseline model.
• Reward Model: Train a reward model $R(x, y)$ on human preference comparisons: "output A is better than output B."
• RL Fine-Tuning: Optimize the SFT model with PPO to maximize the learned reward $R$, with a KL penalty to stay near SFT.
• Iteration: Collect more preferences on the RL-tuned model, retrain reward model, re-optimize.

Why It Matters

• Alignment: RLHF aligns AI behavior with human values and preferences — the key technique behind ChatGPT, Claude.
• Beyond Demonstrations: Preferences are easier to provide than demonstrations — comparing is easier than generating.
• LLMs: RLHF transformed language models from next-word predictors into helpful, harmless assistants.

RLHF is aligning AI with human preferences — using human comparisons to create a reward signal for training helpful, safe AI systems.