Alignment

LLM alignment is the process of training language models to behave in accordance with human values and intentions — using techniques like RLHF (Reinforcement Learning from Human Feedback) and DPO (Direct Preference Optimization) to make models helpful, harmless, and honest, ensuring AI systems do what users actually want rather than just predicting the next token.

What Is Alignment?

• Definition: Training AI to act according to human preferences and values.
• Goal: Models that are helpful, harmless, and honest (HHH).
• Challenge: Base models predict text, not "good" behavior.
• Methods: RLHF, DPO, Constitutional AI, instruction tuning.

Why Alignment Matters

• Safety: Prevent harmful, dangerous, or illegal outputs.
• Usefulness: Models should actually help with user tasks.
• Trust: Users must be able to rely on AI responses.
• Control: Aligned models follow instructions and boundaries.
• Scaling: Alignment must hold as models become more capable.
• Existential: Long-term AI safety depends on alignment.

The Alignment Problem

Base Model Behavior:

Prompt: "How do I pick a lock?"

Base Model (unaligned):
→ Has seen lockpicking instructions in training data
→ May helpfully provide detailed instructions
→ No concept of "should I answer this?"

Aligned Model:
→ Considers potential harm of response
→ May refuse or provide only legal context
→ Balances helpfulness with safety

Alignment Methods

Supervised Fine-Tuning (SFT):
- Train on demonstrations of desired behavior.
- (Instruction, good-response) pairs.
- Shows what good responses look like.
- Foundation for further alignment.

RLHF (Reinforcement Learning from Human Feedback):

Step 1: Collect comparisons
   Prompt → Response A vs Response B
   Human labels which is better

Step 2: Train reward model
   Reward(prompt, response) → score
   Predicts human preference

Step 3: Optimize policy
   Use PPO to maximize reward
   Policy = original model + value head
   Iterate with fresh feedback

DPO (Direct Preference Optimization):

Insight: Skip reward model, directly use preferences

Loss = -log σ(β × (log π(y_w|x)/π_ref(y_w|x) 
                  - log π(y_l|x)/π_ref(y_l|x)))

y_w = preferred response
y_l = dis-preferred response

Simpler, often matches RLHF quality

Constitutional AI (CAI):

1. Generate response to harmful prompt
2. Critique: "Does this response violate [principle]?"
3. Revise: "Write a response that doesn't..."
4. Fine-tune on revised responses
5. RLHF with AI feedback (RLAIF)

Principles: List of behavioral guidelines
Reduces need for human labeling

Alignment Comparison

Method       | Human Data  | Complexity | Quality
-------------|-------------|------------|----------
SFT          | Demos       | Simple     | Baseline
RLHF         | Comparisons | Complex    | Best
DPO          | Comparisons | Medium     | Near RLHF
CAI/RLAIF    | Principles  | Medium     | Good

Challenges in Alignment

• Specification: Hard to fully specify "human values."
• Gaming: Models can learn to satisfy reward without true alignment.
• Distribution Shift: Alignment may not generalize to new situations.
• Scalability: Alignment methods must scale with model capability.
• Robustness: Aligned models can still be jailbroken.
• Cultural Variation: Values differ across cultures.

Current State

• Modern chat models (ChatGPT, Claude, etc.) are heavily aligned.
• Alignment reduces raw capability in exchange for safety.
• Open models available in aligned and base versions.
• Active research on more robust alignment methods.

LLM alignment is the critical challenge for beneficial AI — getting powerful AI systems to reliably do what we want, avoid what we don't want, and behave ethically is essential for AI to be a positive force, making alignment research one of the most important areas in AI development.