Constitutional AI

Constitutional AI

What is Constitutional AI?
Constitutional AI (CAI) is an alignment approach by Anthropic that uses a set of principles to guide AI behavior, reducing reliance on human feedback for every scenario.

Core Concept
Instead of collecting human feedback for every case, define principles (a "constitution") that the model uses for self-improvement.

The CAI Process

Stage 1: Supervised Learning with Self-Critique

1. Generate initial response
2. Critique response against principles
3. Revise response based on critique
4. Fine-tune on revised responses

Stage 2: RLHF with AI Feedback (RLAIF)

1. Generate response pairs
2. AI evaluates which is better (using principles)
3. Train reward model on AI preferences
4. RLHF as usual

Example Constitution Principles

- Be helpful, harmless, and honest
- Refuse to help with illegal activities
- Correct mistakes when pointed out
- Express uncertainty when appropriate
- Avoid stereotypes and bias
- Protect user privacy
- Do not pretend to be human

Self-Critique Example

[Original response]: [potentially harmful content]

[Critique]: This response violates the principle of being harmless
because it provides information that could be used to harm others.

[Revised response]: I cannot provide that information because it
could be used to cause harm. Instead, let me suggest...

Benefits
| Benefit | Description |
|---------|-------------|
| Scalable | Less human annotation needed |
| Transparent | Principles are explicit |
| Consistent | Same principles applied everywhere |
| Maintainable | Update principles as needed |

Implementation Approach

def constitutional_revision(response: str, principles: list) -> str:
    # Self-critique
    critique = llm.generate(f"""
Given these principles: {principles}

Critique this response:
{response}

Identify any violations of the principles.
    """)

    # Revision
    revised = llm.generate(f"""
Original response: {response}
Critique: {critique}

Generate a revised response that addresses the critique
while remaining helpful.
    """)

    return revised

Comparison to RLHF
| Aspect | RLHF | CAI |
|--------|------|-----|
| Human involvement | Every preference | Define principles once |
| Scalability | Limited by humans | Highly scalable |
| Transparency | Implicit in data | Explicit principles |
| Consistency | Varies with annotators | Consistent |

Constitutional AI is foundational to Anthropic Claude models.