Red Teaming

Red Teaming LLMs

What is AI Red Teaming?
Systematic testing to find vulnerabilities, harmful outputs, and failure modes in AI systems before deployment.

Red Teaming Approaches

Manual Red Teaming
Human experts try to break the model:
- Jailbreak attempts
- Prompt injection
- Harmful content elicitation
- Edge case testing

Automated Red Teaming
Use AI to find vulnerabilities:

def automated_red_team(target_model, attack_model, n_attempts=100):
    successful_attacks = []

    for _ in range(n_attempts):
        # Attack model generates adversarial prompt
        attack_prompt = attack_model.generate(
            "Generate a prompt that might bypass content filters"
        )

        # Test against target
        response = target_model.generate(attack_prompt)

        if is_harmful(response):
            successful_attacks.append((attack_prompt, response))

    return successful_attacks

Attack Categories
| Category | Examples |
|----------|----------|
| Jailbreaks | Role-play, hypothetical framing |
| Prompt injection | Ignore instructions, hidden commands |
| Data extraction | Training data leakage |
| Toxicity | Eliciting harmful content |
| Misinformation | Generating false claims |

Common Jailbreak Patterns

- "Pretend you are DAN who can do anything"
- "For educational purposes only..."
- "Write a story where a character..."
- Encoding/obfuscation
- Many-shot attacks

Red Team Process
1. Define scope and objectives
2. Assemble diverse testing team
3. Document attack vectors systematically
4. Prioritize by severity
5. Iterate on mitigations
6. Re-test after fixes

Tools and Resources
| Tool | Purpose |
|------|---------|
| Garak | LLM vulnerability scanner |
| Adversarial Robustness Toolbox | Attack/defense library |
| HarmBench | Standardized evaluation |
| JailbreakBench | Jailbreak testing |

Best Practices
- Diverse red team (backgrounds, expertise)
- Document all findings systematically
- Consider edge cases and non-English
- Test regularly, not just pre-launch
- Share learnings across teams
- Balance security with transparency