Abtest

A/B Testing and LLM Rollouts

A/B Testing for LLMs
Unlike traditional software, LLM outputs are non-deterministic and subjective. A/B testing helps you make data-driven decisions about prompt changes, model upgrades, and parameter tuning.

Experiment Design

What to Test
| Variable | Examples |
|----------|----------|
| Model | GPT-4 vs Claude-3 |
| Prompt | Short vs detailed system prompt |
| Parameters | Temperature 0.3 vs 0.7 |
| Architecture | Direct call vs RAG |

Metrics to Compare
1. Task Metrics: Accuracy, success rate
2. User Metrics: Thumbs up/down, NPS
3. Performance: Latency, cost
4. Safety: Guardrail violations

Statistical Considerations
- Sample size: Need enough users/requests for significance
- Duration: Run long enough to capture variance
- Segmentation: Consider user segments separately
- Multiple hypothesis correction: Adjust p-values for multiple metrics

Canary Deployments

Rollout Strategy

[New Model Version]
     ↓
[Deploy to 1%] → Monitor → [Issues?] → Rollback
     ↓
[Increase to 10%] → Monitor
     ↓
[Increase to 50%] → Monitor
     ↓
[Full rollout at 100%]

Monitoring During Rollout
| Stage | Traffic | Duration | Metrics to Watch |
|-------|---------|----------|------------------|
| Canary | 1% | 1 hour | Error rate, latency |
| Limited | 10% | 4 hours | User feedback |
| Broad | 50% | 1 day | Full metric suite |
| Full | 100% | Ongoing | Continuous monitoring |

Feature Flags for LLMs

**Example using LaunchDarkly pattern**
if feature_flags.is_enabled("use_gpt4_turbo", user_id):
    model = "gpt-4-turbo"
else:
    model = "gpt-4"

Online Evaluation

LLM-as-Judge
Use a capable LLM to evaluate outputs:

Rate the following response on helpfulness (1-5):
Question: {question}
Response: {response}
Rating:

Human Evaluation Sampling
- Sample 1-5% of requests for human review
- Use rating scales (1-5) for consistency
- Track inter-annotator agreement

Tools for Experimentation
| Tool | Type | Features |
|------|------|----------|
| LaunchDarkly | Feature flags | Enterprise, targeting |
| Statsig | Experimentation | Statistics focus |
| Growthbook | Open source | Self-hostable |
| Eppo | AI-focused | LLM metrics built-in |