Human Eval

Human Evaluation for LLMs

Why Human Evaluation?
Automated metrics miss nuances that humans catch: creativity, helpfulness, safety, and overall quality.

Evaluation Types
| Type | What it Measures |
|------|------------------|
| Absolute rating | Rate response 1-5 |
| Pairwise comparison | A vs B, which is better? |
| Ranking | Order N responses |
| Task completion | Did it accomplish goal? |
| Aspect-based | Rate helpfulness, accuracy, etc. |

Annotation Platforms
| Platform | Type | Cost |
|----------|------|------|
| Amazon MTurk | Crowdsource | Low |
| Scale AI | Managed | High |
| Surge AI | Quality focus | Medium |
| Prolific | Academic | Medium |
| In-house | Expert | Variable |

MTurk Setup

import boto3

mturk = boto3.client("mturk",
    region_name="us-east-1",
    endpoint_url="https://mturk-requester.us-east-1.amazonaws.com"
)

# Create HIT
response = mturk.create_hit(
    Title="Evaluate AI Response",
    Description="Rate the quality of AI responses",
    Keywords="AI, evaluation, rating",
    Reward="0.10",
    MaxAssignments=5,
    LifetimeInSeconds=86400,
    Question=open("eval_template.xml").read()
)

Evaluation Template

Inter-Annotator Agreement

from sklearn.metrics import cohen_kappa_score

# Measure agreement between annotators
kappa = cohen_kappa_score(annotator1_ratings, annotator2_ratings)
# kappa > 0.8: strong agreement
# kappa 0.6-0.8: substantial
# kappa 0.4-0.6: moderate

Quality Control
| Method | Purpose |
|--------|---------|
| Gold questions | Catch low-effort workers |
| Redundancy | Multiple annotators per item |
| Qualification tests | Filter workers |
| Time limits | Prevent rushing |

Best Practices
- Clear, detailed instructions
- Use multiple annotators (3-5)
- Include quality control items
- Pay fairly for quality work
- Measure inter-annotator agreement