Human Evaluation

Human evaluation has humans directly judge AI output quality, providing gold-standard assessment that automated metrics approximate. Why needed: Automated metrics imperfectly correlate with quality. Humans assess nuances like creativity, helpfulness, and safety that metrics miss. Evaluation dimensions: Fluency, coherence, relevance, factuality, helpfulness, harmlessness, style, engagement. Task-specific criteria. Methods: Likert scales: Rate outputs 1-5 on dimensions. Pairwise comparison: Which of two outputs is better? Often more reliable. Ranking: Order multiple outputs by quality. Absolute rating: Assign score without comparison. Challenges: Expensive, slow, inter-annotator disagreement, subjective judgments vary. Best practices: Clear guidelines, multiple annotators, measure agreement (Cohens kappa), calibration, diverse annotator pool. Crowdsourcing: Amazon MTurk, Scale AI, Surge AI for large-scale evaluation. Quality control critical. When to use: Final model assessment, benchmark creation, validating automated metrics, safety evaluation. Trade-off: Gold standard quality but doesnt scale for training signal (hence RLHF reward models).