Inter-Annotator Agreement
Inter-annotator agreement (IAA) measures how consistently multiple human evaluators assign the same labels or scores to the same data. It is a critical quality metric for any dataset, benchmark, or evaluation process that relies on human judgment.
Why IAA Matters
- Data Quality Signal: Low agreement suggests the task is poorly defined, guidelines are unclear, or the task is inherently ambiguous.
- Upper Bound on ML Performance: If humans can't agree on the correct label, a machine learning model trained on that data has an inherent ceiling on achievable accuracy.
- Evaluation Validity: Benchmarks with low IAA produce unreliable rankings — random variation in labels means model comparisons are noisy.
Common IAA Metrics
- Percent Agreement: Simply the fraction of examples where annotators agree. Easy to compute but doesn't account for chance agreement.
- Cohen's Kappa (κ): Measures agreement between two annotators, correcting for chance agreement. Values: 0 = chance, 1 = perfect agreement.
- Fleiss' Kappa: Extends Cohen's Kappa to more than two annotators.
- Krippendorff's Alpha: Most general — handles multiple annotators, missing data, and various measurement scales (nominal, ordinal, interval, ratio).
Interpretation Guidelines (Landis & Koch)
- κ < 0.20: Poor agreement
- 0.21–0.40: Fair agreement
- 0.41–0.60: Moderate agreement
- 0.61–0.80: Substantial agreement
- 0.81–1.00: Almost perfect agreement
Best Practices
- Pilot Annotation: Have a small group annotate the same examples first, measure IAA, and refine guidelines before large-scale annotation.
- Calibration Sessions: Regular meetings where annotators discuss disagreements and align their interpretation of guidelines.
- Adjudication: For low-agreement examples, have a senior annotator or committee make the final decision.
IAA should be reported in every paper that introduces a new dataset or evaluation — it quantifies the reliability ceiling of the human labels.