Crows-Pairs

CrowS-Pairs is the fairness benchmark based on paired minimally different sentences that contrast stereotypical and anti-stereotypical statements - it measures whether models assign higher likelihood to biased phrasing.

What Is CrowS-Pairs?

• Definition: Dataset of sentence pairs differing mainly in stereotype direction for protected groups.
• Evaluation Mechanism: Compare model preference or pseudo-likelihood between paired sentences.
• Bias Dimensions: Covers categories such as race, gender, religion, age, and disability.
• Metric Goal: Lower stereotype-preference bias indicates fairer language modeling behavior.

Why CrowS-Pairs Matters

• Fine-Grained Testing: Minimal-pair setup isolates bias signal from unrelated content variation.
• Model Comparison: Supports consistent fairness ranking across architectures and versions.
• Mitigation Validation: Sensitive to changes from debiasing interventions.
• Interpretability: Pairwise outcomes are easy to inspect for qualitative error analysis.
• Governance Support: Useful for regression monitoring in release pipelines.

How It Is Used in Practice

• Batch Scoring: Evaluate model likelihood preference across full pair set by subgroup.
• Disparity Breakdown: Report results by protected category to localize weaknesses.
• Integrated Review: Use with complementary benchmarks to avoid single-metric blind spots.

CrowS-Pairs is a widely used minimal-pair fairness benchmark for LLMs - pairwise stereotype preference testing provides clear, actionable bias diagnostics for model evaluation workflows.