Bbq
BBQ is the Bias Benchmark for Question Answering that evaluates social bias under both ambiguous and disambiguated context conditions - it tests whether models choose stereotyped answers when evidence is insufficient.
What Is BBQ?
- Definition: QA benchmark designed to measure biased response tendencies across social dimensions.
- Context Design: Includes ambiguous scenarios where correct answer should be unknown and clarified scenarios with explicit evidence.
- Bias Signal: Measures stereotype-consistent answer preference when uncertainty is present.
- Evaluation Output: Reports both accuracy and bias-related behavior metrics.
Why BBQ Matters
- Ambiguity Stress Test: Reveals whether models guess using stereotypes instead of abstaining.
- Fairness Diagnostics: Distinguishes true reasoning from socially biased shortcuts.
- Mitigation Benchmarking: Useful for assessing prompt and model debias interventions.
- Risk Relevance: QA systems are common in support and decision-assist applications.
- Governance Utility: Provides interpretable bias indicators for model release review.
How It Is Used in Practice
- Split Analysis: Evaluate performance separately on ambiguous and disambiguated subsets.
- Behavioral Metrics: Track stereotype-choice rates in uncertain contexts.
- Regression Tracking: Compare BBQ outcomes across model updates and alignment changes.
BBQ is an important fairness benchmark for QA behavior under uncertainty - it highlights whether models handle ambiguity responsibly or default to stereotype-based guessing.