Bbh

BBH (BIG-bench Hard) is the curated subset of 23 BIG-bench tasks where state-of-the-art language models scored below average human performance — forming the primary evaluation suite for testing Chain-of-Thought reasoning and identifying the genuine reasoning boundaries of large language models beyond knowledge retrieval.

What Is BBH?

• Origin: Derived from BIG-bench (Beyond the Imitation Game benchmark), a community effort with 204 tasks. BBH isolates the 23 tasks where PaLM-540B performed below the average human rater.
• Scale: ~6,511 total examples across 23 tasks, roughly 250-350 examples per task.
• Format: Mix of multiple-choice and free-form generation tasks.
• Purpose: Distinguishes models that reason from models that merely retrieve — the tasks require multi-step logical manipulation, not just knowledge lookups.

The 23 BBH Tasks

Logical Deduction:
- Logical Deduction (3/5/7 objects): "Alice is taller than Bob, Bob is taller than Carol. Who is tallest?" — scaled to 7 objects.
- Causal Judgement: Given a scenario, determine which event caused the outcome.
- Formal Fallacies: Identify whether a syllogism is valid or contains a named fallacy (affirming the consequent, circular reasoning, etc.).

Symbolic and Algorithmic:
- Dyck Languages: Determine if a sequence of brackets is properly nested.
- Boolean Expressions: Evaluate compound boolean logic ("True AND (False OR NOT True)").
- Multi-step Arithmetic: Evaluate expressions with multiple operations and parentheses.
- Word Sorting: Sort a list of words alphabetically — tests character-level reasoning.
- Object Counting: Count objects satisfying compound predicates.

Language and World Model:
- Disambiguation QA: Resolve pronoun references in ambiguous sentences.
- Salient Translation Error Detection: Find meaningful errors in MT output.
- Penguins in a Table: Answer questions about structured data presented in natural language tables.
- Temporal Sequences: Determine the order of events described in text.
- Tracking Shuffled Objects: Track which object ends up where after a sequence of swaps.

Knowledge and Reasoning:
- Date Understanding: Calculate dates from relative descriptions ("What date is 3 weeks after March 15?").
- Sports Understanding: Determine if a sports statement is plausible.
- Ruin Arguments: Identify what would most damage a given argument.
- Hyperbaton: Detect unusual adjective ordering in English.
- Snarky Movie Reviews: Detect if a movie review is actually negative despite positive-sounding language.

Why BBH Matters

• Chain-of-Thought Calibration: BBH is the primary benchmark showing that standard prompting fails but Chain-of-Thought (CoT) prompting dramatically improves performance. Without CoT, GPT-3.5 achieves ~50% on BBH; with CoT, ~70%+.
• Reasoning vs. Retrieval Separation: Unlike MMLU (knowledge), BBH tasks have minimal knowledge requirements — they test symbolic manipulation, logical inference, and multi-step tracking.
• Model Discrimination: BBH separates GPT-4 from GPT-3.5 more cleanly than knowledge benchmarks, because reasoning ability scales differently from memorization capacity.
• Architecture Insights: Attention mechanisms theoretically support the tracking and comparison operations in BBH — but empirically, models struggle without explicit CoT scaffolding.
• Few-Shot Sensitivity: BBH performance is highly sensitive to prompt format and few-shot example quality, making it a probe for instruction following robustness.

Performance Comparison

Evaluation Protocol

• 3-shot CoT: Provide 3 examples with step-by-step reasoning chains before the test question.
• Exact Match: Answers must exactly match the gold label (normalized for case and whitespace).
• Macro-average: Average accuracy across all 23 tasks — prevents easy tasks from dominating.

Limitations and Critiques

• Contamination Risk: Some BBH tasks (date understanding, boolean expressions) have templates easily regenerable — training data may contain similar examples.
• Task Diversity: The 23 tasks were selected by a specific metric (human > PaLM-540B) that may not reflect all important reasoning dimensions.
• English Only: No multilingual version, limiting cross-lingual reasoning assessment.

BBH is the reasoning filter for language models — isolating the 23 tasks that genuinely require thinking rather than knowing, making it the gold standard for evaluating Chain-of-Thought prompting and measuring how close AI comes to human-level logical reasoning.