Reclor
ReClor (Reading Comprehension from Examinations for Logical Reasoning) is a benchmark built from standardized graduate-admissions exam questions, primarily LSAT and GMAT-style critical reasoning problems, designed to test whether AI systems can analyze arguments, identify assumptions, and perform structured logical reasoning rather than simple pattern matching. Introduced by Yu et al. in 2020, ReClor became one of the clearest stress tests for the gap between language fluency and genuine reasoning, because the benchmark is deliberately constructed from questions meant to fool intelligent humans, not to reward superficial lexical cues.
What ReClor Contains
Each ReClor example typically includes:
- A short passage presenting an argument or scenario
- A question asking for the logically correct conclusion, assumption, weakening statement, strengthening statement, or explanation
- Four answer choices, often all plausible on first read
Typical question types:
- Weaken the argument
- Strengthen the argument
- Identify the assumption
- Infer the conclusion
- Resolve the paradox
- Parallel reasoning
This mirrors the structure of LSAT Logical Reasoning sections, where success depends on carefully modeling the argument rather than recalling facts.
Why ReClor Is Hard
ReClor is difficult because the wrong choices are intentionally crafted to look reasonable. A model must separate:
- What the passage explicitly states
- What the argument implicitly assumes
- What would genuinely affect the conclusion
- What is merely related but logically irrelevant
For example, in a weaken question, a distractor answer may mention the same nouns and context as the passage but not actually undermine the causal or logical link in the argument. Models that rely on semantic similarity often pick these distractors.
What Skills ReClor Measures
| Skill | Why It Matters |
|---|---|
| Argument structure tracking | Identify premises, conclusions, and hidden assumptions |
| Counterfactual reasoning | Test what happens if a new fact is introduced |
| Distractor resistance | Ignore plausible but irrelevant answer choices |
| Abstract reasoning | Generalize beyond surface wording |
| Careful reading | Small wording changes can reverse logical meaning |
This makes ReClor different from ordinary reading comprehension. The challenge is not reading the passage, but reasoning about it correctly.
Historical Performance Trend
ReClor was especially notable because early transformer models that looked strong on many NLP benchmarks performed poorly:
- Random baseline: 25% for four-choice questions
- Early BERT/RoBERTa systems: only modestly above random on hard subsets
- Larger pretrained models improved, but progress was slower than on other datasets
- Chain-of-thought prompting and frontier LLMs later produced major gains
Why the slow progress? Because ReClor penalizes shortcut learning. Many NLP benchmarks contain annotation artifacts or lexical regularities that models can exploit. ReClor, drawn from exam questions refined by humans to test reasoning, contains fewer such shortcuts.
Why ReClor Matters in the LLM Era
Modern LLMs are much better at ReClor than earlier models, especially when given:
- Chain-of-thought prompting
- Self-consistency sampling
- Debate or verifier-style reranking
- Tool-assisted logic checking in some experimental setups
But ReClor still matters because it probes a failure mode that remains important in production: a model can sound persuasive while following invalid reasoning. This matters in:
- Legal analysis
- Financial decision support
- Medical explanation systems
- Compliance workflows
- Multi-step agent planning
A fluent but logically weak model is dangerous in all of these domains.
Comparison With Related Benchmarks
| Benchmark | Focus | Difference From ReClor |
|---|---|---|
| MMLU | Broad academic knowledge | More breadth, less concentrated logical trap design |
| HellaSwag | Commonsense completion | More world knowledge, less explicit argument structure |
| GSM8K | Arithmetic reasoning | Numeric reasoning rather than verbal logic |
| LogiQA | Logical reasoning from text | Similar family, but ReClor is closely tied to LSAT and GMAT quality |
| ARC | Science exam QA | Fact and reasoning mix, less adversarial logic structure |
Main Limitations
- Small dataset size by modern LLM standards
- English-only and culturally specific to Western standardized tests
- Multiple-choice format allows some answer elimination strategies
- Frontier models are narrowing the benchmark's headroom
Even with those limitations, ReClor remains one of the most respected benchmarks for verbal logical reasoning. It asks a sharper question than many general NLP tests: not whether a model can read, but whether it can follow an argument carefully enough to avoid being fooled by plausible nonsense.