Drop Benchmark

DROP (Discrete Reasoning Over Paragraphs) is a reading comprehension benchmark requiring numerical reasoning operations like addition, counting, and sorting over text passages.

## What Is DROP?

• Size: 96,000+ question-answer pairs
• Source: Wikipedia paragraphs (sports, history)
• Challenge: Requires arithmetic, not just text extraction
• Operations: Count, add, subtract, compare, sort

## Why DROP Matters

Most QA benchmarks test text extraction. DROP tests whether models truly understand quantities and can perform discrete reasoning.

DROP Example:
Passage: "The Lions scored 14 points in the first
quarter, 7 in the second, and 21 in the third."

Question: "How many total points did the Lions
score in the first two quarters?"

Reasoning: 14 + 7 = 21

Traditional QA: Extract "14" or "7"
DROP: Compute 14 + 7 = 21 (not directly in text)

Model Performance (2024):
| Model | DROP F1 |
|-------|---------|
| GPT-4 | ~88% |
| Human | ~96% |
| BERT (original) | ~31% |
| NumNet+ | ~83% |

Key: Models need both reading comprehension AND numerical reasoning.