Numglue

NumGLUE is the multi-task benchmark specifically targeting the numerical reasoning capabilities of NLP models — aggregating 8 distinct datasets that require quantitative understanding embedded in natural language, exposing the systematic weakness of pre-BERT and early transformer models in treating numbers as meaningful quantities rather than arbitrary tokens.

What Is NumGLUE?

• Scale: ~101,000 examples across 8 tasks.
• Format: Multi-task evaluation — each task tests a different facet of numerical reasoning.
• Motivation: Standard NLU benchmarks (GLUE, SuperGLUE) contain minimal numerical content. NumGLUE fills this gap by explicitly requiring arithmetic, comparison, and quantitative inference.

The 8 NumGLUE Tasks

Task 1 — Arithmetic QA (MathQA origins):
- Fill-in-the-blank math word problems.
- "If a car travels 60 mph for 2.5 hours, the distance traveled is ___ miles."

Task 2 — Fill-in-the-Blank NLI:
- Given a context with numbers, fill in a missing quantity that makes an entailment valid.

Task 3 — Numerical QA (DROP-style):
- Discrete operations over reading comprehension passages: add, subtract, sort, count.
- "How many more points did Team A score than Team B?" over sports reports.

Task 4 — Comparison (greater/less/equal):
- "A cheetah runs at 70 mph. A human runs at 10 mph. The cheetah runs ___ times faster."

Task 5 — Listing / Sorting:
- Sort a set of quantities in ascending or descending order from a paragraph.

Task 6 — Number Conversion / Format:
- Recognize equivalent representations (fractions, decimals, percentages).

Task 7 — Unit Conversion:
- "Convert 3.5 miles to kilometers." Requires world knowledge of conversion factors.

Task 8 — Quantitative NLI:
- "Context states 5 million people. Does it entail that more than 3 million are affected?" Binary yes/no.

Why NumGLUE Matters

• Tokenization Blindness: Standard BPE tokenizers split numbers into sub-word pieces ("1995" → "19" + "95") losing magnitude information. NumGLUE highlighted this as a systematic failure mode.
• Embedding Space Numbers: Research (Wallace et al., 2019) showed that BERT representations lack a coherent linear number line — numbers close in value are not close in embedding space. NumGLUE quantified the performance consequence.
• Cross-Task Transfer: A model that handles arithmetic well should also handle comparison well (they require the same underlying magnitude understanding). NumGLUE tests whether this transfer actually occurs.
• Real-World Ubiquity: Numbers appear everywhere — financial reports, scientific papers, news articles, contracts. A model without numerical grounding fails on all of these.
• Hallucination Root Cause: LLMs that generate plausible-sounding but numerically wrong facts (dates, statistics, measurements) often fail because of the exact weaknesses NumGLUE measures.

Performance Results

Improvements from Number-Aware Architecture

Specialized models (NumBERT, GenBERT) that modify tokenization for numbers (digit-by-digit encoding, numericalized representations, injection of number magnitude embeddings) consistently outperform standard transformer baselines by 8-15 points.

Connection to DROP and TATQA

NumGLUE overlaps conceptually with:
- DROP (Discrete Reasoning Over Paragraphs): Reading comprehension with numerical operations.
- TATQA: Table and text QA with financial arithmetic.
- FinQA: Financial report numerical reasoning.

All require numerical grounding; NumGLUE is distinctive in explicitly categorizing the required operation type across 8 distinct dimensions.

NumGLUE is literacy plus numeracy combined — testing the critical intersection where language understanding meets quantitative reasoning, ensuring AI models can handle the numerical fabric of real-world text rather than treating every number as an arbitrary symbol.