Gsm8k

GSM8K is a benchmark dataset of 8,500 linguistically diverse grade school math word problems designed to evaluate mathematical reasoning capabilities in language models.

## What Is GSM8K?

• Size: 7,500 training + 1,000 test problems
• Difficulty: Grade school level (ages 6-12)
• Format: Word problems requiring 2-8 step solutions
• Metric: Exact match accuracy on final numerical answer

## Why GSM8K Matters

Math reasoning is a key capability for AI assistants. GSM8K tests multi-step deductive reasoning that requires understanding language and applying arithmetic.

GSM8K Example Problem:
"Janet has 3 apples. She buys 2 more apples, then
gives half of all her apples to her friend. How
many apples does Janet have?"

Solution steps:
1. Start: 3 apples
2. Buy more: 3 + 2 = 5 apples
3. Give half: 5 / 2 = 2.5 → 2 apples (integer)

Answer: 2

Model Performance (2024):
| Model | GSM8K Accuracy |
|-------|----------------|
| GPT-4 | ~92% |
| Claude 3 | ~88% |
| Llama 2 70B | ~57% |
| Chain-of-thought prompting | +10-20% improvement |