Mmlu

MMLU (Massive Multitask Language Understanding) is the benchmark of 57 academic and professional subjects — from elementary mathematics to medical licensing exams — that became the de facto standard for measuring LLM knowledge depth and breadth — first exposing the massive gap between early language models and human expert performance, then tracking the rapid progress that brought AI to near-expert levels within three years.

What Is MMLU?

• Scale: 15,908 multiple-choice questions across 57 subjects.
• Format: 4-option multiple-choice (A/B/C/D) with a single correct answer.
• Subjects: Organized into four domains — STEM (math, physics, chemistry, biology, computer science), Humanities (history, philosophy, law), Social Sciences (economics, psychology, sociology), and Professional (medical licensing, legal bar, accounting).
• Difficulty: Ranges from high-school level (elementary mathematics) to professional certification level (USMLE, LSAT, CPA exams).
• Human Baseline: Non-expert humans score ~34.5% (essentially random for hard topics); expert humans score ~89.8%.

The 57 Subjects

STEM:
- Abstract Algebra, College Chemistry, College Mathematics, College Physics, Computer Security, Electrical Engineering, High School Biology, High School Chemistry, Machine Learning, Virology

Humanities:
- High School World History, International Law, Jurisprudence, Logical Fallacies, Moral Disputes, Philosophy, Prehistory, World Religions

Social Sciences:
- Econometrics, High School Government and Politics, Human Sexuality, Professional Psychology, Sociology

Professional / Applied:
- Clinical Knowledge, Medical Genetics, Anatomy, Professional Medicine, Professional Law, Professional Accounting, Nutrition, Management

Why MMLU Became the Standard

• GPT-3 Failure (2020): When MMLU was released, GPT-3 (175B parameters) scored ~43% — barely above random chance on hard subjects. This galvanized the field.
• Single Number Comparability: MMLU provides one average accuracy across all 57 subjects — making it easy to compare models in papers and leaderboards.
• Knowledge vs. Reasoning: MMLU tests factual recall AND multi-step reasoning (medical diagnosis questions, legal analysis). This dual test exposes models that rely solely on pattern matching.
• Broad Coverage: No single training set can cover all 57 domains — MMLU tests genuine cross-domain knowledge transfer.
• Progressive Bar: GPT-4 (~86%+), Claude 3 Opus (~88%), Gemini Ultra (~90%) approaching but not exceeding average expert human performance.

Performance Timeline

MMLU Variants and Extensions

• MMLU-Pro: Harder version with 10 answer choices and more reasoning-heavy questions.
• MMLU-Redux: Cleaned version fixing annotation errors in the original (~450 questions re-evaluated).
• Multilingual MMLU: Translated versions testing cross-lingual knowledge transfer.
• Domain-Specific: Medical MMLU, Legal MMLU subsets for specialized evaluation.

Limitations

• Knowledge Contamination: MMLU questions appear in many pretraining corpora; models may have memorized answers rather than reasoning to them.
• Answer Format Bias: 4-choice format allows positional biases ("C is always correct" patterns in some models).
• No Explanation Required: Correct answer without reasoning path — models can be right for wrong reasons.
• Static Knowledge: Questions frozen at release date — medical and legal knowledge evolve, making some answers outdated.

Evaluation Best Practices

• 5-shot Prompting: Standard evaluation uses 5 few-shot examples per subject to establish format.
• Chain-of-Thought: MMLU-CoT variants require step-by-step reasoning before selecting the answer.
• Calibration: Strong models should be well-calibrated — high confidence on questions they answer correctly.

MMLU is the comprehensive IQ test for language models — measuring not just what a model has memorized but whether it can integrate knowledge across 57 disciplines to correctly answer questions that require the depth of a medical professional, lawyer, or scientist.