Medqa
MedQA is the medical question answering benchmark derived from the United States Medical Licensing Examination (USMLE) and equivalent exams in China and Taiwan — testing whether AI can answer the multi-step clinical reasoning questions that physicians must answer to obtain medical licensure, requiring integration of basic science knowledge, pathophysiology, clinical presentation, and treatment guidelines.
What Is MedQA?
- Origin: Jin et al. (2021).
- Scale: 61,097 multiple-choice questions (4-5 options) across three datasets.
- Sources: USMLE Step 1/Step 2/Step 3, Chinese National Medical Licensing Examination (CNMLE), Taiwanese Medical Licensing Exam.
- Format: Clinical vignette (2-8 sentences describing a patient presentation) + question + answer choices.
- Languages: English (USMLE), Simplified Chinese (CNMLE), Traditional Chinese (Taiwan).
- Difficulty: Requires qualifying for medical licensure — questions test clinical reasoning at the level required of practicing physicians.
The USMLE Clinical Vignette Format
A typical MedQA question:
"A 58-year-old woman presents with 3 days of progressive shortness of breath, orthopnea, and bilateral leg edema. She has a history of hypertension treated with lisinopril. On examination, JVP is elevated, crackles bilaterally at both bases, and an S3 gallop is present. BNP is 1,240 pg/mL. Which of the following is the most appropriate next step in management?
A. IV furosemide
B. Cardiac catheterization
C. Echocardiography
D. Metoprolol titration
E. Digoxin loading dose"
Answering correctly requires: recognizing acute decompensated heart failure from the constellation of signs, knowing that diuresis (furosemide) is first-line acute management, and ruling out premature invasive investigation.
Why MedQA Is Hard
- Multi-Step Clinical Reasoning: Questions require recognizing the diagnosis, understanding the pathophysiology driving each finding, and applying treatment guidelines — not just recalling isolated facts.
- Synthesis Across Disciplines: USMLE Step 1 integrates biochemistry, anatomy, physiology, microbiology, and pharmacology in single vignettes.
- Distractor Quality: Wrong answer choices are common traps — correct drugs for the wrong indication, appropriate management for the misdiagnosed condition.
- Context Sensitivity: The same symptom constellation has different correct answers depending on patient age, comorbidities, and acuity.
Performance Timeline
| Model | MedQA (USMLE) Accuracy |
|---|---|
| Human (passing threshold) | 60% |
| Human (first-time takers) | ~67% |
| GPT-3 (few-shot) | 44.7% |
| PubMedBERT fine-tuned | 55.9% |
| GPT-3.5 | 57.6% |
| Med-PaLM | 67.6% |
| GPT-4 | 86.7% |
| Med-PaLM 2 | 86.5% |
GPT-4 and Med-PaLM 2 exceeded expert physician performance on MedQA — a landmark result that triggered significant discussion about AI-assisted clinical decision support.
Why MedQA Matters
- Clinical Decision Support Validation: A system that scores above 80% on MedQA can assist physicians with differential diagnosis and treatment selection at near-expert level.
- AI Medical Licensing: MedQA provides the objective standard for "can AI practice medicine?" — a question with profound regulatory and liability implications.
- Multilingual Medical AI: The Chinese and Taiwanese versions enable medical AI for 1.4 billion people in a healthcare system with different epidemiological patterns and drug formularies.
- Reasoning Chain Development: MedQA vignettes are ideal training data for medical CoT — step-by-step clinical reasoning chains derived from USMLE explanations.
MedQA is the medical licensing exam for AI — measuring whether language models can reason through the complex clinical scenarios that define physician competence, with performance crossing the passing threshold representing a genuine milestone in AI-assisted healthcare.