Med Palm
Med-PaLM is a medical domain language model developed by Google Research that was the first AI system to achieve "expert" level performance on the US Medical Licensing Exam (USMLE) — with Med-PaLM 2 reaching 86.5% accuracy on MedQA (surpassing the ~60% passing threshold by a wide margin), built by fine-tuning Google's PaLM foundation model using instruction tuning on curated medical question-answering datasets (MultiMedQA) and rigorously evaluated for clinical safety, accuracy, and potential harm.
What Is Med-PaLM?
- Definition: A series of medical AI models built on top of Google's PaLM (Pathways Language Model) — fine-tuned with medical instruction data and evaluated against clinical expert benchmarks, designed for medical question answering, clinical reasoning, and health information retrieval.
- Med-PaLM 1: First AI to pass the USMLE with ~67% accuracy — demonstrating that LLMs fine-tuned on medical data could match the knowledge threshold required of human physicians.
- Med-PaLM 2: Dramatically improved to 86.5% accuracy on MedQA — reaching "expert physician" level performance and significantly outperforming both ChatGPT (~60%) and the original Med-PaLM on all medical benchmarks.
- MultiMedQA: Google's comprehensive medical evaluation benchmark combining MedQA (USMLE questions), MedMCQA (Indian medical entrance exams), PubMedQA (biomedical literature questions), and additional clinical QA datasets.
Performance Evolution
| Model | MedQA (USMLE) | MedMCQA | PubMedQA | Notes |
|---|---|---|---|---|
| Med-PaLM 1 (2022) | 67.6% | 57.6% | 79.0% | First AI to pass USMLE |
| Med-PaLM 2 (2023) | 86.5% | 72.3% | 81.8% | Expert physician level |
| GPT-4 (2023) | ~86% | ~70% | ~80% | Comparable to Med-PaLM 2 |
| ChatGPT (GPT-3.5) | ~60% | ~55% | ~75% | Near passing threshold |
Safety and Evaluation
- Clinical Expert Review: Med-PaLM responses evaluated by panels of practicing physicians across 9 dimensions — scientific accuracy, potential harm, evidence of reasoning, demographic bias, likelihood of clinical action disagreement.
- Harm Assessment: Every response evaluated for potential patient harm — critical for medical AI where incorrect advice could lead to delayed treatment, wrong medication, or missed diagnoses.
- Physician Comparison: In blind evaluations, Med-PaLM 2 answers were preferred over physician answers 40% of the time — showing AI can match or exceed human quality on structured medical questions.
- Limitations Acknowledged: Google explicitly documents that Med-PaLM is not approved for clinical use — it's a research demonstration, not a diagnostic tool, and real clinical deployment requires regulatory approval (FDA, CE marking).
Deployment and Access
- Pilot Program: Google piloting Med-PaLM with select healthcare partners (Mayo Clinic, HCA Healthcare) for clinical decision support — not direct patient interaction.
- Google Cloud integration: Available through Vertex AI for approved healthcare research institutions.
- Not publicly available: Unlike open-source alternatives (Meditron), Med-PaLM remains closed-access due to safety concerns around uncontrolled medical AI deployment.
Med-PaLM is the benchmark-setting medical AI that proved language models can reach physician-level accuracy on structured medical examinations — while simultaneously demonstrating the critical importance of rigorous safety evaluation, harm assessment, and controlled deployment for AI systems operating in high-stakes clinical domains.