Bioasq
BioASQ is the large-scale biomedical question answering and information retrieval challenge — running since 2013 as an annual shared task requiring systems to retrieve relevant PubMed articles, extract exact answer snippets, and generate well-formed natural language answers to biomedical research questions, directly targeting the information overload problem in scientific literature.
What Is BioASQ?
- Origin: Tsatsaronis et al. (2015); annual challenge run by the BioASQ organization.
- Scale: 4,234+ biomedical questions (growing annually); linked to the full PubMed corpus (35M+ articles).
- Format: Expert-formulated questions by biomedical scientists + gold standard annotations for relevant documents, snippets, exact answers, and ideal answers.
- Question Types: Yes/No, Factoid (single entity answer), List (multiple entities), Summary (paragraph answer).
- Challenge Phases: Phase A (document and snippet retrieval) and Phase B (answer generation).
The Four Question Types
Yes/No: "Is the protein BRCA1 involved in DNA repair?" → "yes" + supporting snippets.
Factoid: "What is the mechanism of action of imatinib?" → "selective BCR-ABL tyrosine kinase inhibitor" + exact snippet spans.
List: "Which genes are known to be associated with cystic fibrosis?" → ["CFTR", "TGFB1", "MUC5B", ...] + supporting documents.
Summary: "What is known about the role of PCSK9 in cholesterol metabolism?" → Multi-sentence synthesized answer from retrieved literature.
Why BioASQ Is Hard
- Biomedical Terminology: Questions use precise MESH/UMLS terminology ("phospholipase A2 group VII" not "platelet-activating factor acetylhydrolase"). Systems must handle synonym explosion in biomedical nomenclature.
- Literature Scale: PubMed grows by ~1 million articles per year. Systems must retrieve the relevant needle from 35M+ papers.
- Multi-Hop Evidence: Summary questions require synthesizing findings from multiple conflicting or complementary studies.
- Answer Granularity: For factoid questions, the exact answer span (gene name, drug name, measurement value) must be extracted — not just the document.
- Scientific Precision: "Which kinase phosphorylates Ser473 of AKT?" has a specific correct answer (PDK2/mTORC2) with no tolerance for close-but-wrong responses.
Performance Results (BioASQ Phase B)
| System | Factoid MRR | List F1 | Yes/No Accuracy | Summary ROUGE |
|---|---|---|---|---|
| IR baseline | 0.22 | 0.31 | 72% | 0.28 |
| BioBERT fine-tuned | 0.48 | 0.49 | 81% | 0.38 |
| PubMedBERT | 0.51 | 0.52 | 83% | 0.41 |
| GPT-4 + RAG (PubMed) | 0.62 | 0.58 | 87% | 0.52 |
| BioGPT (domain-pretrained) | 0.66 | 0.60 | 88% | 0.55 |
Why BioASQ Matters
- Research Acceleration: Scientists spend ~20% of their work time searching literature. BioASQ-capable systems can instantly synthesize the current evidence base for any biomedical question.
- Clinical Evidence Retrieval: At the point of care, physicians need rapid answers to specific drug-mechanism, dosing, or interaction questions — BioASQ tests exactly this capability.
- Drug Discovery Applications: "Which proteins interact with target X?" and "Which compounds inhibit pathway Y?" are BioASQ-style queries for computational drug target identification.
- Systematic Review Foundation: Literature-grounded QA systems can semi-automate the retrieval and evidence extraction phases of systematic reviews.
- Domain Pretraining Validation: BioASQ is the primary benchmark validating that BioBERT, PubMedBERT, BioGPT, and BioMedLM outperform generic models — demonstrating the value of biomedical corpus pretraining.
BioASQ is the biomedical literature intelligence test — measuring whether AI can navigate the 35 million papers of PubMed to retrieve, extract, and synthesize precise scientific answers to the questions that drive biomedical research and clinical evidence-based practice.