Climate-Fever

Climate-FEVER is the specialized fact-checking benchmark that applies the FEVER methodology to real-world climate change claims — testing whether NLP models can verify scientific statements against evidence while navigating the complexity of nuanced, contested, and cherry-picked environmental science claims.

What Is Climate-FEVER?

• Origin: Extends FEVER (Fact Extraction and VERification), adapting its pipeline to climate science.
• Claims Source: Claims scraped from real web articles, climate-skeptic sites, scientific publications, and news media — not artificially mutated sentences.
• Label Set: SUPPORTED, REFUTED, NOT_ENOUGH_INFO, and DISPUTED — the last label acknowledging that some claims are genuinely contested even among experts.
• Evidence: Wikipedia articles serve as the evidence corpus, requiring retrieval-then-verify pipelines.
• Scale: ~1,535 climate claims, each requiring multi-sentence evidence retrieval and label prediction.

Why Climate-FEVER Is Harder Than FEVER

• Scientific Nuance: A claim can be "technically true but misleading." For example, "Arctic sea ice has recovered since 2012" is factually accurate for one year but ignores the long-term declining trend. FEVER binary labels cannot handle this.
• Cherry-Picking: Climate misinformation often involves citing real data out of context. The model must understand statistical trends, not just sentence-level facts.
• Consensus vs. Outlier: Some claims cite genuine scientific papers that are outliers. The model must distinguish mainstream consensus from fringe positions.
• Temporal Sensitivity: Climate data changes. A claim verified true in 2010 may be refuted by 2023 measurements.

Why Climate-FEVER Matters

• Misinformation Combat: Climate change disinformation is among the most consequential misinformation categories. Automated fact-checking can assist human moderators at scale.
• Scientific Literacy AI: Forces models to develop genuine understanding of scientific evidence, not just surface pattern matching.
• Benchmark Gap: General fact-checking benchmarks (FEVER, VitaminC) use balanced Wikipedia-style claims. Climate-FEVER exposes domain-specific failure modes.
• Policy Impact: Erroneous AI fact-checking of climate claims can directly influence public discourse and policy narratives.
• Retrieval Pressure: Requires retrieving domain-specific scientific evidence from a large corpus — stress-testing retrieval-augmented verification systems.

The Pipeline

Step 1 — Document Retrieval:
- Given claim: "The Greenland ice sheet is gaining ice."
- Retrieve top-k Wikipedia documents about Greenland glaciology.

Step 2 — Sentence Selection:
- Extract the 5 most relevant sentences from retrieved documents.
- These become the evidence set for classification.

Step 3 — Label Prediction:
- Classify: SUPPORTED / REFUTED / NOT_ENOUGH_INFO / DISPUTED.
- State-of-the-art models (2024): ~65-72% label accuracy — well below human performance (~85%).

Comparison to Related Benchmarks

Key Challenges for Models

• Long-chain Reasoning: Climate claims often require synthesizing data from multiple evidence sentences.
• Quantitative Reasoning: "Temperatures rose 1.1°C since pre-industrial levels" requires numeric comparison.
• Counter-intuitive Labels: Factually accurate statements can still be REFUTED if they imply a false overall conclusion.

Tools & Repositories

• Dataset: Available via Hugging Face Datasets (`climate_fever`).
• Baseline Models: TF-IDF + BERT-based verifiers; DPR-based dense retrievers.
• Evaluation: Standard FEVER scorer — label accuracy + evidence F1.

Climate-FEVER is the stress test for scientific reasoning in AI — moving fact-checking from simple Wikipedia trivia into the contested, consequential territory of environmental science where model failures have real-world impact.