Reference-Based Evaluation

Reference-based evaluation is an assessment approach where a model's output is compared against a gold standard reference answer — a human-written or expert-verified "correct" response. The similarity between the model output and the reference determines the evaluation score.

Common Reference-Based Metrics

• BLEU (Bilingual Evaluation Understudy): Measures n-gram overlap between the candidate and reference. Originally designed for machine translation. Score range 0–1.
• ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Measures recall-oriented n-gram overlap. Widely used for summarization evaluation (ROUGE-1, ROUGE-2, ROUGE-L).
• METEOR: Considers synonyms, stemming, and word order beyond simple n-gram matching. More nuanced than BLEU.
• BERTScore: Uses contextual embeddings from BERT to compute semantic similarity between tokens, capturing meaning beyond surface-level word matching.
• Exact Match (EM): Binary — does the output exactly match the reference? Used for QA tasks with short, definitive answers.

Advantages

• Automated and Fast: No human judges needed once references are created. Evaluations run in seconds.
• Reproducible: Same references, same metric, same score every time.
• Well-Understood: Decades of research on these metrics, with known strengths and limitations.

Limitations

• Reference Dependency: The evaluation is only as good as the reference. For open-ended tasks, there are many valid responses not captured by a single reference.
• Surface Matching: Metrics like BLEU penalize valid paraphrases that use different words to express the same meaning.
• Creativity Penalty: For creative writing and open-ended generation, diverging from the reference is desirable but gets penalized.
• Single Reference Problem: Using one reference misses the diversity of valid answers. Multiple references help but are expensive to create.

Reference-based evaluation works best for constrained tasks (translation, summarization, factual QA) and is less suitable for open-ended generation where reference-free evaluation is preferred.