Absolute Grading

Absolute grading is an evaluation approach where a model's output is scored individually on a numeric scale (e.g., 1–5 or 1–10) against defined criteria, without comparison to another response. Unlike pairwise comparison, each response is evaluated on its own merits.

How It Works

• Input: A prompt and a single model response.
• Criteria: The evaluator (human or LLM) assesses the response against explicit rubric dimensions — such as accuracy, helpfulness, coherence, safety, and completeness.
• Output: A numeric score and optionally a written justification.

Common Rating Scales

• Binary (0/1): Simple pass/fail — meets criteria or doesn't.
• Likert (1–5): Five-point scale from "very poor" to "excellent." Most common in research.
• Fine-Grained (1–10): More discriminative but harder for humans to use consistently.
• Multi-Dimensional: Separate scores for different quality dimensions (accuracy: 8, fluency: 9, safety: 10).

Advantages

• Independent Scoring: Each response gets a score without needing another response for comparison.
• Scalability: Can evaluate many responses in parallel without generating all pairwise combinations.
• Dimensional Analysis: Multi-criteria scoring reveals which aspects of quality are strong or weak.

Disadvantages

• Calibration Issues: Different evaluators interpret scales differently — one person's 7 is another's 5. Inter-annotator agreement is typically lower than for pairwise comparisons.
• Central Tendency Bias: Evaluators tend to cluster around middle scores, avoiding extremes.
• Difficult for Subtle Differences: Two responses of similar quality may receive the same score, losing discriminative information.

Best Practices

• Provide detailed rubrics with examples for each score level.
• Use calibration sets where evaluators score the same examples to ensure consistency.
• Consider combining absolute grading with pairwise comparison for the most comprehensive evaluation.

Absolute grading is used in benchmarks like MT-Bench (1–10 scoring by GPT-4) and many production quality monitoring systems.