Mt-Bench

MT-Bench (Multi-Turn Bench) is an evaluation benchmark designed to assess LLMs on multi-turn conversational ability — testing not just single-response quality but how well models handle follow-up questions, maintain context, and engage in sustained dialogue.

Benchmark Design

• 80 High-Quality Questions: Covering 8 categories with 10 questions each — writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities.
• Two-Turn Format: Each question has a first turn (initial question) and a second turn (follow-up question that builds on the first). This tests context retention and instruction following.
• Automated Judging: A strong LLM (GPT-4) scores each response on a 1–10 scale, providing reasoning for its judgment.

Example

• Turn 1: "Compose a short poem about the beauty of mathematics."
• Turn 2: "Now rewrite the poem so that every line starts with a letter that spells out the word 'MATH'." (Tests instruction following + context awareness)

Scoring

• Per-Category Scores: Models receive average scores for each of the 8 categories, revealing strengths and weaknesses.
• Overall Score: Average across all categories. Frontier models typically score 8.5–9.5 out of 10.
• Turn-by-Turn: Separate scores for first and second turns, showing how well models handle follow-ups.

Significance

• Multi-Turn Gap: MT-Bench revealed that many models that perform well on single-turn evaluations struggle with follow-ups — failing to maintain context or follow complex instructions.
• Category Insights: Models often excel at writing and humanities but struggle more with math, coding, and precise reasoning.
• Complementary to Arena: MT-Bench provides controlled, reproducible evaluation while the Chatbot Arena provides open-ended human preference signals.

Developed By: The LMSYS team at UC Berkeley, alongside the Chatbot Arena. MT-Bench is part of their comprehensive evaluation framework for instruction-tuned LLMs.