Leaderboard

LLM Leaderboards and Rankings

Major Leaderboards

Chatbot Arena (LMSYS)
Human preference-based ranking using Elo scores:
- Users chat with two anonymous models
- Choose which response is better
- Elo rating updated based on votes

Leaderboard (example scores):
1. GPT-4o: 1290
2. Claude 3.5 Sonnet: 1271
3. Gemini 1.5 Pro: 1260
4. Llama 3.1 405B: 1250
...

Open LLM Leaderboard (HuggingFace)
Automated benchmarks for open models:
- MMLU, ARC, HellaSwag, TruthfulQA, Winogrande, GSM8K

HELM (Stanford)
Holistic evaluation with many metrics:
- Accuracy, calibration, robustness, fairness, efficiency

Elo Rating System

def update_elo(winner_elo, loser_elo, k=32):
    expected_winner = 1 / (1 + 10 ** ((loser_elo - winner_elo) / 400))
    expected_loser = 1 - expected_winner

    new_winner_elo = winner_elo + k * (1 - expected_winner)
    new_loser_elo = loser_elo + k * (0 - expected_loser)

    return new_winner_elo, new_loser_elo

Interpreting Leaderboards
| Elo Difference | Win Probability |
|----------------|-----------------|
| 0 | 50% |
| 100 | 64% |
| 200 | 76% |
| 400 | 91% |

Leaderboard Limitations
| Issue | Mitigation |
|-------|------------|
| Selection bias | Random sampling |
| Prompt diversity | Topic stratification |
| Position bias | Randomize A/B order |
| Length bias | Evaluate conciseness |
| Time | Ratings change over time |

Domain-Specific Leaderboards
| Domain | Leaderboard |
|--------|-------------|
| Coding | SWE-bench, LiveCodeBench |
| Math | MATH leaderboard |
| Safety | HarmBench |
| RAG | MTEB embeddings |
| Agents | AgentBench |

Best Practices
- Dont rely on single leaderboard
- Consider use case fit
- Check benchmark methodology
- Evaluate on your own data
- Monitor for gaming/overfitting