Bradley-Terry Model

The Bradley-Terry model is a probabilistic framework for modeling pairwise comparison outcomes — given two options, it predicts the probability of each one being preferred. It is the mathematical foundation underlying reward model training in RLHF.

The Model

Each option i has a latent strength parameter $\beta_i$ . The probability that option i is preferred over option j is:

P(i \succ j) = \frac{e^{\beta_i}}{e^{\beta_i} + e^{\beta_j}} = \sigma(\beta_i - \beta_j)

Where $\sigma$ is the sigmoid function. The preference probability depends only on the difference in strengths, not their absolute values.

Connection to RLHF

In RLHF reward modeling, the reward model assigns scores $$r(x, y)$$ to each response y given prompt x.
The Bradley-Terry model assumes the probability of preferring response $$y_w$$ over $$y_l$$ is:

P(y_w \succ y_l | x) = \sigma(r(x, y_w) - r(x, y_l))

The reward model is trained by maximizing the log-likelihood of the observed human preferences under this model.

Key Properties

Transitivity: The model assumes consistent preferences — if A is strongly preferred over B and B over C, then A will be strongly preferred over C.
Scale Invariance: Adding a constant to all strengths doesn't change preferences — only differences matter.
Maximum Likelihood: Parameters are estimated by maximizing the likelihood of observed comparison outcomes.

Extensions

Thurstone Model: Alternative where strengths are sampled from Normal distributions rather than Gumbel distributions.
Plackett-Luce Model: Extends Bradley-Terry to rankings of more than two items.
Ties: Extensions exist for handling "equally good" outcomes.

Practical Usage

Beyond RLHF, the Bradley-Terry model is used in chess/Elo ratings, sports ranking, A/B testing, and any domain involving pairwise comparisons. The LMSYS Chatbot Arena leaderboard uses it to rank LLMs based on human votes.

Bradley-Terry Model

Take bradley-terry model further