Bradley–Terry Calculator
The Bradley–Terry strength ratio and win probability between two models or responses from how often each was preferred — with an Elo-style rating gap and a confidence interval on the win rate.
In the Bradley–Terry model each competitor has a strength, and the probability A beats B is s_A ÷ (s_A + s_B).
How the bradley–terry calculator works
In the Bradley–Terry model each competitor has a strength, and the probability A beats B is s_A ÷ (s_A + s_B). From head-to-head counts the maximum-likelihood ratio is simply wins_A ÷ wins_B, so a 60–40 preference is a strength ratio of 1.5. Chatbot arenas convert it to an Elo-like scale: 400 × log₁₀ of the ratio — 60% is about 70 points. The Wilson interval shows how sure the counts make you.
Formula: P(A beats B) = s_A ÷ (s_A + s_B); s_A ÷ s_B = wins_A ÷ wins_B; rating gap = 400 log₁₀(s_A ÷ s_B)
Worked examples
| Inputs | Probability A beats B | Note |
|---|---|---|
| 120 to 80 with 20 ties | 59.1 % | 59.1%, ratio 1.44, +64 points |
| A narrow lead from few comparisons | 55 % | 55% but not significant |
| A dominant model | 90 % | 90% — +382 points |
FAQFrequently asked questions
What is the Bradley–Terry model used for?
Any ranking from pairwize comparisons: chess and sports ratings (Elo is a special case), chatbot leaderboards built from human preferences, and reward models in RLHF, which are trained to reproduce these preference probabilities.
How many comparisons do I need?
The interval row tells you: 200 comparisons at 60–40 give roughly 53–67%, enough to be confident A is better; 20 comparisons at the same rate are not. Aim for a few hundred per pair for a stable rating.
How do ties count?
Half a win to each side, the usual convention. Some arenas drop ties instead; the effect on the ratio is small unless ties dominate.
How does this extend to many models?
Fit strengths for all models jointly by maximum likelihood from every pair’s counts (the arena leaderboards do this with a logistic regression). The two-model case here is the building block.
Where these figures come from
- Vaswani et al. (2017) — Attention Is All You Need — the transformer architecture the memory arithmetic follows
- Kaplan et al. (2020) — Scaling Laws for Neural Language Models — the compute relationship used for training estimates
- Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models — the tokens-per-parameter guidance ("Chinchilla")
- IEEE 754 — Standard for Floating-Point Arithmetic — the numeric formats behind bytes per parameter
- NIST AI Risk Management Framework — the US federal AI framework
Last checked: September 2026. The relationships here are architectural, not vendor-specific: bytes per parameter follow the numeric format, KV-cache size follows the transformer definition, and token-per-word ratios come from published tokeniser behaviour.