Preference Win Rate Calculator
From a set of pairwise preference judgements between two models or responses — wins, losses and ties — the win rate, its confidence interval, the Elo-style rating gap it implies, and how many more comparisons you need to tighten it.
Preference data is the raw material of reward models and of every leaderboard that ranks by head-to-head votes.
How the preference win rate calculator works
Preference data is the raw material of reward models and of every leaderboard that ranks by head-to-head votes. A win rate of 60% is an Elo gap of about 70 points; 75% is about 190. But a win rate from thirty comparisons is a guess — the Wilson interval says how wide — and the comparisons needed for a given margin grow with the square of the precision you want.
Ties count as half a win, which is the convention in rating systems and in most preference datasets.
Formula: p = (wins + ½ ties) / n; Elo gap = 400 log₁₀(p / (1 − p)); n for margin m ≈ z² p(1−p) / m²
Worked examples
| Inputs | Win rate for A | Note |
|---|---|---|
| 62 to 38 with 10 ties | 60.91% | 61% — clearly better |
| A handful of judgements | 70% | 70%, but the interval touches 50 |
| A large evaluation | 52.5% | 52.5%, and significant |
FAQFrequently asked questions
What is a preference win rate?
The share of pairwise judgements in which one model's output was preferred over another's, with ties as half. It is the basic statistic of RLHF data and of chatbot arenas.
How does win rate become an Elo gap?
Through the logistic rating model: a rating gap of 400 points means a 10:1 win expectation. Sixty per cent is about 70 points; seventy-five per cent about 190.
Why a Wilson interval?
Because the usual normal approximation misbehaves at small counts and near 0 or 100%. Wilson stays inside the possible range and is right at small n.
How many comparisons do I need?
For a ±3-point margin at 95%, about a thousand. For ±1 point, nearly ten thousand. The cost grows with the square of the precision.
Do annotator disagreements matter?
Very much — a win rate has an irreducible ceiling set by how often humans agree with each other. Measure inter-annotator agreement alongside this.
Where these figures come from
- Vaswani et al. (2017) — Attention Is All You Need — the transformer architecture the memory arithmetic follows
- Kaplan et al. (2020) — Scaling Laws for Neural Language Models — the compute relationship used for training estimates
- Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models — the tokens-per-parameter guidance ("Chinchilla")
- IEEE 754 — Standard for Floating-Point Arithmetic — the numeric formats behind bytes per parameter
- National AI Centre — Australia's national AI body
Last checked: September 2026. The relationships here are architectural, not vendor-specific: bytes per parameter follow the numeric format, KV-cache size follows the transformer definition, and token-per-word ratios come from published tokeniser behaviour.