Expected Calibration Error Calculator
Expected calibration error from binned confidence and accuracy — whether a model that says "80% sure" is right 80% of the time — with the maximum bin gap and which bins are over- or under-confident.
Group predictions by their confidence, and for each bin compare the average confidence with the share that were actually correct.
How the expected calibration error calculator works
Group predictions by their confidence, and for each bin compare the average confidence with the share that were actually correct. ECE is the gap in each bin weighted by how many predictions fell in it. A perfectly calibrated model has zero; deep networks are typically over-confident, with gaps of 5 to 15 points before calibration.
Calibration is separate from accuracy: a model can be accurate and badly calibrated, and its probabilities are then useless for any decision that depends on them.
Formula: ECE = Σ (nᵢ / N) × |acc(bᵢ) − conf(bᵢ)|; MCE = max |acc − conf|
Worked examples
| Inputs | Expected calibration error | Note |
|---|---|---|
| A typically over-confident network | 6.266% | ECE about 7 |
| Perfectly calibrated | 0% | zero |
| Under-confident | 5.357% | accuracy above confidence |
FAQFrequently asked questions
What is calibration?
Whether a model's stated confidence matches how often it is right. A calibrated model that says 70% is correct seven times in ten.
What is expected calibration error?
The average gap between confidence and accuracy across confidence bins, weighted by how many predictions land in each. Zero is perfect; modern classifiers often start around 5 to 15 points.
Why do neural networks over-predict confidence?
Cross-entropy training keeps pushing probabilities toward 1 on the training set. The result is high accuracy with confidence that outruns it.
How is it fixed?
Temperature scaling — dividing the logits by one fitted constant — removes most of it without touching accuracy. Isotonic regression and Platt scaling are the alternatives.
How many bins?
Ten to fifteen equal-width bins is standard. Equal-mass bins are more stable when most predictions crowd the top of the range, as they usually do.
Where these figures come from
- Vaswani et al. (2017) — Attention Is All You Need — the transformer architecture the memory arithmetic follows
- Kaplan et al. (2020) — Scaling Laws for Neural Language Models — the compute relationship used for training estimates
- Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models — the tokens-per-parameter guidance ("Chinchilla")
- IEEE 754 — Standard for Floating-Point Arithmetic — the numeric formats behind bytes per parameter
- NIST AI Risk Management Framework — the US federal AI framework
Last checked: September 2026. The relationships here are architectural, not vendor-specific: bytes per parameter follow the numeric format, KV-cache size follows the transformer definition, and token-per-word ratios come from published tokeniser behaviour.