Perplexity Calculator
Convert a language model’s cross-entropy loss to perplexity and back — in nats, bits per token or bits per byte — with what the number means about the model’s predictions.
Perplexity is the exponential of the average cross-entropy per token: e^loss with loss in nats, or 2^loss in bits.
How the perplexity calculator works
Perplexity is the exponential of the average cross-entropy per token: e^loss with loss in nats, or 2^loss in bits. It is the effective number of equally likely choices the model faces at each token — a perplexity of 10 means the model is as uncertain as a fair ten-sided die. Lower is better, and it is only comparable between models that share a tokenizer and test set; bits per byte removes the tokenizer.
Formula: PPL = e^(loss in nats) = 2^(bits per token); bits per byte = bits per token ÷ bytes per token
Worked examples
| Inputs | Perplexity | Note |
|---|---|---|
| Loss 2.3 nats | 9.974 | perplexity 9.97 |
| 1.2 bits per token | 2.297 | 2.30 |
| Perplexity 20 → loss | 20 | 3.0 nats, 4.32 bits |
FAQFrequently asked questions
What is a good perplexity?
It depends on the text and the tokenizer: large models score 2–4 on clean web text and under 2 on code, small models 10–30. Compare only on the same test set with the same tokenizer — or use bits per byte.
Why does the tokenizer matter?
A tokenizer that splits text into bigger pieces makes each token harder to predict but there are fewer of them; per-token perplexity rises while the total information is unchanged. Bits per byte (or per character) is the tokenizer-free measure.
Does lower perplexity mean a better chatbot?
Only loosely. Perplexity measures how well the model predicts held-out text; instruction following, helpfulness and safety are trained afterwards and measured with preference and task benchmarks.
How do I get the loss?
Run the model over a test set and average the negative log-probability of each actual next token. Training frameworks report it directly; most report nats, some bits — check which.
Where these figures come from
- Vaswani et al. (2017) — Attention Is All You Need — the transformer architecture the memory arithmetic follows
- Kaplan et al. (2020) — Scaling Laws for Neural Language Models — the compute relationship used for training estimates
- Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models — the tokens-per-parameter guidance ("Chinchilla")
- IEEE 754 — Standard for Floating-Point Arithmetic — the numeric formats behind bytes per parameter
- NIST AI Risk Management Framework — the US federal AI framework
Last checked: September 2026. The relationships here are architectural, not vendor-specific: bytes per parameter follow the numeric format, KV-cache size follows the transformer definition, and token-per-word ratios come from published tokeniser behaviour.