KL Divergence Calculator
The KL divergence of one discrete distribution from another — in nats and bits, both directions — and the symmetric Jensen–Shannon divergence, from two lists of counts or probabilities.
KL divergence D(P‖Q) = Σ p log(p ÷ q) measures the extra information needed to describe P when you assumed Q.
How the kl divergence calculator works
KL divergence D(P‖Q) = Σ p log(p ÷ q) measures the extra information needed to describe P when you assumed Q. It is zero only when they match, grows without bound where Q gives near-zero probability to something P does, and is not symmetric. Jensen–Shannon takes the KL of each against their average and is symmetric and bounded (1 bit at most). Both are used to detect data drift between a training distribution and live traffic.
Formula: D(P‖Q) = Σ pᵢ ln(pᵢ ÷ qᵢ); JSD = ½ D(P‖M) + ½ D(Q‖M), M = ½(P + Q)
Worked examples
| Inputs | KL divergence D(P‖Q) (nats) | Note |
|---|---|---|
| 50/30/15/5 against 40/35/20/5 | 0.02217 | 0.022 nats |
| Identical distributions | 0 | 0 |
| A category missing from Q, no smoothing | — | infinite — reported with the JSD instead |
FAQFrequently asked questions
Which way round should P and Q go?
P is the true or current distribution, Q the model or reference you compare it with: D(P‖Q) is the cost of using Q to encode data from P. For drift detection, P is live traffic and Q the training distribution.
Why is it infinite?
A category that appears in P but has zero probability in Q makes the log blow up. Real distributions from samples often have empty bins; the smoothing field adds a tiny mass to every bin so the divergence stays finite.
What threshold signals drift?
There is no universal number; teams often alert on KL above 0.1–0.2 nats or JSD above 0.1 bits for a feature, after checking what normal week-to-week variation looks like. The population stability index calculator is the credit-scoring convention for the same job.
Bits or nats?
The same quantity with a different log base: nats use the natural log, bits log base 2, and 1 nat = 1.443 bits. Machine-learning papers usually report nats; information theory bits.
Where these figures come from
- Vaswani et al. (2017) — Attention Is All You Need — the transformer architecture the memory arithmetic follows
- Kaplan et al. (2020) — Scaling Laws for Neural Language Models — the compute relationship used for training estimates
- Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models — the tokens-per-parameter guidance ("Chinchilla")
- IEEE 754 — Standard for Floating-Point Arithmetic — the numeric formats behind bytes per parameter
- NIST AI Risk Management Framework — the US federal AI framework
Last checked: September 2026. The relationships here are architectural, not vendor-specific: bytes per parameter follow the numeric format, KV-cache size follows the transformer definition, and token-per-word ratios come from published tokeniser behaviour.