Entropy Calculator (Shannon)
How much information a probability distribution carries — Shannon entropy in bits, with the redundancy and the perplexity.
H = −Σ pᵢ log₂ pᵢ.
How the entropy calculator (shannon) works
H = −Σ pᵢ log₂ pᵢ. Each outcome contributes its probability times its surprize, and the sum is the average number of bits needed to encode one draw.
Entropy is maximized when everything is equally likely — log₂ n bits — and falls to zero when one outcome is certain. It is the theoretical floor for lossless compression, which is why it also measures redundancy.
Formula: H = −Σ pᵢ log₂ pᵢ
Worked examples
| Inputs | Entropy | Note |
|---|---|---|
| Four unequal outcomes | 1.742738 | 1.74 bits of a possible 2 |
| A fair coin | 1 | exactly 1 bit |
| A certainty | 0 | 0 bits |
FAQFrequently asked questions
What is Shannon entropy?
The average information content of a draw from a distribution, measured in bits — equivalently, the minimum bits per symbol any lossless code can achieve.
Why is a fair coin exactly one bit?
Because two equally likely outcomes need exactly one binary digit to distinguish them. That is the definition of the bit.
What does zero entropy mean?
No uncertainty — the outcome is already known, so observing it tells you nothing.
What is perplexity?
2^H — the number of equally likely options that would carry the same uncertainty. Language models are scored on it.
How does this relate to compression?
Entropy is the hard floor. English text has an entropy of roughly 1 to 1.5 bits per character against the 8 bits of ASCII, which is why text compresses so well.
Where these figures come from
- NIST Digital Library of Mathematical Functions — reference definitions for elementary and special functions
- Wolfram MathWorld — definitions and formulas for every topic on this page
- NIST/SEMATECH e-Handbook of Statistical Methods — the statistical formulas (mean, variance, z, confidence intervals, sample size)
- Common Core State Standards — Mathematics — the terms and methods taught in US schools
Last checked: September 2026. Formulas are fixed by mathematics and do not change with tax years or regulations.