Shannon Information Theory Lab

Explore the mathematical foundations of information theory through interactive visualizations

1 Self-Information

$$ I(x) = -\log_2 P(x) $$

The surprise, or information content, of a single event.

Rare (0.01) Equally likely (0.5) Common (0.99)

Self-information:

{{ selfInformation.toFixed(4) }} bits

Surprise level:

{{ getSurpriseLevel(selfInformation) }}
I(x)
{{ selfInformation.toFixed(2) }}

2 Information Entropy

$$ H(X) = -\sum_{i=1}^n P(x_i) \log_2 P(x_i) $$

The average uncertainty of a random variable.

$x_{{index+1}}$ {{ (prob/100).toFixed(3) }}

Entropy $H(X)$:

{{ entropy.toFixed(4) }} bits

Maximum possible entropy:

{{ maxEntropy.toFixed(4) }} bits

Uncertainty level: {{ (entropy/maxEntropy*100).toFixed(1) }}%

3 Binary Entropy

$$ H(p) = -p \log_2 p - (1-p) \log_2(1-p) $$

A special case of the Bernoulli distribution.

$P(A)$

{{ pBinary.toFixed(3) }}

$P(\bar{A})$

{{ (1-pBinary).toFixed(3) }}

Binary entropy:

{{ binaryEntropy.toFixed(4) }} bits

{{ getBinaryEntropyInterpretation(pBinary) }}

4 Mutual Information

$$ I(X;Y) = \sum_{x\in X}\sum_{y\in Y} P(x,y) \log_2\frac{P(x,y)}{P(x)P(y)} $$

The amount of information shared by two random variables.

Negative (-1) Independent (0) Positive (1)

$H(X)$

{{ marginalEntropyX.toFixed(3) }}

$I(X;Y)$

{{ mutualInformation.toFixed(3) }}

$H(Y)$

{{ marginalEntropyY.toFixed(3) }}

Shared information
{{ mutualInformation.toFixed(2) }} bits

5 Shannon-Hartley Theorem

$$ C = B \log_2\left(1 + \frac{S}{N}\right) $$

The upper capacity limit of an AWGN channel.

Narrowband (1 MHz) Wideband (50 MHz) Ultra-wideband (100 MHz)
Poor (-10 dB) Medium (15 dB) Strong (40 dB)

Linear SNR

{{ snrLinear.toFixed(2) }}

Spectral efficiency

{{ spectralEfficiency.toFixed(2) }} bps/Hz

Channel capacity limit:

{{ channelCapacity.toFixed(2) }} Mbps

{{ Math.floor(channelCapacity * 1e6 / 8).toLocaleString() }} bytes per second

6 Text Entropy Analysis

$$ H_{\text{text}} = -\sum_{c \in \text{chars}} p(c) \log_2 p(c) $$

Estimate the information density of natural-language text.

Characters

{{ textInput.length }}

Unique characters

{{ uniqueChars }}

Maximum entropy

{{ maxTextEntropy.toFixed(3) }}

Observed entropy:

{{ textEntropy.toFixed(4) }} bits/char

Compression efficiency:

{{ compressionEfficiency.toFixed(1) }}%

Built around Claude Shannon's 1948 paper 'A Mathematical Theory of Communication' • formulas rendered with MathJax

Interactive teaching tool for information theory • refresh formulas with MathJax.typeset()