CHAPTER 18 / 40
Information Theory
3 lessons · Explore, experiment and understand.
DOJO PETYour little coding companion
DOJO / YOUR GUIDE
Let’s figure it out.
Ask me about this course or where to start.
Chapter lessons
3 lessons
Why this matters in ML
Entropy, cross-entropy and KL divergence are the losses inside every classifier and the regularizers inside every VAE — information theory is how ML measures 'wrongness'.
- cross-entropy loss
- KL divergence
- InfoNCE
- Step 83~10m13
Entropy and Surprise
Shannon entropy turns a probability distribution into a single number that quantifies its unpredictability — the average bits of surprise pe…
Open lesson - Step 84~10m13
Cross-Entropy and KL Divergence
Cross-entropy and KL divergence measure 'how wrong' an estimated distribution is — and they are the loss functions behind virtually every cl…
Open lesson - Step 85~10m13
Mutual Information
Mutual information measures how much knowing one variable reduces the uncertainty about another — the principled answer to 'is this feature…
Open lesson