11 · Bernoulli: single yes/no trial
The simplest distribution in the book — a single trial with two outcomes and one parameter. Every binary classifier is built on this.
One idea, one try. At your own pace.
LEARN BY DOINGPut your intuition to work3 steps
11 · Bernoulli: single yes/no trial
Use the lesson’s examples and explanation to check your idea.
Personal reflection. Nothing is sent to an AI service.The Bernoulli is the simplest distribution — a single trial with two outcomes, parameter p.
Without this:
Without it, you can't model a coin flip, a binary classification label, or a click/no-click event.
A Bernoulli random variable is the simplest possible distribution: a single trial that has exactly two outcomes — success (1) with probability p, or failure (0) with probability 1 − p.
Probability Mass Function (PMF):
- P(X = 1) = p
- P(X = 0) = 1 − p
- Written compactly: P(X = k) = p^k · (1 − p)^(1−k) for k ∈ {0, 1}
Mean and variance:
- E[X] = p (the average of 0s and 1s from many trials equals the success probability)
- Var(X) = p(1 − p)
Maximum variance at p = 0.5: The variance function p(1 − p) is a downward-opening parabola with maximum at p = 0.5, where Var(X) = 0.25. This is the point of maximum uncertainty — when a coin is perfectly fair, you know the least about any single toss. At p = 0 or p = 1 the outcome is certain, so variance collapses to 0.
Connection to binary classification: Every supervised binary classification example carries a label y ∈ {0, 1}. That label is a draw from a Bernoulli(p) distribution, where p = P(positive class | features). The model's job is to estimate p given the features. This is why logistic regression outputs a value in (0, 1) — it is estimating the Bernoulli parameter.
Log-likelihood and binary cross-entropy: The likelihood of observing label y given model probability p is: L(p ; y) = p^y · (1 − p)^(1−y)
Taking the log: log L = y·log(p) + (1−y)·log(1−p)
Negating to turn maximization into minimization: −log L = −y·log(p) − (1−y)·log(1−p)
This is exactly the binary cross-entropy loss. The loss function in logistic regression IS the Bernoulli negative log-likelihood.
Python (in browser)
scipy.stats.bernoulli(p) gives you a distribution object. .mean() returns p, .var() returns p(1−p), and .rvs(n) draws n independent samples. With only 10 samples the empirical mean is noisy — that's expected.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The variance curve Var(X) = p(1−p) peaks at p=0.5 (Var=0.25) and falls to zero at both extremes. This parabola is the geometric picture of uncertainty: a fair coin is the hardest to predict; a completely biased one is trivial. Open /tmp/bern_var.png to view.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
With 10 000 draws the empirical mean converges to p=0.3 (≈ 0.21 variance). rng.binomial(1, p, size=n) is the canonical way to generate Bernoulli samples with NumPy — it is simply the Binomial with n=1.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Binary cross-entropy is not arbitrary — it falls directly out of the Bernoulli negative log-likelihood. When you call model.compile(loss='binary_crossentropy') you are asking the network to maximize the Bernoulli likelihood over the training labels.
What is Var(X) for a Bernoulli with p = 0.1?
Discrete and Continuous Probabilities — the MML chapter that formalises the PMF, defines expectation and variance from first principles, and shows how the Bernoulli is the building block for every more complex discrete distribution.
- **PMF**: P(X=1) = p, P(X=0) = 1−p. **Mean** = p, **Var** = p(1−p). Maximum variance 0.25 at p=0.5.
- Binary classification labels are Bernoulli draws. Estimating p given features is the universal binary classification task.
- **Binary cross-entropy = Bernoulli negative log-likelihood.** The loss is not arbitrary; it is derived directly from the probabilistic model.
Logistic regression and binary classifiers output a Bernoulli probability; the loss IS the Bernoulli log-likelihood (binary cross-entropy).
If you remove it: You can't write down the loss function for any binary classification model.