2 · Cost function & gradient descent
The universal optimizer — take the slope of the loss, step opposite to it, repeat until convergence.
One idea, one try. At your own pace.
LEARN BY DOINGPut your intuition to work3 steps
2 · Cost function & gradient descent
Use the lesson’s examples and explanation to check your idea.
Personal reflection. Nothing is sent to an AI service.Gradient descent is how almost every modern ML model learns — take the slope of the loss, step opposite to it, repeat.
Without this:
Without gradient descent you can't optimise neural networks (no closed form exists) — and the OLS closed form breaks at scale anyway.
In the previous lesson we found the best line by deriving the OLS closed form — setting the gradient to zero and solving analytically. That works beautifully for linear regression with moderate data, but it has hard limits:
- The normal equation requires inverting an (n × n) matrix — O(n³) time. At 10 million rows, that's infeasible.
- Neural networks have millions of parameters with no closed-form solution — every weight must be found iteratively.
Gradient descent is the answer. The key insight is geometric: the gradient ∇L at a point in parameter space tells you which direction makes the loss increase fastest. Step in the opposite direction, and you reduce the loss. Repeat, and you spiral down the loss surface toward the minimum.
For MSE loss with parameters (w, b), the gradients are:
∂L/∂w = −(2/n) · Σ(yᵢ − ŷᵢ) · xᵢ
∂L/∂b = −(2/n) · Σ(yᵢ − ŷᵢ)
The update rule at each iteration t is:
wₜ₊₁ = wₜ − η · ∂L/∂w
bₜ₊₁ = bₜ − η · ∂L/∂b
where η (eta) is the learning rate — a hyperparameter that controls step size. Getting η right is the most important practical skill in training ML models.
Python (in browser)
Watch w and b converge toward 2.5 and 1.0 over 200 iterations. The MSE column shows the loss falling — that's learning.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Semi-log y-axis reveals the rapid early drop and asymptotic approach. Three learning rates, three stories: too slow, just right, diverges.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
The characteristic loss curve: a steep drop in the first few dozen iterations, then a long plateau. The plateau means the model is near its minimum — further iterations return diminishing improvements.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Mini-batch GD is the industry default — batch size is tuned, not chosen arbitrarily. Start with 32 or 64.
3-D lab: roll gradient descent down the cost surface
The mean squared error over the weights is a bowl. Launch a marble, change the learning rate, and watch it crawl, converge or overshoot.
If gradient descent's loss oscillates between high and low values without converging, the most likely cause is:
Gradient Descent — the MML chapter covering the update rule, convergence conditions, and the geometric intuition of the loss surface.
Write the update you just read — w ← w − η·∇L — on a 2-D bowl, and find the learning rate where it starts to zig-zag.
- The gradient ∂L/∂w tells you which direction increases the loss — step opposite to it: w ← w − η·∂L/∂w.
- Learning rate η is the most critical hyperparameter: too small → slow convergence; too large → diverges or oscillates.
- Batch GD uses all data per step (stable, slow). SGD uses one sample (fast, noisy). Mini-batch (32–256 samples) is the industry default.
- Adam, RMSprop, AdaGrad are adaptive variants of GD — the core step is identical; they add per-parameter learning rate scaling.
Every neural network is trained by gradient descent (with Adam etc. as variants). XGBoost performs gradient descent in function space. SGD with momentum is the universal default for deep learning.
If you remove it: You'd be stuck with closed-form-only models — no neural nets, no online learning, no models with millions of parameters.