CHAPTER 34 / 40
The Mathematics of Model Adaptation
4 lessons · Explore, experiment and understand.
DOJO PETYour little coding companion
DOJO / YOUR GUIDE
Let’s figure it out.
Ask me about this course or where to start.
Chapter lessons
4 lessons
- Step 139~20m1Code · 16⚡Challenge
Low-Rank Adaptation (LoRA): Fine-Tuning with Tiny Matrices
Fine-tune a billion-parameter model by training two thin matrices per layer: the rank factorization ΔW = BA turns a 16M-number update into 6…
Open lesson - Step 140~20m1Code · 15⚡Challenge
Mixture-of-Experts: The Mathematics of Sparse Routing
A softmax router turns one big network into a team of specialists: top-k routing decouples parameter count from per-token compute, and a one…
Open lesson - Step 141~15m1Code · 15⚡Challenge
Knowledge Distillation: Temperature, Soft Targets, and KL
A big teacher's near-misses encode how classes relate. Raise the softmax temperature to expose that dark knowledge, match it with a KL diver…
Open lesson - Step 142~20m1Code · 16⚡Challenge
Neural Scaling Laws: Power Laws and Compute-Optimal Training
Loss falls as a power law in model size, data, and compute — a straight line on log-log axes. One constrained optimization over that law dec…
Open lesson