19 · LoRA in one cell
Full fine-tuning updates every weight — billions of them. LoRA freezes the original weights and learns a tiny low-rank update B·A instead, cutting trainable parameters by orders of magnitude with almost no loss.
One idea, one try. At your own pace.
LEARN BY DOINGPut your intuition to work3 steps
19 · LoRA in one cell
Use the lesson’s examples and explanation to check your idea.
Personal reflection. Nothing is sent to an AI service.LoRA (Low-Rank Adaptation) freezes the pretrained weight W and learns a low-rank update ΔW = B·A, where B is d×r and A is r×d with rank r tiny (e.g. 8). The effective weight is W + (α/r)·B·A. You train only 2·d·r numbers instead of d·d — a reduction of orders of magnitude.
Without this:
Without parameter-efficient fine-tuning, adapting a billion-parameter model means storing and updating a full copy of every weight per task — infeasible on consumer hardware and wasteful when one adapter per task would do.
Full fine-tuning re-trains every weight in the model. For a layer whose weight matrix W is d × d, that's d² trainable parameters — and a real model has hundreds of such matrices, totaling billions. You also have to store a whole new copy of the model per task. That's the wall LoRA was built to climb.
The key observation behind LoRA (Low-Rank Adaptation): the update you need to specialize a pretrained model usually has very low intrinsic rank — it can be well-approximated by the product of two skinny matrices. So instead of learning a full d × d update ΔW, LoRA:
- Freezes the original weight
W(no gradients, never changes). - Learns two small matrices:
Bof shaped × randAof shaper × d, where the rankris tiny (often 4, 8, 16). - Uses the effective weight
W + (α/r)·B·Aat inference, whereαis a scaling constant.
The update ΔW = B·A is a full d × d matrix, but it's built from only d·r + r·d = 2·d·r trainable numbers instead of d·d. When r ≪ d that's a massive saving. At training time you back-prop only into A and B; W sits frozen. At deploy time you can either keep the adapter separate (swap adapters per task) or fold (α/r)·B·A into W so inference costs nothing extra.
This is exactly the low-rank / matrix-factorization idea from the math track — a big matrix approximated by the product of two thin ones — applied to weight updates.
Here is LoRA in one NumPy cell. We take a 512 × 512 weight W, build a low-rank update from B (512×r) and A (r×512), form the effective weight W + (α/r)·B·A, and — the headline — count the trainable parameters for full fine-tuning versus LoRA at several ranks. Watch the percentage of parameters you actually train collapse toward a fraction of a percent.
Python (in browser)
LoRA in NumPy: a full d×d update ΔW=(α/r)·B·A is built from only 2·d·r trainable numbers. At rank 8 on a 512×512 layer you train ~3% of the parameters; the matrix interface is unchanged.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The savings compound at real scale. A 7-billion-parameter model has thousands of weight matrices; full fine-tuning means a 7B-parameter optimizer state and a 7B-parameter checkpoint per task. With LoRA at rank 8 you train and ship a few million parameters — small enough to keep dozens of task adapters on disk and hot-swap them onto one frozen base model. That is what peft does in practice; read the real config below.
PEFT's LoraConfig is exactly our NumPy demo at scale: rank r, alpha scaling, and a list of which weight matrices get a B·A adapter — training well under 0.1% of the parameters.
In LoRA, for a d×d weight matrix with rank r, how many parameters are actually trained, and what stays frozen?
- LoRA freezes the pretrained weight W and learns a low-rank update ΔW = B·A (B is d×r, A is r×d) with tiny rank r.
- You train 2·d·r parameters instead of d·d — orders of magnitude fewer; the effective weight is W + (α/r)·B·A.
- B inits to zero so the adapter starts as a no-op (stable training), and one frozen base model can hot-swap many small task adapters.
LoRA and its quantized variant QLoRA are how almost every open-weight model is fine-tuned today — PEFT's LoraConfig trains under 1% of a 7B model so it fits on a single consumer GPU.
If you remove it: Without parameter-efficient adapters, every task needs a full copy of the model's billions of weights trained and stored — infeasible on commodity hardware and wasteful at scale.