2 · PCA: intuition and math
Find the orthogonal axes of maximum variance — and use them to compress features without losing the signal.
One idea, one try. At your own pace.
LEARN BY DOINGPut your intuition to work3 steps
2 · PCA: intuition and math
Use the lesson’s examples and explanation to check your idea.
Personal reflection. Nothing is sent to an AI service.PCA finds the orthogonal directions of maximum variance in the data — the first few capture most of the signal, the rest are dropped.
Without this:
Without PCA you'd lack the universal first-pass dimensionality reduction algorithm and the conceptual basis for SVD-based methods.
Principal Component Analysis (PCA) is the workhorse of linear dimensionality reduction. Given a matrix X (n samples × d features), PCA finds k orthogonal directions — the principal components — along which the data spreads the most. Project X onto those k directions and you have a compressed representation that preserves as much variance as possible.
The mathematical recipe:
- Center the data: subtract the column mean. Scale if features are in different units (scale-sensitive!).
- Covariance matrix: Σ = (1/n) · XᵀX — a d×d matrix where Σᵢⱼ captures how features i and j co-vary.
- Eigendecompose: Σ = VΛVᵀ — eigenvalues λᵢ are the variances along each PC direction; eigenvectors vᵢ (columns of V) are the PC directions.
- Sort: largest λ first — the first PC captures the most variance.
- Project: Z = X · V_k (the top k eigenvectors), yielding an n×k matrix.
Explained variance ratio: λᵢ / Σλ is the fraction of total variance captured by the i-th PC. Plot the cumulative version to pick k — a common heuristic is "keep enough PCs to explain 90% of variance".
Why SVD? sklearn's PCA uses SVD of X (not eigendecomp of Σ) for numerical stability. SVD never needs to explicitly form Σ, which can be ill-conditioned. We'll unpack the equivalence in the next lesson.
Python (in browser)
The two principal components of correlated 2D data — PC1 aligns with the main diagonal (max variance), PC2 is perpendicular.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Scree plot — the 'elbow' shows how quickly variance is captured. 90% of variance in the breast cancer dataset is captured by just 7-10 PCs out of 30.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Accuracy on 2 PCA components is surprisingly close to using all 30 features — PCA preserves the discriminative signal.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Python (in browser)
Breast cancer data projected to first 2 PCs — the two classes already separate without using any label information.
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
The SVD ↔ PCA equivalence — understanding this lets you read sklearn's components_ and explained_variance_ without confusion.
Singular Value Decomposition — the numerical engine inside sklearn's PCA. Understanding SVD makes the components_ attribute and the reconstruction formula click.
3-D lab: find the axis of maximum variance
Turn a beam through the cloud until its shadow is as long as possible — that direction is the first principal component, and its variance is λ₁.
Why must you scale features before applying PCA?
- PCA finds orthogonal directions (principal components) of maximum variance by eigendecomposing the covariance matrix Σ = XᵀX/n.
- Always StandardScale before PCA — a feature in a large numeric range will dominate the covariance structure and distort all components.
- The explained variance ratio λᵢ/Σλ tells you what fraction of total variance each PC captures — use a cumulative scree plot to pick k.
- sklearn implements PCA via SVD of X (not eigendecomp of Σ) — more numerically stable, especially for wide matrices.
- PCA is class-unaware — if discriminative directions are low-variance, prefer LDA when labels are available.
Image compression (eigenfaces), gene expression analysis, NLP word embedding visualisation via PCA-to-2D, exploratory data analysis on any high-D dataset.
If you remove it: You'd lack the canonical compressor and the entry point to all SVD-based techniques — from eigenfaces to latent semantic analysis to matrix factorisation in recommendation systems.