CHAPTER 14 / 40

Deep Learning Building Blocks

8 lessons · Explore, experiment and understand.

DOJO PETYour little coding companion

DOJO / YOUR GUIDE

Let’s figure it out.

Ask me about this course or where to start.

Chapter lessons

8 lessons

14Deep Learning Building Blocks

0/88 lessons

Step 58
~10m3D · 1Code · 12
★Most cinematic
Convolutional Neural Networks Math
A convolution slides a learnable filter across an input feature map and outputs, at each spatial position, the dot product between the filte…
Open lesson
Step 59
~5mCode · 11
🎯Quick win
Recurrent Neural Networks: RNNs, LSTMs, and GRUs
An RNN processes a sequence by carrying a hidden state . Each step folds the new input into a running summary of everything seen so far. Out…
Open lesson
Step 60
~5mCode · 11
🎯Quick win
Batch Normalization and Layer Normalization
Batch normalization (Ioffe & Szegedy 2015) normalizes each feature across a mini-batch: , then scales and shifts with learnable . At test ti…
Open lesson
Step 61
~5mCode · 11
🎯Quick win
Dropout: The Math of Stochastic Regularization
Dropout (Srivastava et al. 2014) randomly zeros each activation with probability during training, then scales the survivors by so the expect…
Open lesson
Step 62
~5m3D · 1Code · 11
★Most cinematic
Activation Functions: ReLU, GELU, Swish, and Friends
Activation functions introduce the non-linearity that makes neural networks more than just compositions of affine maps. Without them, any st…
Open lesson
Step 63
~5mCode · 11
🎯Quick win
Positional Encodings: How Transformers See Order
Attention is permutation-invariant — shuffling the input tokens shuffles the output tokens but doesn't change which tokens attend to which.…
Open lesson
Step 64
~5m3D · 1Code · 11
★Most cinematic
Modern Optimizers: SGD, Momentum, Adam, AdamW
Vanilla SGD: . Simple, well-understood, sometimes the best choice for CNNs with the right learning-rate schedule.
Open lesson
Step 65
~5mCode · 11
🎯Quick win
Learning Rate Schedules: Warmup, Cosine, Cyclic
The learning rate is the single most important hyperparameter. A fixed is rarely optimal — most of training benefits from a schedule that de…
Open lesson

Deep Learning Building Blocks

Let’s figure it out.

Convolutional Neural Networks Math

Recurrent Neural Networks: RNNs, LSTMs, and GRUs

Batch Normalization and Layer Normalization

Dropout: The Math of Stochastic Regularization

Activation Functions: ReLU, GELU, Swish, and Friends

Positional Encodings: How Transformers See Order

Modern Optimizers: SGD, Momentum, Adam, AdamW

Learning Rate Schedules: Warmup, Cosine, Cyclic

Deep Learning Building Blocks

Let’s figure it out.

Convolutional Neural Networks Math

Recurrent Neural Networks: RNNs, LSTMs, and GRUs

Batch Normalization and Layer Normalization

Dropout: The Math of Stochastic Regularization

Activation Functions: ReLU, GELU, Swish, and Friends

Positional Encodings: How Transformers See Order

Modern Optimizers: SGD, Momentum, Adam, AdamW

Learning Rate Schedules: Warmup, Cosine, Cyclic