29 · Attention: the decoder looks at every encoder state
Attention lets the decoder query all encoder hidden states at every output step — weighted by relevance — fixing the context-vector bottleneck.
One idea, one try. At your own pace.
LEARN BY DOINGPut your intuition to work3 steps
29 · Attention: the decoder looks at every encoder state
Use the lesson’s examples and explanation to check your idea.
Personal reflection. Nothing is sent to an AI service.Attention lets the decoder look at ALL encoder hidden states at every output step — weighted by relevance to what's being generated.
Without this:
Plain seq2seq compresses an entire input into a single context vector; attention fixes this by letting the decoder query the full input history.
In the last lesson we built seq2seq and saw its central weakness: the entire meaning of a sentence — however long — must fit in a single context vector of fixed size. For short sentences this works surprisingly well. For long sentences (20+ tokens), performance degrades sharply.
Think about what a human translator does. When translating "The cat sat on the mat" to French and generating the word "chat" (cat), you are not trying to remember the whole English sentence — you're looking specifically at "cat." When generating "sur" (on), you look at "on." Attention gives the neural network the same capability: at each output step, look back at ALL the encoder's hidden states and decide which ones are most relevant right now.
Formally, at decoder step t with current hidden state s_t:
- Alignment scores: for each encoder position
i, compute a scalar scoree_{t,i} = score(s_t, h_i)measuring how much positionimatters for this output step. - Attention weights: softmax over scores:
α_{t,i} = exp(e_{t,i}) / Σ_j exp(e_{t,j}). These sum to 1 and form a soft distribution over input positions. - Context vector: weighted sum of encoder hidden states:
c_t = Σ_i α_{t,i} · h_i. - Decoder input: concatenate
c_twith the decoder's current state and feed into the next LSTM step.
The key insight: c_t is different at EVERY output step. The decoder gets a fresh, task-specific summary of the input at each timestep — not one fixed vector for the whole output sequence.
Attention recipe: alignment scores → softmax → weighted sum of encoder states → context vector
Python (in browser)
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Hover a row (target token) to highlight its attention distribution. Switch heads above.
Python (in browser)
Attention heatmap for a 5-token source, 4-token target pair — brighter cells indicate higher attention weight
Python runs entirely in your browser via Pyodide (~6 MB on first Run, cached after).
Bahdanau vs Luong attention: scoring functions, where context is injected, parameter cost
Attention as differentiable soft lookup: query, keys, values — the abstraction that leads to transformers
What does the attention weight α_{t,i} represent?
- Attention computes a dynamic context vector c_t at each decoder step as a softmax-weighted sum of ALL encoder hidden states — not one fixed vector for the whole output.
- The three score functions are dot-product, bilinear (Luong), and MLP (Bahdanau). Dot-product attention generalizes directly to transformer self-attention.
- Attention is a differentiable soft lookup: query, keys, values — the same abstraction that powers BERT, GPT, and every modern transformer.
Attention is THE central idea of transformers. The same softmax(QK^T)V formula runs in BERT, GPT, T5, Stable Diffusion, AlphaFold — all of them.
If you remove it: You'd use transformers as black boxes without understanding why query-key-value is the right abstraction — a gap that makes debugging, fine-tuning, and architecture choices opaque.