Module 4: Transformers - The Modern AI Revolution
3. The Attention Mechanism: Transformers' Key Idea
Description
In the previous lesson you saw why RNNs are limited (sequential, vanishing gradients, forgetting context). Now you're going to understand the solution: the attention mechanism.
Attention is the central idea of Transformers. Without attention, there would be no ChatGPT, no BERT, and no modern AI revolution.
In this lesson you'll understand:
- What attention is (conceptually, without complex formulas).
- How it works (analogies and concrete examples).
- Why it solves RNNs' problems.
Approach: Conceptual, with analogies and visualizations. You will NOT see mathematical formulas (those are optional at the end).
What Attention is
Simple definition
Attention: A mechanism that lets the network focus on relevant parts of the input as it processes each word.
Analogy: Reading a long document.
- Without attention (an RNN): You read sequentially (word 1, word 2, ..., word 100). To remember something from word 10, you depend on your memory (limited, you forget).
- With attention (a Transformer): When you read word 100 and need context, you can go back directly to any earlier word (e.g. word 10) to look up the relevant information.
Moral: Attention lets the network "go back" and look up relevant information without depending on compressed memory.
How Attention works (conceptually)
Example: translating "The cat sat on the mat" into Spanish
Input: "The cat sat on the mat"
Task: Translate word by word into Spanish: "El gato se sentó en la alfombra"
Step 1: Translating "El" (the translation of "The")
Question: Which words of the input should I "attend" to in order to translate "The"?
Answer: Mainly "The" (word 1). But also a little to "cat" (word 2) in order to understand the context (a masculine/feminine article in Spanish depends on the noun).
Attention weights (conceptual):
The: 0.9 (high attention)
cat: 0.1 (little attention)
sat, on, the, mat: 0.0 (no attention)
Interpretation: The network "attends" mainly to "The" (90% of the attention) and a little to "cat" (10%).
Step 2: Translating "gato" (the translation of "cat")
Question: Which words of the input should I attend to in order to translate "cat"?
Answer: Mainly "cat" (word 2). But also a little to "The" (word 1) to confirm that it's the subject, and a little to "sat" (word 3) to understand that the cat is the one performing the action.
Attention weights:
cat: 0.7 (high attention)
The: 0.2 (moderate attention)
sat: 0.1 (little attention)
on, the, mat: 0.0 (no attention)
Step 3: Translating "se sentó" (the translation of "sat")
Question: Which words of the input should I attend to?
Answer: Mainly "sat" (word 3). But also to "cat" (word 2, the subject performing the action) and to "on" (word 4, the preposition that follows).
Attention weights:
sat: 0.6
cat: 0.3
on: 0.1
The, the, mat: 0.0
The general pattern
To translate each output word:
- The network attends to multiple input words (with different attention weights).
- It combines information from those words (weighted by the attention weights) in order to generate the translation.
Key point: The network can attend to any word of the input (not just the earlier ones sequentially, as RNNs do).
Visualizing Attention
The attention matrix (conceptual)
Rows: Output words (Spanish)
Columns: Input words (English)
Values: Attention weights (how much each output attends to each input)
The cat sat on the mat
El 0.9 0.1 0.0 0.0 0.0 0.0
gato 0.2 0.7 0.1 0.0 0.0 0.0
se 0.0 0.1 0.7 0.2 0.0 0.0
sentó 0.0 0.2 0.6 0.2 0.0 0.0
en 0.0 0.0 0.1 0.8 0.1 0.0
la 0.0 0.0 0.0 0.1 0.8 0.1
alfombra 0.0 0.0 0.0 0.0 0.2 0.8
Interpretation:
- "El" attends mainly to "The" (0.9).
- "gato" attends mainly to "cat" (0.7), but also to "The" (0.2).
- "alfombra" attends mainly to "mat" (0.8), but also to "the" (0.2).
Moral: Attention captures which words are relevant for generating each output.
Why Attention solves RNNs' problems
Problem 1: Sequential processing (solved)
RNN: It processes word by word sequentially (it doesn't parallelize).
A Transformer with attention: It processes all the words simultaneously. Each word computes its attention with all the others in parallel.
Advantage: 10-100× faster (GPUs process in parallel).
Problem 2: Vanishing gradients (solved)
RNN: To capture the relationship between word 1 and word 100, the gradient has to propagate through 99 steps → it vanishes.
A Transformer with attention: Word 100 can attend directly to word 1 (a direct connection) → there are no intermediate steps → no vanishing gradients.
Advantage: It captures long-range dependencies without degradation.
Problem 3: Forgetting context (solved)
RNN: A fixed-size hidden state has to compress information from all the earlier words → it forgets.
A Transformer with attention: Each word can attend directly to any earlier word (without depending on compressed memory) → it doesn't forget.
Advantage: Long context (1,000-100,000+ tokens) vs an RNN (~50-200 words).
Types of Attention
1. Encoder-Decoder Attention (Cross-Attention)
What it is: The output attends to the input (e.g. Spanish attends to English in translation).
Example: You already saw it above (translating "The cat sat" → "El gato se sentó").
Use: Encoder-decoder Transformers (T5, BART, translation models).
2. Self-Attention
What it is: Each word of the input attends to other words of the same input (not to the output).
Example: In "The cat sat on the mat":
- "cat" attends to "The" (the related article).
- "sat" attends to "cat" (the subject performing the action).
- "mat" attends to "on" and "the" (the related preposition and article).
Use: Encoder-only Transformers (BERT), decoder-only ones (GPT). You'll see it in detail in Lesson 04.
How Attention is computed (optional, conceptual)
Three components: Query, Key, Value
For each word, three vectors are computed:
- Query (Q): "What am I looking for?"
- Key (K): "What information do I have?"
- Value (V): "The information I offer."
Process:
- Compare word A's Query with all the words' Keys (e.g. a dot product).
- Compute the attention weights (a softmax of the Q-K similarity).
- Combine all the words' Values (weighted by the attention weights).
Formula (for reference only, you do NOT need to memorize it):
Attention(Q, K, V) = softmax(Q × K^T / √d_k) × V
Moral: Frameworks (PyTorch, Hugging Face) implement this automatically. What matters is understanding what it does (attend to relevant parts), not how exactly it's computed.
Why this matters for an AI Engineer
1. Understanding how LLMs work
ChatGPT, Claude, BERT → they all use attention.
Example: When ChatGPT generates a response, each word it generates attends to:
- Your question (the input).
- The words it has already generated (the partial output).
Moral: Attention is the mechanism that lets LLMs generate coherent and relevant responses.
2. Reasoning about limitations
Context window: LLMs have a limit (e.g. GPT-4: 128K tokens).
Why: Attention scales quadratically with length (if you have N tokens, there are N² Q-K comparisons) → very long sequences are slow/expensive.
Implication: If your app needs to process long documents, you need to split them into chunks or use RAG.
3. Debugging
If an LLM gives an incorrect answer:
Possible cause: Attention doesn't capture the relevant relationship (e.g. the key word is very far away in the prompt).
Solution: Reformulate the prompt (put the relevant information closer), use prompt engineering techniques, or fine-tuning.
Summary
What attention is:
- A mechanism that lets the network focus on relevant parts of the input as it processes each word.
How it works:
- Each word attends to other words with different weights (attention weights).
- It combines information from the relevant words to generate the output.
Why it solves RNNs' problems:
- Parallelization: All the words compute attention simultaneously → 10-100× faster.
- No vanishing gradients: Direct connections between any pair of words.
- It doesn't forget: Each word can attend directly to any other (without compressed memory).
Next step: Lesson 04: Self-Attention — How words "see" each other (a concrete example of attention within the input).
Additional resources
-
The Illustrated Transformer: Attention — Visualizations of attention. In English.
-
Attention Is All You Need (paper) — The original paper. In English. Very technical.
-
3Blue1Brown: Attention (video) — A visual video about attention. In English with subtitles.