Module 4: Transformers - The Modern AI Revolution
2. The Problem with RNNs: Why We Need Transformers
Description
Before understanding why Transformers are revolutionary, you need to understand what problem they solve. That problem is the limitations of RNNs (Recurrent Neural Networks), which were the dominant architecture in NLP before 2017.
In this lesson you'll understand:
- What RNNs are (a quick review from Module 3).
- Why RNNs are limited (sequential processing, vanishing gradients, forgetting long context).
- Why those limitations made Transformers necessary.
Approach: Conceptual, with diagrams and concrete examples. You'll see RNN vs Transformer comparisons (sequential vs parallel, short vs long context).
Review: What RNNs are
Definition
RNN (Recurrent Neural Network): A neural network designed to process sequences (text, time series, audio). It has a hidden state that is updated at each step of the sequence, maintaining a "memory" of previous steps.
Diagram (processing "The cat is"):
Input: "The" → "cat" → "is"
↓ ↓ ↓
State: h₁ → h₂ → h₃ → Output
At each step:
- The RNN receives the current input (e.g. "cat") + the previous state (h₁).
- It processes both and generates a new state (h₂).
- The new state "remembers" information from earlier words.
Problem 1: Sequential processing (slow)
How processing works in RNNs
RNNs process words one at a time (sequentially):
Step 1: Process "The" → generates h₁
Step 2: Process "cat" + h₁ → generates h₂
Step 3: Process "is" + h₂ → generates h₃
Step 4: Process "on" + h₃ → generates h₄
...
Problem: Each step depends on the previous one. You can't process "is" until you've processed "cat". You can't process "cat" until you've processed "The".
Consequence: You can't parallelize (use multiple GPUs/cores to process several words simultaneously). You have to wait step by step.
Impact on speed
Example: A 100-word sentence.
RNN: It processes word 1, then word 2, ..., then word 100 → 100 sequential steps.
A modern GPU (e.g. NVIDIA A100): It has thousands of cores that could process many words simultaneously, but an RNN CAN'T take advantage of them (sequential processing).
Result: Training RNNs on large datasets (millions of sentences) is very slow (days, weeks).
How Transformers solve this
Transformers process the whole sentence at once (in parallel):
All the words processed simultaneously:
"The" "cat" "is" "on" "the" "roof"
↓ ↓ ↓ ↓ ↓ ↓
[Attention captures the relationships among all of them]
Advantage: GPUs process all the words in parallel → 10-100× faster than RNNs.
Problem 2: Vanishing gradients (slow learning)
What vanishing gradients are
Reminder from Module 3: During backpropagation, gradients propagate from the output toward the input. If the gradients become very small (nearly 0), the early layers don't learn.
In RNNs: To process a long sentence (e.g. 100 words), the gradient has to propagate through 100 steps (word 100 → 99 → 98 → ... → 1).
Problem: At each step, the gradient is multiplied by values < 1 (the derivatives of activation functions like tanh). After 100 multiplications, the gradient is nearly 0.
Consequence: Early words (e.g. word 1) barely learn → the RNN can't capture long-range dependencies.
Concrete example
Sentence: "The cat that my sister adopted last year is sleeping on the roof."
Long-range dependency: "The cat" (words 1-2) is the subject of "is sleeping" (words 11-12). There are 9 words in between.
RNN: For the gradient from "is sleeping" to reach "The cat", it has to propagate through 9 intermediate steps → the gradient becomes very small → the RNN learns little about the "cat - is sleeping" relationship.
Result: The RNN captures who the subject is poorly (it can predict "is sleeping" without correctly associating it with "cat").
LSTMs: a partial solution
LSTM (Long Short-Term Memory): A variant of the RNN designed to mitigate vanishing gradients using "gates" that control which information to keep or discard.
Advantage: LSTMs capture long-range dependencies better than simple RNNs.
Disadvantage: They're still sequential (slow) and they have limits (after 50-100 words, they start forgetting).
How Transformers solve this
Transformers use attention: Each word can "attend" directly to any other word (without passing through many intermediate steps).
Example: In "The cat ... is sleeping", "is sleeping" can attend directly to "The cat" (without passing through the 9 intermediate words) → there are no vanishing gradients.
Advantage: They capture long-range dependencies without degradation (even in sentences of 100+ words).
Problem 3: They forget long context (limited memory)
How RNNs "remember"
RNNs keep their "memory" in the hidden state:
h₁ (after "The") → contains information about "The"
h₂ (after "cat") → contains information about "The" + "cat"
h₃ (after "is") → contains information about "The" + "cat" + "is"
Problem: The hidden state has a fixed size (e.g. 512 dimensions). As you process more words, the hidden state has to compress all the previous information into those 512 dimensions.
Consequence: Information from early words is forgotten or degraded as you process more words.
Concrete example
A long text (200 words): A paragraph describing a cat, then 100 words about other topics, then a question: "What color was the cat?"
RNN: When it reaches the question (word 200), the hidden state has gone through 200 updates → the information about the cat (mentioned in words 10-20) is heavily compressed or lost → the RNN can't answer correctly.
LSTM: Better than a simple RNN, but after 100-200 words it also starts forgetting.
How Transformers solve this
Transformers with attention: Each word can attend directly to any earlier word (without depending on a compressed hidden state).
Example: When it processes "What color was the cat?" (word 200), it can attend directly to words 10-20 (where the cat is described) → it doesn't depend on compressed memory → it doesn't forget.
Limitation: The context window (you'll see it in Module 5). Transformers can't process infinite texts, but they can handle 1,000-100,000+ tokens (far more than RNNs).
Comparison: RNN vs Transformer
| Aspect | RNN/LSTM | Transformer |
|---|---|---|
| Processing | Sequential (word by word) | Parallel (the whole sentence at once) |
| Speed (training) | Slow (it doesn't parallelize) | Fast (10-100× faster) |
| Long dependencies | Poor (vanishing gradients) | Excellent (direct attention) |
| Memory | Limited (a fixed hidden state) | Excellent (attention to any word) |
| Context window | ~50-200 words | 1,000-100,000+ tokens |
| Scalability | Difficult (it doesn't parallelize well) | Excellent (it scales to massive GPUs) |
Why this matters for an AI Engineer
1. Understanding why we use Transformers today
Before 2017: RNNs/LSTMs were state-of-the-art in NLP.
After 2017: Transformers completely replaced RNNs in NLP (BERT, GPT, T5, etc.).
Reason: Transformers are better at everything (faster, better context, more scalable).
Consequence: As an AI Engineer (2024), you'll almost never use RNNs for text. You'll use Transformers (BERT, GPT, Claude, etc.).
2. Understanding Transformers' limitations
Context window: Even though Transformers handle long context better than RNNs, they still have limits (e.g. GPT-4: 128K tokens).
Why: Attention scales quadratically with length → very long sequences are slow/expensive.
Implication: If your app needs to process documents of 1M words, you need to split them into chunks or use RAG (Retrieval-Augmented Generation).
3. Reasoning about costs
Transformers are more expensive to run (inference) than RNNs:
- Attention requires more compute than an RNN's simple sequential processing.
- Large models (GPT-4: ~1T parameters) require powerful GPUs.
Trade-off: Transformers are more expressive (better accuracy) but more expensive. For apps with a limited budget, you can use smaller models (DistilBERT, GPT-3.5 instead of GPT-4).
Common mistakes
1. Thinking RNNs are no longer used
Mistake: Believing that RNNs are "obsolete" and nobody uses them.
Reality: In NLP, Transformers replaced RNNs. But RNNs/LSTMs are still used in:
- Time series: Predicting stock prices, product demand.
- Audio: Speech recognition (although Transformers are also replacing them here).
- System control: Robotics, motor control.
Moral: RNNs are NOT dead, but for text (NLP), Transformers are superior.
2. Confusing "sequential" (processing) with "sequential" (data)
Mistake: Thinking that Transformers CAN'T process sequences (because they aren't "sequential" like RNNs).
Reality:
- Transformers do process sequential data (text, audio, time series).
- But they process the complete sequence in parallel (not word by word).
Moral: Transformers DO process sequences, but in a parallel (not sequential) way.
Summary
Three main problems with RNNs:
- Sequential processing: Slow (they don't parallelize) → training on large datasets takes days/weeks.
- Vanishing gradients: They capture long-range dependencies poorly (they forget early words).
- Limited memory: A fixed hidden state → they forget context after 50-200 words.
How Transformers solve these problems:
- Parallel processing: They process the whole sentence at once → 10-100× faster.
- Direct attention: Each word can attend directly to any other → no vanishing gradients.
- They don't depend on fixed memory: Attention captures relationships without compressing into a hidden state → they don't forget.
Why it matters:
- Understanding RNNs' limitations helps you appreciate why Transformers are revolutionary.
- As an AI Engineer, you'll use Transformers (BERT, GPT) instead of RNNs for text.
- But you need to understand the trade-offs (Transformers are faster to train, but more expensive for inference if you don't optimize).
Next step: Lesson 03: The Attention Mechanism — What attention is and how it works (conceptually).
Additional resources
-
Colah's Blog: Understanding LSTMs — A visual explanation of LSTMs. In English.
-
The Unreasonable Effectiveness of RNNs (Karpathy) — A classic post about RNNs. In English.
-
Why Transformers are Better than RNNs (video) — A video comparing RNNs and Transformers. In English.