Module 4: Transformers - The Modern AI Revolution
7. Transformers in Action: BERT, GPT, T5 and Real Applications
Description
You already understand how Transformers work (attention, encoder-decoder, why they're better than RNNs). Now you're going to see Transformers in the real world: the most important models (BERT, GPT, T5) and their applications.
What you'll learn:
- The 3 most influential Transformer models: BERT, GPT, T5.
- What problems they solve and when to use each one.
- Real applications in products you use.
BERT (Bidirectional Encoder Representations from Transformers)
What it is
Organization: Google (2018)
Architecture: Encoder-only (no decoder)
Goal: To generate contextual representations of text for comprehension tasks (classification, NER, Q&A).
How it works
Pre-training:
- Trained on Wikipedia + books (a massive corpus).
- Task: Masked Language Modeling (MLM) → hide words at random and predict them.
- Example: "The [MASK] sat on the mat" → the model predicts [MASK] = "cat".
Fine-tuning:
- After pre-training, you adjust BERT to your specific task (e.g. classifying product reviews).
- You add a classification layer on top of the encoder.
- You fine-tune with your labeled data (e.g. 10K reviews with a positive/negative label).
Common uses
- Text classification: Sentiment analysis, spam detection, news classification.
- NER (Named Entity Recognition): Extracting names, places, organizations from text.
- Q&A (Question Answering): Given a paragraph and a question, extract the answer.
- Similarity analysis: Computing how similar two texts are.
Example: Google Search
Problem: The query "2019 brazil traveler to usa need a visa" → before BERT, Google didn't properly understand the "traveler to usa" relationship (a traveler going TO the usa).
Solution (with BERT): BERT understands bidirectional context → "traveler to usa" means a traveler going TO the usa (not from the usa) → better results.
Impact: BERT was integrated into Google Search (2019) → 10% of English queries improved significantly.
GPT (Generative Pre-trained Transformer)
What it is
Organization: OpenAI (GPT: 2018, GPT-2: 2019, GPT-3: 2020, GPT-4: 2023)
Architecture: Decoder-only (no encoder)
Goal: To generate coherent text (autocomplete, chat, generation).
How it works
Pre-training:
- Trained on a massive corpus from the internet (CommonCrawl, books, Wikipedia, etc.).
- Task: Next Token Prediction → predict the next word.
- Example: "The cat sat on" → the model predicts "the" or "a".
Inference (use):
- You give it a prompt (e.g. "Write a poem about cats") and it generates a continuation.
Fine-tuning (optional):
- GPT-3.5/GPT-4 are fine-tuned with RLHF (Reinforcement Learning from Human Feedback) to follow instructions and be useful (→ ChatGPT).
Common uses
- Chat/Conversation: ChatGPT, virtual assistants.
- Text generation: Writing articles, emails, code.
- Autocomplete: GitHub Copilot (based on GPT Codex).
- Translation: Less common than encoder-decoder, but it works.
- Summarization: Condensing long texts.
Example: ChatGPT (GPT-3.5 / GPT-4)
Architecture: GPT-3.5/GPT-4 (a decoder-only Transformer) + RLHF.
Capabilities:
- Answering questions.
- Generating code.
- Explaining concepts.
- Translating languages.
- Writing creatively.
Impact: 100M users in 2 months (the fastest-growing application in history).
T5 (Text-To-Text Transfer Transformer)
What it is
Organization: Google (2019)
Architecture: Encoder-Decoder (complete)
Goal: To formulate every NLP task as a text-to-text transformation (input text → output text).
How it works
Key concept: Every task is text-to-text.
- Translation: Input: "translate English to Spanish: The cat sat" → Output: "El gato se sentó".
- Summarization: Input: "summarize: [long text]" → Output: [summary].
- Classification: Input: "classify sentiment: This movie is great!" → Output: "positive".
Pre-training:
- Trained on C4 (Colossal Clean Crawled Corpus, ~750GB of text).
- Task: Masked span prediction → hide spans (e.g. 3-5 words) and predict them.
Common uses
- Machine translation: Text in language A → text in language B.
- Summarization: A long document → a short summary.
- Format transformation: JSON → readable text, SQL → natural language.
- Data augmentation: Generating text variations for training.
Example: Google Translate
Before Transformers: Google Translate used RNNs (seq2seq).
With Transformers (T5, mT5): Better accuracy, faster, support for more languages.
mT5 (multilingual T5): Trained on 101 languages → quality translation between any pair of languages.
Comparison: BERT vs GPT vs T5
| Aspect | BERT | GPT | T5 |
|---|---|---|---|
| Architecture | Encoder-only | Decoder-only | Encoder-Decoder |
| Goal | Comprehension | Generation | Transformation |
| Pre-training task | Masked LM | Next token prediction | Masked span |
| Main use | Classification, NER, Q&A | Chat, generation, autocomplete | Translation, summarization |
| Does it generate text? | NO | YES | YES |
| Bidirectional? | YES (it attends to future words) | NO (it only attends to earlier words) | The encoder yes, the decoder no |
Real Applications
1. Search
- Google Search: BERT for understanding queries.
- Bing: GPT-4 integrated (chat in the search results).
2. Virtual Assistants
- ChatGPT (OpenAI): GPT-3.5/GPT-4.
- Claude (Anthropic): A decoder-only Transformer.
- Gemini (Google): A multimodal Transformer.
3. Code Generation
- GitHub Copilot: GPT Codex (based on GPT-3).
- Cursor (IDE): GPT-4 for code completion.
4. Translation
- Google Translate: T5/mT5.
- DeepL: An encoder-decoder Transformer.
5. Sentiment Analysis / Classification
- Zendesk: BERT for classifying support tickets.
- Twitter: BERT for detecting toxic content.
Why this matters for an AI Engineer
1. Model selection
Your task: Classifying product reviews (positive/negative).
Recommended model: BERT (encoder-only) → fine-tune with your labeled reviews.
Why NOT GPT: GPT is generative; classification is a comprehension task (better with an encoder).
2. Fine-tuning vs API
Option A: Fine-tuning (BERT, T5)
- Advantage: More control, you can adapt it to your specific domain.
- Disadvantage: It requires labeled data, infrastructure (GPU), time.
Option B: API (GPT-4, Claude)
- Advantage: No fine-tuning required, immediate access, it scales automatically.
- Disadvantage: Cost per token, less control, dependence on the provider.
3. Evaluating trade-offs
Example: A chatbot for customer support.
- GPT-4 (API): Very good responses, but $0.03/1K tokens → expensive if there's a lot of traffic.
- GPT-3.5 (API): Cheaper ($0.002/1K tokens), but less accurate responses.
- A fine-tuned BERT + rules: Cheaper (only hosting cost), but less flexible.
Decision: It depends on budget, traffic volume, required quality.
Summary
Three key Transformer models:
- BERT (encoder-only): Comprehension/classification. Used in Google Search, text classification.
- GPT (decoder-only): Generation/chat. Used in ChatGPT, Copilot, Claude.
- T5 (encoder-decoder): Transformation/translation. Used in Google Translate, summarization.
Real applications:
- Search (BERT in Google).
- Chat (GPT in ChatGPT).
- Translation (T5 in Google Translate).
- Code (GPT Codex in Copilot).
How to choose:
- Classification/NER/Q&A → BERT.
- Generation/chat → GPT.
- Translation/summarization → T5.
Next step: Lesson 08: Integrative Exercise — Visualizing attention, choosing models based on the problem.
Additional resources
- BERT Paper — The original paper. Technical.
- GPT-3 Paper — The GPT-3 paper. Technical.
- T5 Paper — The T5 paper. Technical.
- Hugging Face Models — Thousands of pre-trained Transformer models (BERT, GPT, T5, etc.).