Module 4: Transformers - The Modern AI Revolution

8. Integrative Exercise: Understanding Attention and Choosing Models

Next →

Description

This is the final lesson of Module 4. Here you'll apply everything you learned about Transformers: attention, self-attention, encoder-decoder, BERT vs GPT vs T5.

What you'll practice:

  1. Visualizing attention in sentences (identifying which words attend to which).
  2. Classifying problems and choosing a Transformer architecture (BERT, GPT, T5).
  3. Interpreting Transformer model documentation.
  4. Identifying common mistakes in the use of Transformers.

Part 1: Visualizing Attention

Exercise 1: Self-Attention in "The cat sat on the mat"

Task: For each word, identify the 2-3 words it should attend to most (with high attention).

Format:

Word X attends mainly to:
- Word Y (reason: ...)
- Word Z (reason: ...)

1. "The" (article)

See solution

It attends mainly to:

  • "cat" (the noun it modifies, an article-noun relationship).
  • Itself ("The") (there's always some self-attention).

Reason: Articles modify nearby nouns → "The" needs to attend to "cat" in order to understand that it's the subject's article.

2. "cat" (noun, subject)

See solution

It attends mainly to:

  • "The" (the article that modifies it).
  • "sat" (the verb whose subject "cat" is).

Reason: "cat" is the sentence's subject → it needs to attend to the article ("The") and the verb ("sat") in order to understand its role.

3. "sat" (verb)

See solution

It attends mainly to:

  • "cat" (the subject performing the action).
  • "on" (the preposition that follows the verb).

Reason: The verb needs to attend to the subject (who performs the action) and to the preposition (what comes after).

4. "on" (preposition)

See solution

It attends mainly to:

  • "sat" (the verb that precedes the preposition).
  • "mat" (the object of the preposition).

Reason: The preposition connects the verb with the object → it needs to attend to both.

5. "the" (the second article)

See solution

It attends mainly to:

  • "mat" (the noun it modifies).
  • "on" (the preceding preposition).

Reason: An article modifies a nearby noun → "the" attends to "mat".

6. "mat" (noun, object)

See solution

It attends mainly to:

  • "the" (the article that modifies it).
  • "on" (the preposition that connects it with the verb).

Reason: "mat" is the object of the preposition → it needs to attend to the article and the preposition.

Part 2: Choosing a Transformer model based on the problem

Exercise 2: Classify tasks and choose an architecture

For each problem, identify:

  1. The task: Comprehension, generation, or transformation.
  2. The recommended architecture: BERT (encoder-only), GPT (decoder-only), or T5 (encoder-decoder).
  3. Justification: Why that architecture.

Problem 1: Classify emails as spam or not-spam

See solution

Task: Comprehension/classification (it does NOT generate text).

Architecture: BERT (encoder-only).

Justification:

  • BERT processes the input (the email) and generates a contextual representation.
  • You add a classification layer on top (spam/not-spam).
  • You do NOT need to generate text → a decoder is unnecessary.

Alternative: You can use GPT-4 via API (prompt: "Classify this email as spam or not-spam: ..."), but it's less efficient (more expensive, more latency) than a fine-tuned BERT.

Problem 2: A chatbot that answers customer questions

See solution

Task: Generation (it generates responses in natural language).

Architecture: GPT (decoder-only).

Justification:

  • GPT generates text sequentially (autocomplete).
  • You give it a prompt (the customer's question) and it generates a response.
  • Decoder-only is ideal for conversational generation.

Example: ChatGPT, Claude (both decoder-only).

Problem 3: Translate legal documents from English to Spanish

See solution

Task: Transformation (text in language A → text in language B).

Architecture: T5 (encoder-decoder).

Justification:

  • The encoder processes the input (English), generates a representation.
  • The decoder generates the output (Spanish), attending to the encoder via cross-attention.
  • Encoder-decoder is ideal for transformation (translation, summarization).

Alternative: GPT-4 can translate (via a prompt), but a T5 fine-tuned on legal translation will be more accurate.

Problem 4: Generate Python code from a natural-language description

See solution

Task: Generation (it generates code).

Architecture: GPT (decoder-only).

Justification:

  • You generate code sequentially (line by line).
  • GPT Codex (based on GPT-3) specializes in code.

Example: GitHub Copilot (it uses GPT Codex).

Alternative: A T5 fine-tuned on code generation (it works, but GPT is more common for this task).

Problem 5: Extract the names of people and organizations from news articles (NER)

See solution

Task: Comprehension/extraction (it does NOT generate text).

Architecture: BERT (encoder-only).

Justification:

  • BERT processes the input (the news article), generates a contextual representation of each word.
  • You add a classification layer on top of each word (label: Person, Organization, Place, O (none)).
  • It does NOT generate text → a decoder is unnecessary.

Example: spaCy (an NLP library) uses BERT for NER.

Part 3: Interpreting model documentation

Exercise 3: BERT-base vs BERT-large

BERT-base:

  • 12 encoder layers
  • 768 hidden size
  • 12 attention heads per layer
  • 110M parameters

BERT-large:

  • 24 encoder layers
  • 1024 hidden size
  • 16 attention heads per layer
  • 340M parameters

Questions:

  1. Which is more expressive (able to learn more complex relationships)?
  2. Which is faster (lower latency)?
  3. When would you use BERT-base vs BERT-large?
See answers

1. Which is more expressive?

  • BERT-large (24 layers, 1024 dimensions, 340M parameters) is more expressive → it can learn more complex relationships.

2. Which is faster?

  • BERT-base (12 layers, 110M parameters) is faster → fewer layers, fewer parameters → less compute per forward pass.

3. When to use each one?

  • BERT-base: When latency is critical (responses in <100ms), or when BERT-large's accuracy doesn't justify the additional cost.
  • BERT-large: When accuracy is critical and latency/cost aren't limiting factors (e.g. sentiment analysis for academic research).

Trade-off: BERT-large is ~2-3% more accurate but ~3× slower and more expensive than BERT-base.

Part 4: Identifying common mistakes

Case 1: Using BERT to generate text

Scenario: You want a chatbot that generates responses. You decide to use a fine-tuned BERT.

Question: What's wrong?

See answer

Problem: BERT is encoder-only → it does NOT generate text, it only understands/classifies.

Why: BERT has no decoder → it can't generate output sequences.

Solution: Use GPT (decoder-only) or T5 (encoder-decoder) for generation.

Case 2: The context window was exceeded

Scenario: You pass a 200K-word document (~270K tokens) to GPT-4 (context window: 128K tokens). The model gives an error or an incomplete response.

Question: What happened? How do you fix it?

See answer

Problem: You exceeded the context window (270K tokens > 128K tokens).

What happened: GPT-4 CAN'T process more than 128K tokens → it truncates the input or gives an error.

Solutions:

  1. Split it into chunks: Split the document into 3 chunks of ~90K tokens each, process them separately.
  2. Use Claude 3: A context window of 200K tokens → it can process the complete document.
  3. RAG (Retrieval-Augmented Generation): Instead of passing the complete document, search for the relevant sections (e.g. with embedding search) and pass only those sections to the model.

Case 3: Fine-tuning GPT-4 for classification

Scenario: You want to classify product reviews (positive/negative). You decide to fine-tune GPT-4.

Question: Is it the best option? What alternatives are there?

See answer

Is it the best option? Not necessarily.

Disadvantages of GPT-4 for classification:

  • GPT-4 is generative (decoder-only) → more complex than necessary for classification (you only need comprehension).
  • Fine-tuning GPT-4 is expensive (OpenAI charges for fine-tuning + inference).
  • GPT-4 via API already works well with prompt engineering (zero-shot or few-shot) → fine-tuning may not be necessary.

Better alternatives:

  1. A fine-tuned BERT: Encoder-only is ideal for classification. More efficient (fewer parameters, faster) than GPT.
  2. GPT-4 via API with a prompt: If you don't want to fine-tune, use GPT-4 with a well-designed prompt (e.g. few-shot examples).

When to fine-tune GPT-4: If you need generation specific to your domain (e.g. generating responses in your brand's specific style).

Summary of the exercise

What you practiced:

  1. Visualizing attention: Identifying which words attend to which and why (it captures syntactic and semantic relationships).
  2. Choosing a model: BERT for comprehension, GPT for generation, T5 for transformation.
  3. Interpreting documentation: Understanding what layers, attention heads, parameters mean (accuracy vs latency trade-offs).
  4. Avoiding mistakes: Not using BERT for generation, not exceeding the context window, choosing an appropriate architecture.

Evidence of success:

  • You can choose the correct Transformer model for a new problem (without consulting the lessons).
  • You can justify your choice (why BERT/GPT/T5 depending on the task).
  • You can reason about limitations (context window, latency, costs).

Next module: With this exercise you finish Module 4 (Transformers). In Module 5 (LLMs) you'll see how ChatGPT, Claude, GPT-4 work in detail (tokenization, embeddings, context window, model parameters like temperature).

Additional resources

  1. BertViz — An interactive attention visualizer for BERT. Requires Python.
  2. Hugging Face Transformers Course — A practical course with exercises. In English. With code.
  3. Papers With Code: Transformers — Transformer variants and benchmarks. In English.
← Previous7. Transformers in Action: BERT, GPT, T5 and Real ApplicationsNext →1. Module Introduction: The Era of LLMs