Module 4: Transformers - The Modern AI Revolution

4. Self-Attention: How Words "See" Each Other

Next →

Description

In the previous lesson you saw attention in general (how the output attends to the input, e.g. translation). Now you're going to understand self-attention: how the words of the same input "see" each other in order to understand context.

Self-attention is the central mechanism of BERT, GPT, and all modern Transformers.

What you'll learn:

  • What self-attention is (the difference from encoder-decoder attention).
  • How it works, with a concrete example ("The cat sat on the mat").
  • Why it's crucial for language comprehension.

What Self-Attention is

Definition: Each word of the input attends to all the other words of the same input in order to understand context and relationships.

Difference from encoder-decoder attention:

  • Encoder-decoder attention: The output (Spanish) attends to the input (English).
  • Self-attention: The input (English) attends to itself (English to English).

Analogy: Reading a sentence and relating words to each other.

  • "The cat sat on the mat" → "cat" is the subject, "mat" is the object, they're related by "sat on".
  • Self-attention captures those relationships automatically.

Example: "The cat sat on the mat"

Step 1: Self-attention for "cat"

Question: To understand what "cat" means in this sentence, which other words should I attend to?

Answer:

  • "The" (the article that modifies "cat") → high attention.
  • "sat" (the verb whose subject "cat" is) → high attention.
  • "on" (the preposition that relates "sat" to the object) → moderate attention.
  • "the" (the second "the") → little attention.
  • "mat" (the object where the cat is) → moderate attention.

Attention weights (conceptual):

The: 0.4
cat: 0.1 (itself, always a little)
sat: 0.3
on: 0.1
the: 0.05
mat: 0.05

Interpretation: "cat" attends mainly to "The" (the article) and "sat" (the verb) in order to understand that it's the subject of the action.

Step 2: Self-attention for "sat"

Question: To understand "sat", which other words should I attend to?

Answer:

  • "cat" (the subject performing the action) → very high attention.
  • "on" (the preposition that follows the verb) → high attention.
  • "mat" (the object of the preposition) → moderate attention.
  • "The", "the" → little attention.

Attention weights:

The: 0.05
cat: 0.5
sat: 0.1
on: 0.25
the: 0.05
mat: 0.05

Interpretation: "sat" attends mainly to "cat" (the subject) and "on" (the preposition) in order to understand the "subject + verb + preposition + object" structure.

Step 3: Self-attention for "mat"

Question: To understand "mat", which other words should I attend to?

Answer:

  • "the" (the second "the", the article that modifies "mat") → very high attention.
  • "on" (the preposition that connects to "mat") → high attention.
  • "sat" (the verb that relates "cat" to "mat") → moderate attention.
  • "cat" (the subject related to "mat") → little attention.

Attention weights:

The: 0.05
cat: 0.05
sat: 0.15
on: 0.3
the: 0.4
mat: 0.05

Interpretation: "mat" attends mainly to "the" (the article) and "on" (the preposition) in order to understand that it's the object of the preposition.

Visualizing Self-Attention

The self-attention matrix (each row represents how much each word attends to the others):

       The  cat  sat   on  the  mat
The   [0.3  0.4  0.1  0.1  0.05 0.05]
cat   [0.4  0.1  0.3  0.1  0.05 0.05]
sat   [0.05 0.5  0.1  0.25 0.05 0.05]
on    [0.05 0.1  0.3  0.1  0.3  0.15]
the   [0.05 0.05 0.1  0.3  0.1  0.4]
mat   [0.05 0.05 0.15 0.3  0.4  0.05]

How to read it:

  • Row "cat", column "The": 0.4 → "cat" attends a lot to "The".
  • Row "sat", column "cat": 0.5 → "sat" attends a lot to "cat".

Why Self-Attention is powerful

1. It captures syntactic relationships

Example: "sat" attends to "cat" → it captures the subject-verb relationship.

Use: Syntactic analysis (parsing) tasks, NER (Named Entity Recognition).

2. It captures semantic relationships

Example: "mat" attends to "on" → it understands that "mat" is the object of the preposition.

Use: Reading comprehension, Q&A (Question Answering).

3. It resolves ambiguity

Example: "The bank by the river is closed."

  • "bank" can mean "a financial institution" or "the edge (of the river)".
  • Self-attention: "bank" attends to "river" → it understands that it's the riverbank.

Use: Word sense disambiguation.

Multi-Head Attention

Problem: A single self-attention layer may not capture every relationship (syntactic, semantic, etc.).

Solution: Multi-head attention → multiple attention "heads" that capture different types of relationships.

Example (conceptual):

  • Head 1: Captures subject-verb relationships ("cat" - "sat").
  • Head 2: Captures article-noun relationships ("The" - "cat", "the" - "mat").
  • Head 3: Captures preposition-object relationships ("on" - "mat").

Advantage: Each head specializes in one type of relationship → the model learns richer representations.

Implementation: BERT-base has 12 attention heads per layer. GPT-3 has 96 attention heads per layer.

Why this matters for an AI Engineer

1. Understanding model architectures

When you read model documentation:

  • "BERT-base: 12 layers, 12 attention heads" → 12 Transformer layers, each layer with 12 multi-head attention heads.
  • "GPT-3: 96 layers, 96 attention heads" → 96 layers, 96 heads per layer.

Interpretation: More heads → more capacity to capture complex relationships (but more compute).

2. Debugging

If a model fails at a task:

Example: BERT misclassifies "The bank by the river is closed" (it confuses "bank" with a financial institution instead of a riverbank).

Possible cause: Attention doesn't capture the "bank - river" relationship correctly.

Solution: Fine-tuning with more similar examples, or using a larger model (more heads, more layers).

3. Attention visualization

Tools: BertViz (an interactive attention visualizer for BERT).

Use: Seeing which words attend to which → understanding what the model learned.

Application: Debugging, explainability (explaining to stakeholders why the model makes a certain decision).

Summary

Self-attention: Each word of the input attends to all the other words of the same input.

How it works:

  • Each word computes attention weights with all the others.
  • It combines information from the relevant words (weighted by the attention weights).

Why it's powerful:

  • It captures syntactic relationships (subject-verb, article-noun).
  • It captures semantic relationships (preposition-object, related words).
  • It resolves ambiguity (e.g. "bank" near "river" → riverbank).

Multi-head attention: Multiple heads that capture different types of relationships → richer representations.

Next step: Lesson 05: The Transformer Architecture — How self-attention, the encoder, the decoder, positional encoding, etc. are integrated.

Additional resources

  1. BertViz: Visualizing Attention — A tool for visualizing attention in BERT. In English. Requires Python.

  2. The Illustrated Transformer: Self-Attention — Visualizations of self-attention. In English.

← Previous3. The Attention Mechanism: Transformers' Key IdeaNext →5. The Transformer Architecture: Encoder, Decoder and Positional Encoding