Contextualization: Embeddings That Understand Context

Capsule overview

The fundamental difference between modern embeddings (BERT, OpenAI) and legacy ones (Word2Vec) is contextualization: the ability to generate different embeddings for the same word depending on its context.

In this capsule you'll learn what contextual embeddings are, why they're superior to static embeddings, how self-attention achieves contextualization, and you'll see concrete examples of polysemy (words with multiple meanings). You'll also write code to demonstrate that contextual embeddings capture nuances that static ones cannot.

By the end, you'll understand why the Transformer revolution was so disruptive for NLP.

Static vs contextual embeddings

Static embeddings (Word2Vec, GloVe):

A fixed embedding per word, independent of context.

# Word2Vec (legacy - pre-2018):
vocab = {
    "bank": [0.5, 0.3, 0.8, -0.2, 0.6],  # ALWAYS this vector
    "river": [0.2, 0.6, 0.1, 0.4, -0.3],
    "money": [0.9, 0.1, 0.7, -0.5, 0.2]
}

# Problem: "bank" is ALWAYS the same embedding
text_1 = "The river bank"
text_2 = "The bank is closed"

# In both cases:
embedding_bank = vocab["bank"]  # [0.5, 0.3, 0.8, -0.2, 0.6]
# ❌ No difference between "riverbank" and "financial"

Contextual embeddings (BERT, OpenAI):

A dynamic embedding that depends on the full context.

# Transformers (modern - post-2018):
text_1 = "The river bank was flooded"
text_2 = "The bank is closed today"

# Generate contextual embeddings
emb_bank_1 = get_contextual_embedding(text_1, word="bank")
# → [0.2, 0.5, 0.1, ...]  (contextualized with "river")

emb_bank_2 = get_contextual_embedding(text_2, word="bank")
# → [0.8, 0.3, 0.6, ...]  (contextualized with "closed")

# ✅ DIFFERENT embeddings depending on context
similarity = cosine_similarity(emb_bank_1, emb_bank_2)
print(f"Similarity: {similarity:.4f}")  # ~0.70 (related but distinct)

Real example: Polysemy with OpenAI

Demonstration of contextualization:

from openai import OpenAI
import numpy as np
import os
from dotenv import load_dotenv

load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def get_embedding(text):
    """Generate a full embedding (sentence)"""
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return np.array(response.data[0].embedding)

def cosine_similarity(vec_a, vec_b):
    return np.dot(vec_a, vec_b) / (np.linalg.norm(vec_a) * np.linalg.norm(vec_b))

# Different contexts of "bank"
context_1 = "The river bank is polluted"        # Riverbank
context_2 = "The bank is closed on Sundays"      # Financial
context_3 = "The pilot had to bank the plane"    # Aviation (tilt)

# Generate embeddings (full sentences)
emb_1 = get_embedding(context_1)
emb_2 = get_embedding(context_2)
emb_3 = get_embedding(context_3)

# Compare similarities
print("Contextualization of 'bank':")
print(f"Riverbank vs Financial: {cosine_similarity(emb_1, emb_2):.4f}")
print(f"Riverbank vs Aviation: {cosine_similarity(emb_1, emb_3):.4f}")
print(f"Financial vs Aviation: {cosine_similarity(emb_2, emb_3):.4f}")

Expected output:

Contextualization of 'bank':
Riverbank vs Financial: 0.72  ← Related but different
Riverbank vs Aviation: 0.68   ← Somewhat related
Financial vs Aviation: 0.65   ← Less related

Interpretation: Embeddings capture that these are different contexts (scores <0.80).

How self-attention achieves contextualization

Mechanism:

Text: "The river bank"

Step 1: Initial embedding of "bank" (no context)
bank_initial = [0.5, 0.3, 0.8, ...]  (generic)

Step 2: Self-attention computes attention weights
- "bank" attends to "river" (weight = 0.8)
- "bank" attends to "The" (weight = 0.1)
- "bank" attends to itself (weight = 1.0)

Step 3: Weighted sum updates "bank"
bank_contextualized = 0.8 × river_emb + 0.1 × the_emb + 1.0 × bank_emb
                     → [0.2, 0.5, 0.1, ...]  (now contextualized with "river")

Result: "bank" incorporates information from "river" into its embedding.

Multiple layers refine the contextualization:

Layer 1: "bank" sees "river" (immediate context)
      ↓
Layer 2: "bank" sees "river" + "The" (broader context)
      ↓
Layer 3-6: Semantic refinement
      ↓
Layer 7-12: Global context (the whole sentence)
      ↓
Final embedding: "bank" fully contextualized

More layers → better contextualization.

Examples of polysemy

1. "Apple" (polysemy in English):

contexts = [
    "I ate an apple for breakfast",           # Fruit
    "Apple released a new iPhone",            # Company
    "The apple tree is in the garden"         # Tree
]

# Generate embeddings
embeddings = [get_embedding(ctx) for ctx in contexts]

# Compare
print("Apple - Polysemy:")
print(f"Fruit vs Company: {cosine_similarity(embeddings[0], embeddings[1]):.4f}")
print(f"Fruit vs Tree: {cosine_similarity(embeddings[0], embeddings[2]):.4f}")
print(f"Company vs Tree: {cosine_similarity(embeddings[1], embeddings[2]):.4f}")

Expected output:

Apple - Polysemy:
Fruit vs Company: 0.68  ← Different domains
Fruit vs Tree: 0.82    ← Related (botany)
Company vs Tree: 0.65  ← Unrelated

2. "Playing" (different senses):

contexts = [
    "She is playing piano beautifully",       # Instrument
    "Kids are playing in the park",           # To play (games)
    "The movie is playing at the cinema"      # Showing
]

embeddings = [get_embedding(ctx) for ctx in contexts]

print("Playing - Polysemy:")
print(f"Instrument vs Games: {cosine_similarity(embeddings[0], embeddings[1]):.4f}")
print(f"Instrument vs Showing: {cosine_similarity(embeddings[0], embeddings[2]):.4f}")

Expected output:

Playing - Polysemy:
Instrument vs Games: 0.75      ← Somewhat related (both "playing")
Instrument vs Showing: 0.70    ← Less related

Advantages of contextualization for AI Engineering

1. More precise semantic search:

# User searches: "bank"

# With Word2Vec (static):
# Returns: "river bank", "financial bank", "plaza bench" (all similar)

# With BERT/OpenAI (contextual):
# If the query is "river bank" → Returns "The river bank" (context matches)
# If the query is "money bank" → Returns "financial bank" (context matches)

2. More effective RAG:

# User asks: "How do I open a bank account?"

# The RAG system searches for relevant docs:
# With contextual embeddings:
# ✅ Finds: "Opening an account at a bank" (high similarity)
# ❌ Does NOT find: "The river bank" (low similarity, different context)

# With static embeddings:
# ⚠️ Finds both (doesn't distinguish contexts)

3. More precise classification:

# Email: "The bank rejected my payment"

# Static embeddings:
# Category: Financial or Nature? (ambiguous)

# Contextual embeddings:
# Category: Financial ✅ (the context "rejected" + "payment" clarifies it)

Limitations of contextualization

1. Requires enough context:

# Little context:
text_minimal = "bank"
emb = get_embedding(text_minimal)
# → Generic embedding (can't contextualize without more text)

# Enough context:
text_full = "I'm going to the bank to deposit money"
emb = get_embedding(text_full)
# → Contextualized embedding ✅

Recommendation: A minimum of 3-5 words of context.

2. Limited global context:

# Transformers have a token limit (8191 for OpenAI)
# Very long text:
long_doc = "..." * 10000  # 10,000 tokens

# Only the first 8191 tokens are considered
# The rest is lost

Solution: Chunk intelligently (Module 4).

Exercises

Exercise 1: Identify polysemy

Which word has multiple meanings?

A: "Python" (language vs snake) B: "house" C: "fast"

emb_tech = get_embedding("Python is a programming language")
emb_animal = get_embedding("Python is a large snake")

similarity = cosine_similarity(emb_tech, emb_animal)
# → ~0.75 (related because of the common word, but different contexts)

Exercise 2: Compare contexts

Generate embeddings for these 3 contexts and determine which are most similar:

contexts = [
    "The cat sleeps on the couch",
    "The cat hunts mice",
    "The dog sleeps on the couch"
]

# Which pairs are most similar?

# Generate embeddings
embs = [get_embedding(ctx) for ctx in contexts]

# Compare pairs
print("Comparison:")
print(f"Cat sleeps vs Cat hunts: {cosine_similarity(embs[0], embs[1]):.4f}")
print(f"Cat sleeps vs Dog sleeps: {cosine_similarity(embs[0], embs[2]):.4f}")
print(f"Cat hunts vs Dog sleeps: {cosine_similarity(embs[1], embs[2]):.4f}")

Comparison:
Cat sleeps vs Cat hunts: 0.85   ← Same subject (cat), different verbs
Cat sleeps vs Dog sleeps: 0.88  ← Different subjects, SAME verb + context
Cat hunts vs Dog sleeps: 0.78   ← Both different

Exercise 3: Detect polysemy

Implement a function that detects whether a word has very different meanings in two contexts:

def detect_polysemy(word, context_1, context_2, threshold=0.75):
    """
    Detect whether a word has different meanings in 2 contexts
    
    Args:
        word: The word to analyze
        context_1: First sentence with the word
        context_2: Second sentence with the word
        threshold: Similarity threshold (< threshold = polysemy)
    
    Returns:
        True if it has different meanings
    """
    # Implement here
    pass

# Test
is_polysemy = detect_polysemy(
    "bank",
    "The river bank",
    "The bank is closed"
)
print(f"Polysemy? {is_polysemy}")

def detect_polysemy(word, context_1, context_2, threshold=0.75):
    """Detect polysemy by comparing context embeddings"""
    # Generate embeddings of the full contexts
    emb_1 = get_embedding(context_1)
    emb_2 = get_embedding(context_2)
    
    # Calculate similarity
    similarity = cosine_similarity(emb_1, emb_2)
    
    # If similarity < threshold → Different meanings
    return similarity < threshold

# Test
is_polysemy = detect_polysemy(
    "bank",
    "The river bank",
    "The bank is closed"
)

print(f"Does 'bank' have polysemy? {is_polysemy}")  # True

# Check with a word without polysemy
is_polysemy_2 = detect_polysemy(
    "cat",
    "The cat sleeps",
    "The cat eats"
)
print(f"Does 'cat' have polysemy? {is_polysemy_2}")  # False (same meaning)

Does 'bank' have polysemy? True   ← Different contexts
Does 'cat' have polysemy? False   ← Same context (animal)

Summary

What you learned:

• ✅ Static embeddings: 1 vector per word (Word2Vec, GloVe)
• ✅ Contextual embeddings: A dynamic vector based on context (BERT, OpenAI)
• ✅ Polysemy: Words with multiple meanings
• ✅ Self-attention: The mechanism that contextualizes
• ✅ Advantages: More precise semantic search, RAG, classification

Key concepts:

1. Contextualization = a different embedding depending on context
2. Self-attention lets tokens "see" each other
3. Multiple layers refine contextualization progressively

Additional resources

1. Contextualized Word Representations - ELMo paper
2. BERT Paper - Bidirectional contextualization
3. Word Senses and Embeddings - Polysemy handling
4. Illustrated BERT - Visualization
5. Contextual Embeddings Tutorial - SBERT

In the next capsule

Capsule 05: Pooling Strategies

You'll learn:

• Mean pooling (numpy code)
• CLS pooling (BERT-style)
• Max pooling (rare)
• Comparison of strategies
• Manual implementation of pooling

From contextualization to aggregation.

Module 2 - Embeddings Deep Dive Guide Embeddings that understand context: the NLP revolution