Module 2: How Do Embeddings Work?

Welcome to Module 2: How Do Embeddings Work?

Overview

Welcome to the second module of Embeddings Deep Dive, where you'll dig into the technical architecture that turns text into vectors: encoder-only Transformers, tokenization, self-attention, and pooling strategies.

In Module 1 you used embeddings as "black boxes" (text → vector). In this module you'll open that box and understand the full pipeline: from characters to the final 1536-dimensional embedding. You'll also build a robust API client with retry logic, rate limiting, and batch processing.

By the end of this module, you'll be able to make informed decisions about embedding models and write production-ready code to consume APIs.

Why is this module critical?

For your code:

Understanding the technical pipeline lets you:

✅ Choose the right model (OpenAI vs SBERT vs BGE)
✅ Debug when embeddings don't behave as you expect
✅ Optimize tokenization to reduce costs
✅ Implement effective batching (10x faster)
✅ Handle edge cases (empty text, very long text, etc.)

Without understanding the "how", you're just copying code without knowing why it works.

For your career:

In technical interviews for AI Engineer, you'll be asked:

"How does a Transformer generate an embedding?"
"What is self-attention and why is it important?"
"How would you optimize calls to embedding APIs?"
"Why normalize embeddings?"

This module gives you the answers with technical depth.

For production systems:

Production-ready systems require:

Batch processing (not 1 API call per document)
Rate limiting (avoid throttling)
Retry logic (handle transient errors)
Cost optimization (caching, compression)

This module teaches these patterns.

Connection with Module 1

What you already know (Module 1):

✅ What embeddings are (vectors that capture meaning)
✅ Vector properties (cosine similarity)
✅ High-dimensional spaces (1536D)
✅ Real use cases (RAG, search, recommendations)
✅ High-level architecture (4 steps: tokenization → transformer → pooling → normalization)

What you'll learn (Module 2):

🎯 Deep tokenization (BPE, WordPiece, tiktoken code)
🎯 Transformer encoders (self-attention, feed-forward layers)
🎯 Contextualization (how "bank" changes with context)
🎯 Pooling strategies (mean vs CLS vs max, implementation)
🎯 Normalization (why and when to normalize)
🎯 Advanced OpenAI API (batching, rate limiting, error handling)
🎯 Robust API client (production-ready)

From high-level concepts to deep technical implementation.

Objectives of this module

By completing this module, you'll be able to:

Conceptual level (Capsules 02-03):

✅ Explain tokenization (BPE, sub-words)
✅ Understand Transformer encoders (architecture)
✅ Understand self-attention (context)
✅ Differentiate contextual vs static embeddings

Technical level (Capsules 04-05):

✅ Implement tokenization with tiktoken
✅ Compare pooling strategies (mean, CLS, max)
✅ Normalize embeddings (L2 norm)
✅ Use the advanced OpenAI API (parameters)

Production level (Capsules 06-08):

✅ Implement batch processing
✅ Handle rate limiting
✅ Implement retry logic with exponential backoff
✅ Build a production-ready API client

Module roadmap

Capsule 01 (this one): Module Introduction

Context, objectives, progression, connection with Module 1.

Capsule 02: Transformer Encoders

Encoder-only architecture, self-attention (conceptual), feed-forward layers.

Capsule 03: Tokenization

BPE, WordPiece, tiktoken code, sub-words, vocabulary.

Capsule 04: Contextualization

Contextual vs static embeddings, the "bank" example, why context matters.

Capsule 05: Pooling Strategies

Mean, CLS, max pooling, implementation with numpy, comparison.

Capsule 06: Normalization

L2 norm, why normalize, dot product vs cosine similarity.

Capsule 07: OpenAI API Advanced

API parameters, batching, error handling, rate limits.

Capsule 08: Mini-Project - Robust API Client

Retry logic, rate limiting, batch processing, production-ready.

Estimated duration

Activity	Time
Reading capsules 02-07	50-60 min
Practice while reading	30-40 min
Mini-project (capsule 08)	40-50 min
Total	~2.5 hours

Note: This module is more technical than Module 1 (40% theory, 60% code).

Verified prerequisites

Before starting, make sure you've completed:

Module 1 completed:

✅ Basic concepts:

Definition of embeddings
Cosine similarity
Vector spaces
Use cases

✅ Module 1 mini-project:

Working similarity calculator
Basic OpenAI API working
Embedding cache

If you did NOT complete Module 1: Do it first. This module assumes that knowledge.

Technical setup:

✅ Python 3.10+ installed

python --version  # ≥ 3.10

✅ Dependencies installed:

pip install openai numpy python-dotenv tiktoken

✅ OpenAI API key configured:

# .env
OPENAI_API_KEY=sk-proj-your-key-here

Module structure

Phase 1: Deep Architecture (Capsules 02-04)

You'll learn the internal architecture of embedding models.

Result: You'll understand the full pipeline from text to vector.

Phase 2: Implementation (Capsules 05-06)

You'll implement pooling and normalization with code.

Result: Working code for pooling strategies and normalization.

Phase 3: Production (Capsules 07-08)

You'll build a production-ready API client.

Result: A robust system with retry, batching, rate limiting.

Differences from Module 1

Module 1 (Fundamentals):

50/50 theory/practice
High-level concepts
Simple project (basic CLI)
Focus: What are embeddings and why?

Module 2 (Architecture):

40/60 theory/practice
Deep technical architecture
Complex project (robust API client)
Focus: How do embeddings work internally?

Natural progression: From concepts to implementation.

What you will NOT learn in this module

Clear boundaries:

❌ You will NOT train Transformers from scratch

You'll use pre-trained models (OpenAI, SBERT)
Training Transformers is ML Engineering (out of scope)

❌ You will NOT implement self-attention with PyTorch

You'll see concepts and architecture
You will NOT write a backward pass or gradient descent

❌ You will NOT fine-tune models

Fine-tuning is an advanced module (outside this guide)
This module uses out-of-the-box models

❌ You will NOT create custom tokenizers

You'll use tiktoken (OpenAI) and existing tokenizers
Creating tokenizers is a research task

This module is about using embeddings technically, not about training models.

Pedagogical progression

How the learning is structured:

Capsule 02: Transformer Encoders (conceptual architecture)
         ↓
Capsule 03: Tokenization (the first step of the pipeline)
         ↓
Capsule 04: Contextualization (why Transformers > Word2Vec)
         ↓
Capsule 05: Pooling (aggregating tokens)
         ↓
Capsule 06: Normalization (optional final step)
         ↓
Capsule 07: OpenAI API Advanced (parameters and optimizations)
         ↓
Capsule 08: Integrator project (production-ready API client)

Each capsule builds on the previous one. There are no abrupt jumps.

Professional skills you'll develop

By completing this module, you'll be able to:

Explain the technical architecture of embeddings
- Transformers, self-attention, pooling
- At a level that impresses in technical interviews
Debug embedding problems
- Why embeddings don't capture what you expect
- How to optimize tokenization
Write production-ready code
- Batch processing (10x faster)
- Rate limiting (avoid throttling)
- Retry logic (resilience)
Optimize API costs
- Batching reduces calls
- Caching avoids regenerating
- Compressing tokens reduces the price
Make architecture decisions
- OpenAI vs SBERT vs BGE (when to use which)
- Mean pooling vs CLS pooling (trade-offs)
- Normalize or not (depends on the case)

These skills are differentiators in the job market.

Use cases you'll master

This module prepares you to:

1. Optimize existing systems

# Before (naïve):
for doc in documents:
    embedding = get_embedding(doc)  # 1000 API calls 😱

# After (optimized):
embeddings = get_embeddings_batch(documents, batch_size=100)  # 10 API calls ✅

2. Handle errors in production

# Before (fragile):
embedding = get_embedding(text)  # If it fails → crash 😱

# After (robust):
embedding = get_embedding_with_retry(text, max_retries=3)  # Automatic retry ✅

3. Debug unexpected embeddings

# Why is this embedding different from what I expected?
tokens = tokenize(text)  # See how it's tokenized
# → "Ah, it split into 3 sub-words (unexpected)"

4. Compare models with knowledge

# OpenAI: Mean pooling, 1536 dims, $0.00002/1K tokens
# SBERT: Mean pooling, 384 dims, free (local)
# BGE: CLS pooling, 1024 dims, free (local)

# Informed decision based on: cost, dims, pooling

Competitive differentiator

Most AI Engineers:

❌ Use langchain.embeddings as a black box
❌ Don't understand tokenization
❌ Don't optimize batch processing
❌ Don't handle rate limiting correctly

You after this module:

✅ Understand the full architecture
✅ Optimize tokenization and batching
✅ Implement robust retry logic
✅ Write production-ready code

This knowledge sets you apart from 80% of the competition.

Connection with the AI Engineering Learning Path

Where are you in the path?

AI Engineering Learning Path:
├── #5: AI Semantics Guide (completed)
├── #6: Embeddings Deep Dive Guide
│   ├── Module 1: What Are Embeddings? (completed ✅)
│   ├── Module 2: How Do They Work? ← You are here
│   ├── Module 3: Embedding Models (next)
│   └── Modules 4-8: Implementation + Production
└── #7: Vector Databases Guide (future)

This module prepares you for:

Module 3 (Embedding Models):

Compare OpenAI, SBERT, BGE, Instructor
You'll understand technical differences (pooling, dims, training)

Module 4 (Semantic Search from Scratch):

You'll implement a complete search engine
You'll use the pooling and normalization learned here

Module 6 (Production):

Production patterns (batch, cache, lazy loading)
Based on the robust API client from this module

Pedagogical philosophy of this module

Theory/practice balance: 40/60

Theory (40%):
- Transformer architecture
- Conceptual self-attention
- Contextualization

Practice (60%):
- Tokenization with tiktoken (code)
- Pooling with numpy (code)
- Robust API client (project)

More code than Module 1 because you're ready to go deeper.

Appropriate technical depth:

We will NOT do:

❌ Derive the math of attention
❌ Implement a Transformer from scratch with PyTorch
❌ Train models

We WILL do:

✅ Understand conceptually how attention works
✅ Use pre-trained Transformers with code
✅ Implement pooling and normalization manually
✅ Build a production-ready API client

AI Engineering focus (use, not train).

Preparation before starting

1. The right mindset

✅ Yes to:

Going deeper technically (more than Module 1)
Writing code (lots of code in this module)
Experimenting with tiktoken
Building robust systems

❌ No to:

Skipping the code (critical for understanding)
Getting frustrated by math (the minimum necessary)
Expecting it to be "just conceptual" (it's technical)

2. Workspace

# Create a folder for Module 2
mkdir embeddings-module-02
cd embeddings-module-02

# Virtualenv (if you don't have one yet)
python -m venv venv
source venv/bin/activate  # macOS/Linux
# venv\Scripts\activate  # Windows

# Install dependencies
pip install openai numpy python-dotenv tiktoken

3. Verify the setup

# test_setup.py
import openai
import numpy as np
import tiktoken
from dotenv import load_dotenv
import os

load_dotenv()

# Verify the API key
api_key = os.getenv("OPENAI_API_KEY")
if api_key:
    print("✅ OpenAI API key loaded")
else:
    print("❌ OpenAI API key NOT found")

# Verify tiktoken
encoding = tiktoken.encoding_for_model("gpt-4")
tokens = encoding.encode("test")
print(f"✅ tiktoken works ({len(tokens)} tokens)")

# Verify numpy
arr = np.array([1, 2, 3])
print(f"✅ numpy works (array: {arr})")

print("\n🎉 Setup complete!")

python test_setup.py

Module success metrics

By the end of Module 2, you should be able to:

✅ Explain the Transformer architecture at a technical level
✅ Tokenize text with tiktoken and understand the output
✅ Implement pooling strategies with numpy
✅ Normalize embeddings correctly
✅ Use the advanced OpenAI API (batching, error handling)
✅ Build a robust API client that is production-ready

Evidence: You'll be able to complete the mini-project in capsule 08 without assistance.

Optional preparation resources

If you want additional context (optional):

The Illustrated Transformer - Excellent visualization
tiktoken GitHub - OpenAI tokenizer
OpenAI Embeddings Docs - API reference
Sentence-BERT Paper - Mean pooling justification

You don't need to read this now. The capsules are self-contained.

Structure of this module

8 capsules, technical progression:

01. Module Introduction ← You are here
    ↓
02. Transformer Encoders (architecture)
    ↓
03. Tokenization (BPE, tiktoken)
    ↓
04. Contextualization (why context matters)
    ↓
05. Pooling Strategies (mean, CLS, max)
    ↓
06. Normalization (L2 norm)
    ↓
07. OpenAI API Advanced (batching, rate limiting)
    ↓
08. Mini-Project: Robust API Client

Each capsule is technical but accessible.

Summary

Module 2 goes deep into technical architecture.

What you'll learn:

Transformer encoders (self-attention)
Tokenization (BPE, tiktoken)
Contextualization (contextual embeddings)
Pooling strategies (mean, CLS, max)
Normalization (L2 norm)
Advanced OpenAI API (batching, retry)
Production-ready API client

What you will NOT do:

Train models (you'll use pre-trained ones)
Implement Transformers from scratch (conceptual)
Fine-tuning (out of scope)

What you'll build:

A robust API client with retry logic
Batch processing (10x faster)
Rate limiting (avoid throttling)
Production-ready code

Signs of success:

Explain the architecture technically
Optimize API calls
Debug embeddings
Write robust code

Ready to begin?

In the next capsule, you'll dig into:

"Transformer Encoders"

You'll learn:

Encoder-only architecture (BERT-style)
Self-attention mechanisms (conceptual)
Feed-forward layers
Why Transformers > RNNs/LSTMs

You'll also see concrete examples of how self-attention captures context and why it's critical for modern embeddings.

From high-level concepts to deep technical architecture. 🚀

Preparation resources (optional)

If you want to read ahead:

Attention Is All You Need - Original Transformers paper
BERT Explained - Encoder-only architecture
OpenAI Embedding Models - Technical specs

It's not necessary. The capsules cover everything you need.

Module 2 - Embeddings Deep Dive Guide Opening the black box: from text to vector, step by step

← PreviousMini-Project: Your First Embedding - Similarity Calculator Next →Transformer Encoders: The Architecture of Modern Embeddings