Architecture Overview: How Embeddings Are Generated

Capsule overview

You've used embeddings as "black boxes" (text → vector), but how are those vectors actually generated internally?

In this capsule you'll learn the high-level architecture behind modern embedding models: encoder-only Transformers, tokenization, self-attention (conceptual), and pooling strategies. We won't dive into complex math—the goal is for you to understand the end-to-end flow and be able to make informed decisions as an AI Engineer.

You'll also see practical code with tiktoken (OpenAI's tokenizer) and understand why different models produce different embeddings.

The full flow: Text → Embedding

4-step pipeline:

Text:
"Python is a programming language"

        ↓ (1) Tokenization

Tokens:
["Python", "is", "a", "programming", "language"]

        ↓ (2) Transformer Encoder

Hidden states (one vector per token):
[[0.1, 0.2, ...], [0.3, 0.4, ...], [0.5, 0.6, ...], ...]
 ↑ Python          ↑ is             ↑ a

        ↓ (3) Pooling

Final embedding (a single vector):
[0.023, -0.145, 0.892, ..., 0.567]  (1536 dims)

        ↓ (4) Normalization (optional)

Normalized embedding:
[0.014, -0.089, 0.548, ..., 0.348]  (magnitude = 1.0)

Each step has a specific purpose. Let's look at each one in detail.

Step 1: Tokenization

What it is:

Splitting text into "tokens" (sub-words or words) that the model understands.

Why not process characters directly?

• Character vocabulary = ~100 (a-z, A-Z, 0-9, symbols)
• Word vocabulary = millions (inefficient)
• Tokens (sub-words) = the perfect balance (~50K-100K tokens)

Example with tiktoken (OpenAI):

import tiktoken

# Load OpenAI's tokenizer
encoding = tiktoken.encoding_for_model("gpt-4")

# Tokenize text
text = "Python is a programming language"
tokens = encoding.encode(text)

print(f"Text: {text}")
print(f"Tokens (IDs): {tokens}")
print(f"Count: {len(tokens)} tokens")

# Decode individual tokens
for token_id in tokens:
    token_str = encoding.decode([token_id])
    print(f"  Token {token_id}: '{token_str}'")

Output:

Text: Python is a programming language
Tokens (IDs): [31380, 374, 264, 15840, 4221]
Count: 5 tokens

  Token 31380: 'Python'
  Token 374: ' is'
  Token 264: ' a'
  Token 15840: ' programming'
  Token 4221: ' language'

Notice: Each word → 1 token (many common English words are 1 token).

Tokenization of complex words:

# Long words can be multiple tokens
text = "antidisestablishmentarianism"
tokens = encoding.encode(text)

print(f"Text: {text}")
print(f"Tokens: {len(tokens)}")

for token_id in tokens:
    print(f"  '{encoding.decode([token_id])}'")

Output:

Text: antidisestablishmentarianism
Tokens: 6

  'anti'
  'dis'
  'establish'
  'ment'
  'arian'
  'ism'

Advantage of sub-words: The model can understand words it has never seen (it composes them from sub-parts).

Different tokenizers:

BPE (Byte Pair Encoding): An algorithm that learns the most frequent sub-words.

Step 2: Transformer Encoder

What it is:

A neural network that turns tokens into contextual vectors (hidden states).

High-level architecture:

Input tokens:
[Python, is, a, language]

        ↓ (Embedding lookup)

Token embeddings (initial):
[[0.1, 0.2, ...], [0.3, 0.4, ...], ...]

        ↓ (Self-Attention Layers × N)

Contextualized embeddings:
[[0.5, 0.3, ...], [0.7, 0.2, ...], ...]
 ↑ Now "Python" knows it goes with "language"

        ↓ (Feed-Forward Layers)

Hidden states (final):
[[0.8, 0.1, ...], [0.9, 0.3, ...], ...]

Key: Self-attention lets each token "see" all the other tokens.

Self-Attention (conceptual):

Text: "The river bank"

Without context (legacy word embeddings):
bank → [0.5, 0.3, 0.8]  (always the same vector)

With self-attention (contextual embeddings):
"The river bank"
         ↑
         bank sees "river" → embedding_A = [0.2, 0.5, 0.1] (riverbank)

"I went to the bank to withdraw money"
              ↑
              bank sees "money" → embedding_B = [0.8, 0.3, 0.6] (financial)

Self-attention captures that "bank" + "river" ≠ "bank" + "money".

Attention visualization:

# Conceptual (not executable without a loaded model)
text = "Python is a language"

# Attention matrix (simplified):
#           Python    is      a     language
# Python    1.0       0.2     0.1   0.8      ← "Python" attends to "language"
# is        0.2       1.0     0.7   0.1      ← "is" attends to "a"
# a         0.1       0.7     1.0   0.3
# language  0.8       0.1     0.3   1.0      ← "language" attends to "Python"

High values → related tokens.

Note: This matrix is learned automatically during training (you don't define it manually).

Step 3: Pooling

What it is:

Reducing multiple hidden states (one per token) into A SINGLE vector (the sentence embedding).

Problem:

Tokens: ["Python", "is", "a", "language"]

Hidden states (one per token):
[
  [0.8, 0.1, 0.3, ...],  ← Python
  [0.9, 0.3, 0.2, ...],  ← is
  [0.7, 0.2, 0.4, ...],  ← a
  [0.6, 0.4, 0.1, ...]   ← language
]

How do we get A SINGLE vector for the whole sentence?

Solution: Pooling.

Pooling strategies:

1. Mean Pooling (average)

import numpy as np

# Hidden states (simplified to 3 dims)
hidden_states = np.array([
    [0.8, 0.1, 0.3],  # Python
    [0.9, 0.3, 0.2],  # is
    [0.7, 0.2, 0.4],  # a
    [0.6, 0.4, 0.1]   # language
])

# Mean pooling
embedding = np.mean(hidden_states, axis=0)
print(f"Mean pooling: {embedding}")
# → [0.75, 0.25, 0.25]  (average of each dimension)

Advantage: Considers all tokens equally.

Used by: Sentence-BERT, Instructor Embeddings.

2. CLS Pooling (special token)

# BERT adds a special [CLS] token at the start:
tokens = ["[CLS]", "Python", "is", "a", "language"]

# Hidden states:
hidden_states = [
    [0.5, 0.6, 0.7],  # [CLS] ← We use ONLY this one
    [0.8, 0.1, 0.3],  # Python
    [0.9, 0.3, 0.2],  # is
    [0.7, 0.2, 0.4],  # a
    [0.6, 0.4, 0.1]   # language
]

# CLS pooling: Take ONLY the first hidden state
embedding = hidden_states[0]
print(f"CLS pooling: {embedding}")
# → [0.5, 0.6, 0.7]

Advantage: The [CLS] token learned to "summarize" the whole sentence during training.

Used by: Original BERT, some fine-tuned BERT models.

3. Max Pooling (max per dimension)

# Max pooling: Take the maximum value of each dimension
embedding = np.max(hidden_states, axis=0)
print(f"Max pooling: {embedding}")
# → [0.9, 0.4, 0.4]  (max of each column)

Advantage: Captures the most "salient" features.

Used by: Less common, some specialized models.

Comparison of pooling strategies:

Best practice: Mean pooling for sentence embeddings (better balance).

Step 4: Normalization (optional)

What it is:

Scaling the embedding to magnitude = 1.0 (unit vector).

import numpy as np

# Unnormalized embedding
embedding = np.array([3.0, 4.0])
print(f"Original: {embedding}")
print(f"Magnitude: {np.linalg.norm(embedding)}")  # 5.0

# Normalize (L2 normalization)
embedding_normalized = embedding / np.linalg.norm(embedding)
print(f"Normalized: {embedding_normalized}")
print(f"Magnitude: {np.linalg.norm(embedding_normalized)}")  # 1.0

Output:

Original: [3. 4.]
Magnitude: 5.0
Normalized: [0.6 0.8]
Magnitude: 1.0

Why normalize?

Advantage: Dot product = Cosine similarity (more efficient).

# With normalized embeddings:
dot_product = np.dot(emb_a_normalized, emb_b_normalized)
# → dot_product == cosine_similarity(emb_a, emb_b) ✅

# Without normalizing (less efficient):
cosine_sim = np.dot(emb_a, emb_b) / (np.linalg.norm(emb_a) * np.linalg.norm(emb_b))

Models that normalize:

• ✅ Sentence-BERT (normalized by default)
• ❌ OpenAI embeddings (NOT normalized by default)

Architectures of popular models

1. OpenAI text-embedding-3-small

Architecture: Encoder-only Transformer

Specs:
- Layers: 12 transformer layers
- Hidden size: 1536
- Attention heads: 12
- Vocabulary: ~100K tokens (tiktoken)
- Pooling: Not documented (probably mean)
- Normalization: NO (manually if you want)

Training:
- Contrastive learning (similar/dissimilar pairs)
- Dataset: Massive text corpus (web, books, code)

Code (full conceptual flow):

# Step 1: Tokenization
tokens = tiktoken.encode(text)

# Step 2: Transformer encoder (OpenAI internal)
hidden_states = transformer_encoder(tokens)

# Step 3: Pooling (mean)
embedding = mean_pooling(hidden_states)

# Step 4: Return (without normalizing)
return embedding  # [1536 dims]

2. Sentence-BERT (all-MiniLM-L6-v2)

Architecture: BERT fine-tuned for sentence embeddings

Specs:
- Layers: 6 transformer layers (MiniLM = lightweight)
- Hidden size: 384
- Attention heads: 12
- Vocabulary: ~30K tokens (WordPiece)
- Pooling: Mean pooling (explicit)
- Normalization: YES (automatic)

Training:
- Siamese network (pairs of similar sentences)
- Dataset: NLI, STS, QA pairs

Code (real SBERT):

from sentence_transformers import SentenceTransformer

model = SentenceTransformer('all-MiniLM-L6-v2')

# Internal flow:
# 1. Tokenization (WordPiece)
# 2. BERT encoder (6 layers)
# 3. Mean pooling
# 4. L2 normalization

embedding = model.encode("Python is a language")
# → [384 dims], normalized

3. BGE (bge-large-en-v1.5)

Architecture: BERT-large fine-tuned

Specs:
- Layers: 24 transformer layers
- Hidden size: 1024
- Attention heads: 16
- Vocabulary: ~30K tokens
- Pooling: CLS token
- Normalization: Optional

Training:
- Contrastive learning + hard negatives
- Dataset: C-MTEB (Chinese + English)

Key difference: BGE uses CLS pooling (vs SBERT's mean pooling).

Practical example: End-to-end flow

Generating embeddings step by step (conceptual with OpenAI):

from openai import OpenAI
import tiktoken
import os
from dotenv import load_dotenv

load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

text = "Python is a programming language"

# Step 1: Tokenization (manual with tiktoken)
encoding = tiktoken.encoding_for_model("text-embedding-3-small")
tokens = encoding.encode(text)

print("=== STEP 1: TOKENIZATION ===")
print(f"Text: {text}")
print(f"Tokens (IDs): {tokens}")
print(f"Count: {len(tokens)} tokens\n")

for token_id in tokens:
    token_str = encoding.decode([token_id])
    print(f"  Token {token_id}: '{token_str}'")

# Steps 2-4: Transformer + Pooling (OpenAI internal)
print("\n=== STEPS 2-4: TRANSFORMER + POOLING ===")
print("(OpenAI internal processing)\n")

# API call (does steps 2-4 internally)
response = client.embeddings.create(
    model="text-embedding-3-small",
    input=text
)

embedding = response.data[0].embedding

print("=== FINAL RESULT ===")
print(f"Embedding dimensions: {len(embedding)}")
print(f"First 10 values: {embedding[:10]}")
print(f"Last 10 values: {embedding[-10:]}")

# Check whether it's normalized
import numpy as np
embedding_array = np.array(embedding)
magnitude = np.linalg.norm(embedding_array)
print(f"\nMagnitude: {magnitude:.4f}")
if magnitude < 1.1:
    print("✅ Normalized embedding (magnitude ~1.0)")
else:
    print("❌ Embedding NOT normalized")

Output:

=== STEP 1: TOKENIZATION ===
Text: Python is a programming language
Tokens (IDs): [31380, 374, 264, 15840, 4221]
Count: 5 tokens

  Token 31380: 'Python'
  Token 374: ' is'
  Token 264: ' a'
  Token 15840: ' programming'
  Token 4221: ' language'

=== STEPS 2-4: TRANSFORMER + POOLING ===
(OpenAI internal processing)

=== FINAL RESULT ===
Embedding dimensions: 1536
First 10 values: [0.023, -0.145, 0.892, -0.234, 0.567, -0.342, 0.123, -0.678, 0.432, 0.987]
Last 10 values: [0.234, -0.567, 0.890, -0.123, 0.456, -0.789, 0.012, -0.345, 0.678, 0.901]

Magnitude: 1.523
❌ Embedding NOT normalized

Differences between models

Why different models → different embeddings:

Result: You CAN'T compare embeddings from different models (different vector spaces).

Architectural limitations

1. Token limit (context window):

# OpenAI text-embedding-3-small: Max 8191 tokens
long_text = "..." * 10000  # Very long text

tokens = encoding.encode(long_text)
print(f"Tokens: {len(tokens)}")

if len(tokens) > 8191:
    print("❌ Text exceeds the limit. You must truncate or chunk it.")

Solution: Chunk long text (Module 4).

2. Loss of information with pooling:

# Text: "Python is good but JavaScript is better"

# Mean pooling averages EVERYTHING:
# → Loses the nuance of "but" (contrast)

# Alternative: Chunk into 2 sentences:
# Chunk 1: "Python is good"
# Chunk 2: "JavaScript is better"
# → Captures the nuances of each sentence

3. Sensitivity to order:

# Embeddings ARE sensitive to order (self-attention captures it):
text_1 = "The dog chases the cat"
text_2 = "The cat chases the dog"

emb_1 = get_embedding(text_1)
emb_2 = get_embedding(text_2)

# Similarity ~0.85 (high but NOT identical)
# Captures that they're different (order matters)

Exercises

Exercise 1: Tokenize with tiktoken

Tokenize this text and count how many tokens it generates:

text = "antidisestablishmentarianism is a long word"

# How many tokens?

import tiktoken

encoding = tiktoken.encoding_for_model("gpt-4")
tokens = encoding.encode(text)

print(f"Text: {text}")
print(f"Token count: {len(tokens)}")
print("\nIndividual tokens:")
for token_id in tokens:
    print(f"  '{encoding.decode([token_id])}'")

Text: antidisestablishmentarianism is a long word
Token count: 10

Individual tokens:
  'anti'
  'dis'
  'establish'
  'ment'
  'arian'
  'ism'
  ' is'
  ' a'
  ' long'
  ' word'

Exercise 2: Implement mean pooling

Implement mean pooling manually:

import numpy as np

# Hidden states (4 tokens × 3 dims)
hidden_states = np.array([
    [0.8, 0.1, 0.3],
    [0.9, 0.3, 0.2],
    [0.7, 0.2, 0.4],
    [0.6, 0.4, 0.1]
])

# Implement mean pooling

def mean_pooling(hidden_states):
    """
    Mean pooling: The average of all hidden states
    """
    return np.mean(hidden_states, axis=0)

embedding = mean_pooling(hidden_states)
print(f"Embedding: {embedding}")

Embedding: [0.75 0.25 0.25]

Exercise 3: Normalize an embedding

Normalize this embedding to magnitude 1.0:

embedding = np.array([6.0, 8.0])

# Normalize to magnitude = 1.0

def normalize_embedding(embedding):
    """L2 normalization"""
    return embedding / np.linalg.norm(embedding)

embedding = np.array([6.0, 8.0])
embedding_normalized = normalize_embedding(embedding)

print(f"Original: {embedding}")
print(f"Original magnitude: {np.linalg.norm(embedding)}")
print(f"\nNormalized: {embedding_normalized}")
print(f"Normalized magnitude: {np.linalg.norm(embedding_normalized)}")

Original: [6. 8.]
Original magnitude: 10.0

Normalized: [0.6 0.8]
Normalized magnitude: 1.0

Summary

What you learned:

• ✅ Full flow: Tokenization → Transformer → Pooling → Normalization
• ✅ Tokenization: tiktoken (BPE), WordPiece (BERT)
• ✅ Transformer: Self-attention captures context
• ✅ Pooling: Mean (SBERT), CLS (BERT), Max (rare)
• ✅ Normalization: L2 norm for magnitude = 1.0
• ✅ Model differences: Architecture, training data, pooling

Key concepts:

1. Self-attention enables contextual embeddings (vs legacy word embeddings)
2. Pooling reduces multiple tokens → 1 embedding
3. Different models → different vector spaces (incomparable)

Additional resources

1. Illustrated Transformer - Excellent visualization
2. BERT Explained - BERT architecture
3. Sentence-BERT Paper - Mean pooling justified
4. tiktoken GitHub - OpenAI tokenizer
5. Attention Is All You Need - Original Transformer

In the next capsule

Capsule 08: Mini-Project - Your First Embedding

You'll build:

• A similarity calculator CLI
• OpenAI API setup
• Embed 10 documents
• Calculate the top-3 similarities
• Complete production-ready code

From architectural theory to practical implementation.

Module 1 - Embeddings Deep Dive Guide Understanding the magic behind the vectors