Defining Embeddings

Capsule overview

An embedding is a vector (numeric) representation of a piece of data—typically text—that captures its semantic meaning in a high-dimensional space.

In this capsule you'll learn the formal definition of embeddings, how words and phrases are transformed into numeric vectors, what "dimensionality" means, and why this representation is so powerful for AI systems.

You'll also see concrete examples of real embeddings (using OpenAI) and understand how vectors of 1536 numbers can capture the full meaning of a sentence.

What is an embedding?

Simple definition:

An embedding is a numeric vector that represents text in a way that captures its semantic meaning.

Breaking down the definition:

1. Numeric vector: A list of numbers (e.g., [0.023, -0.145, 0.892, ...])
2. Represents text: "Python is a programming language" → [vector of numbers]
3. Captures meaning: Texts with similar meaning have similar vectors

Conceptual visualization:

Original text:
"The cat sleeps on the couch"

           ↓  (Embedding Model)

Vector (1536 dimensions):
[0.0234, -0.1456, 0.8923, -0.2341, 0.5678, ..., 0.1234]
          ^         ^         ^
       dim 1     dim 2     dim 3     ... 1536 dimensions

The model automatically learns what these dimensions mean (they are not human-interpretable).

From text to numbers: The transformation

Why we need a numeric representation:

Computers don't understand text directly:

# ❌ The computer can't "compute" with this:
text_1 = "dog"
text_2 = "cat"

# How similar are they? There's no math operation to tell

Computers DO understand numbers:

# ✅ With embeddings:
embedding_dog = [0.23, 0.45, -0.12, ...]   # 1536 numbers
embedding_cat = [0.25, 0.43, -0.10, ...]   # 1536 numbers

# Now you CAN compute similarity:
similarity = cosine_similarity(embedding_dog, embedding_cat)
# → 0.85 (very similar because both are domestic animals)

Properties of an embedding

1. Fixed dimensionality

All embeddings from the same model have the same dimension:

# OpenAI text-embedding-3-small = 1536 dimensions
embedding_short = embed("Hi")
embedding_long = embed("This is a very long sentence with many words...")

print(len(embedding_short))  # 1536
print(len(embedding_long))   # 1536  ← Same size

Text length doesn't matter: 1 word or 1000 words → same vector size.

2. Dense representation

An embedding is a "dense" vector (almost all values are non-zero):

# Typical embedding (simplified to 5 dimensions):
[0.234, -0.156, 0.893, -0.234, 0.567]
  ^       ^       ^       ^       ^
 Non-zero Non-zero Non-zero Non-zero Non-zero

Contrast with "sparse" representations (mostly zeros):

# One-hot encoding (sparse):
[0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]  # Only 1 value != 0

Advantage: Dense embeddings capture much more information.

3. Semantic capture

The key: similar texts → similar vectors

# Embeddings (conceptual):
embed("king") = [0.5, 0.3, 0.8, ...]
embed("queen") = [0.5, 0.3, 0.7, ...]  ← Very similar
embed("monarch") = [0.4, 0.3, 0.8, ...]  ← Similar
embed("pizza") = [-0.2, 0.9, -0.1, ...]  ← Very different

Semantically related texts are "close" in the vector space.

Dimensionality: Why 1536 dimensions?

Comparison of dimensionalities:

Trade-off: More dimensions = more information captured, but higher computational cost.

What do those dimensions represent?

Honest answer: We don't know exactly.

The model automatically learns what each dimension captures:

• Dimension 1 might capture the "formality" of the text
• Dimension 2 might capture "technical vs conversational topic"
• Dimension 500 might capture "positive/negative sentiment"
• ...

But we can't manually inspect and say "this dimension means X".

What matters: They work empirically. You don't need to interpret them.

Real example with OpenAI

Basic code to generate embeddings:

from openai import OpenAI
import os
from dotenv import load_dotenv

# Setup
load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

# Generate embedding
response = client.embeddings.create(
    model="text-embedding-3-small",
    input="Python is a programming language"
)

# Extract vector
embedding = response.data[0].embedding

# Inspect
print(f"Dimensions: {len(embedding)}")
print(f"First 10 values: {embedding[:10]}")

Output (real example):

Dimensions: 1536
First 10 values: [0.0234, -0.1456, 0.8923, -0.2341, 0.5678, -0.3421, 0.1234, -0.6789, 0.4321, 0.9876]

Comparing embeddings of similar texts:

# 3 related texts
texts = [
    "Python is a programming language",
    "Python is a language for programming",
    "JavaScript is a programming language"
]

# Generate embeddings
embeddings = []
for text in texts:
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    embeddings.append(response.data[0].embedding)

# The first 5 values of each embedding:
print("Text 1:", embeddings[0][:5])
print("Text 2:", embeddings[1][:5])
print("Text 3:", embeddings[2][:5])

Output (conceptual example):

Text 1: [0.023, -0.145, 0.892, -0.234, 0.567]
Text 2: [0.025, -0.143, 0.895, -0.230, 0.570]  ← Very similar to Text 1
Text 3: [0.012, -0.120, 0.750, -0.190, 0.490]  ← Somewhat similar (both are languages)

Notice: Text 1 and Text 2 are almost identical (only "language" → "language for programming" changes), and their embeddings are very similar.

Embeddings vs other representations

1. One-Hot Encoding (legacy)

# Vocabulary: ["dog", "cat", "house"]
one_hot_dog = [1, 0, 0]
one_hot_cat = [0, 1, 0]
one_hot_house = [0, 0, 1]

# Problem: "dog" and "cat" are just as different from each other as from "house"
# It doesn't capture that "dog" and "cat" are animals (related)

Limitations:

• ❌ Doesn't capture semantic similarity
• ❌ Size = vocabulary size (inefficient)
• ❌ Doesn't handle new words (out-of-vocabulary)

2. TF-IDF (keyword-based)

# TF-IDF: Term Frequency × Inverse Document Frequency
tfidf_doc = [0.5, 0.0, 0.3, 0.0, 0.8, 0.0, ...]  # Sparse (many 0s)

# Captures word importance, but NOT meaning

Limitations:

• ❌ Doesn't understand synonyms ("car" ≠ "automobile")
• ❌ Doesn't understand context ("bank" = riverbank vs financial bank)
• ⚠️ Useful for keyword matching (BM25), not for semantic search

3. Embeddings (modern)

# Embeddings: A dense vector that captures meaning
embedding = [0.023, -0.145, 0.892, ..., 0.567]  # 1536 values

# ✅ Captures semantic similarity
# ✅ Understands synonyms (car ≈ automobile)
# ✅ Understands context (different "bank" → different embeddings)

Advantages:

• ✅ Captures deep semantic meaning
• ✅ Handles synonyms, paraphrases
• ✅ Fixed size independent of vocabulary

Types of embeddings

1. Word Embeddings (legacy)

One embedding per word:

embed("king") = [0.5, 0.3, 0.8]
embed("queen") = [0.5, 0.3, 0.7]

Problem: Doesn't capture context.

# "bank" has multiple meanings:
embed("bank")  # Riverbank or financial institution?
# → Same embedding for both contexts ❌

Examples: Word2Vec, GloVe (pre-2018)

2. Contextual Embeddings (modern)

One embedding per full sentence:

embed("I'm going to the river bank") = [0.2, 0.5, 0.1, ...]
embed("I'm going to the bank to withdraw money") = [0.8, -0.3, 0.6, ...]
                                        ↑ Different because the context is different

Advantage: Same word → different embeddings depending on context.

Examples: BERT, GPT, OpenAI embeddings, Sentence-BERT (post-2018)

How are embeddings generated?

High-level flow (we go deeper in Module 2):

Text:
"Python is a programming language"

           ↓ (1) Tokenization

Tokens:
["Python", "is", "a", "programming", "language"]

           ↓ (2) Transformer Encoder

Hidden states:
[[0.1, 0.2, ...], [0.3, 0.4, ...], [0.5, 0.6, ...], ...]
 (each token has its own intermediate vector)

           ↓ (3) Pooling (mean, CLS)

Final embedding:
[0.023, -0.145, 0.892, ..., 0.567]  (1536 dims)

Critical step: Pooling reduces multiple vectors (one per token) into a single vector (the sentence embedding).

Module 2 will go deeper into each step.

Example: Conceptual 2D visualization

In reality: 1536 dimensions (impossible to visualize)

But we can reduce to 2D to illustrate:

       High tech
              ^
              |
        "Python" ●
              |    "JavaScript" ●
              |
─────────────────────────────────────> Formality
              |
         "Dog" ●    "Cat" ●
              |
              v
         Animals

Conceptual:

• "Python" and "JavaScript" are close (both languages)
• "Dog" and "Cat" are close (both animals)
• "Python" and "Dog" are far apart (different domains)

In reality: A 1536-dimensional space captures MANY more relationships.

Mathematical properties of embeddings

1. Embeddings as points in space

# Each embedding is a point in 1536-dimensional space:
embedding_1 = [x1, x2, x3, ..., x1536]  # Point in R^1536
embedding_2 = [y1, y2, y3, ..., y1536]  # Another point

Distance between points = semantic similarity

2. Vector operations

Sum, subtraction, average:

import numpy as np

# Average of embeddings:
avg_embedding = np.mean([emb_1, emb_2, emb_3], axis=0)
# → Captures the "average meaning" of multiple texts

Semantic arithmetic (conceptual):

embed("king") - embed("man") + embed("woman") ≈ embed("queen")

Note: This arithmetic is more effective with word embeddings (Word2Vec). Modern sentence embeddings are more complex.

Immutability of embeddings

Critical property:

# Same text → ALWAYS the same embedding (same model)
emb_1 = embed("Python is great")
emb_2 = embed("Python is great")

assert emb_1 == emb_2  # ✅ True (deterministic)

Advantage: You can cache embeddings.

# Embed once, save, reuse:
embeddings = {}
embeddings["doc_1"] = embed("Content of document 1")
embeddings["doc_2"] = embed("Content of document 2")

# Save to file:
np.save("embeddings.npy", embeddings)

# Load later (no need to recalculate):
embeddings = np.load("embeddings.npy", allow_pickle=True).item()

Implication: Generating embeddings is expensive (API call), but you only do it once.

Analogies to understand embeddings

Analogy 1: GPS coordinates

"Paris" → (48.8566, 2.3522)  # Latitude, Longitude (2D)
"London" → (51.5074, -0.1278)

# Geographically close cities → close coordinates
# Paris and London are relatively close in Europe

Embeddings:
"Paris" → [0.2, 0.5, -0.3, ..., 0.8]  (1536D)
"London" → [0.1, 0.6, -0.4, ..., 0.7]
# Conceptually close cities → close embeddings

Analogy 2: Genetic DNA

Human DNA = a sequence of 3 billion base pairs
→ Defines biological characteristics

Embedding = a sequence of 1536 numbers
→ Defines the semantic characteristics of the text

Genetic similarity → related species Vector similarity → related texts

Analogy 3: Fingerprint

Fingerprint = a unique pattern that identifies a person

Embedding = a "semantic fingerprint" that identifies the meaning of the text

Same person → same fingerprint (invariant) Same text → same embedding (invariant)

Limitations of embeddings

1. They're not perfect

# Sometimes similar texts have slightly distant embeddings
embed("The cat sleeps") vs embed("The feline rests")
# → Similar but not identical (because the words are different)

Solution: Larger models (OpenAI large) capture better, but are more costly.

2. Model dependency

# Different models → different embeddings
openai_emb = embed_openai("Python")
sbert_emb = embed_sbert("Python")

# You CANNOT compare them directly:
similarity(openai_emb, sbert_emb)  # ❌ Meaningless (different spaces)

Solution: Always use the same model.

3. Generation cost

# OpenAI API: $0.00002 / 1K tokens
# 1 million documents × 500 tokens average = $10 USD

# It's not free, but it's one-time (then you cache)

Solution: Batch processing + caching (Module 6).

Exercises

Exercise 1: Identify embeddings

Which of these is a valid embedding?

A:

[1, 0, 0, 0, 0, 0]  # 6 dimensions, sparse (mostly 0s)

B:

[0.023, -0.145, 0.892, -0.234, 0.567, 0.123]  # 6 dims, dense

C:

["Python", "programming", "language"]  # List of words

Exercise 2: Predict similarity

Given these texts, which should have the most similar embeddings?

A: "The dog barks" B: "The hound howls" C: "The pizza is delicious"

Exercise 3: Fix the code

What's wrong here?

from openai import OpenAI
client = OpenAI()

# Generate embeddings for 2 texts
embedding_1 = client.embeddings.create(
    model="text-embedding-3-small",
    input="Python"
)

embedding_2 = client.embeddings.create(
    model="text-embedding-3-large",  # ← Different model
    input="JavaScript"
)

# Calculate similarity
similarity = cosine_similarity(embedding_1, embedding_2)

# Use the SAME model for both:
embedding_1 = client.embeddings.create(
    model="text-embedding-3-small",  # ← Same
    input="Python"
).data[0].embedding

embedding_2 = client.embeddings.create(
    model="text-embedding-3-small",  # ← Same
    input="JavaScript"
).data[0].embedding

# Now you CAN compare:
similarity = cosine_similarity(embedding_1, embedding_2)

Exercise 4: Debugging

Why does this code give an error?

import numpy as np

embedding = [0.1, 0.2, 0.3]
similarity = np.dot(embedding, embedding)  # Error?

similarity = np.dot(embedding, embedding)
# → 0.14  (dot product of a vector with itself)

from numpy.linalg import norm

embedding = np.array([0.1, 0.2, 0.3])
similarity = np.dot(embedding, embedding) / (norm(embedding) * norm(embedding))
# → 1.0 (always)

emb_1 = np.array([0.1, 0.2, 0.3])
emb_2 = np.array([0.2, 0.3, 0.4])

similarity = np.dot(emb_1, emb_2) / (norm(emb_1) * norm(emb_2))
# → ~0.998 (very similar but not identical)

Summary

What you learned:

• ✅ Definition: Embedding = numeric vector that captures semantic meaning
• ✅ Transformation: Text → Tokenization → Transformer → Pooling → Embedding
• ✅ Dimensionality: Typically 300-3072 dimensions (OpenAI: 1536)
• ✅ Properties: Dense, fixed size, deterministic, captures semantics
• ✅ Advantages vs legacy: Better than one-hot, TF-IDF for semantic search
• ✅ Types: Word embeddings (legacy) vs Contextual embeddings (modern)
• ✅ Limitations: Not perfect, model-dependent, generation cost

Key concepts:

1. Similar texts → similar embeddings (proximity = similarity)
2. Embedding space of N dimensions (e.g., 1536D)
3. Same text → same embedding (deterministic, cacheable)

Additional resources

1. OpenAI Embeddings Guide - Official docs
2. What are Embeddings? (Video) - Visual explanation
3. Word Embeddings (Paper) - Original Word2Vec (legacy but historic)
4. BERT Explained - Contextual embeddings
5. Sentence Transformers - Open-source sentence embeddings
6. Embeddings at Scale - Production applications

In the next capsule

Capsule 03: Vector Properties

You'll learn:

• Semantic similarity (cosine similarity)
• Directionality in vector space
• Magnitude vs direction (which one matters)
• Natural clustering of embeddings

From conceptual definition to mathematical properties.

Module 1 - Embeddings Deep Dive Guide Turning text into vectors that machines understand