4. Embeddings: Vector Representations of Meaning

Description

Embeddings are vector representations (numbers) of text that capture semantic meaning. They're fundamental for:

• Semantic search (searching by meaning, not just keywords).
• RAG (Retrieval-Augmented Generation) (searching for relevant documents for an LLM).
• Clustering (grouping similar texts).
• Classification (e.g. sentiment analysis).

What you'll learn:

• What embeddings are (vectors of meaning).
• How they work (similar words → similar vectors).
• Operations with vectors (cosine similarity, arithmetic).
• Practical applications (semantic search, RAG).

---

What Embeddings are

Embedding: A vector (a list of numbers) that represents the meaning of a text.

Example (simplified):

"king" → [0.5, 0.8, 0.1, 0.3, ...]  (e.g. 768 dimensions)
"queen" → [0.52, 0.78, 0.12, 0.32, ...]
"man" → [0.1, 0.2, 0.9, 0.1, ...]
"woman" → [0.12, 0.22, 0.88, 0.11, ...]

Key point: Words with similar meaning have nearby vectors in the vector space.

---

How Embeddings are Generated

Embedding models

Examples:

• OpenAI: text-embedding-ada-002 (1,536 dimensions).
• Sentence Transformers: all-MiniLM-L6-v2 (384 dimensions).
• Cohere: embed-english-v3.0 (1,024 dimensions).

Process:

1. You pass text (e.g. "Hello world") to the model.
2. The model generates an N-dimensional vector.

Code (OpenAI):

import openai

response = openai.embeddings.create(
    model="text-embedding-ada-002",
    input="Hello world"
)

embedding = response.data[0].embedding
print(len(embedding))  # 1536 dimensions

---

Cosine Similarity: Measuring Closeness

Cosine similarity: A metric that measures how similar two vectors are (range: -1 to 1).

• 1.0: Identical (the same meaning).
• 0.0: Orthogonal (unrelated).
• -1.0: Opposite.

Formula:

similarity = (A · B) / (||A|| × ||B||)

Example:

from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

vec_king = embedding_model("king")
vec_queen = embedding_model("queen")
vec_car = embedding_model("car")

print(cosine_similarity([vec_king], [vec_queen]))  # ~0.85 (very similar)
print(cosine_similarity([vec_king], [vec_car]))    # ~0.2 (barely related)

---

Embedding Arithmetic

The famous example:

king - man + woman ≈ queen

Interpretation: If you subtract "masculinity" (man) from "king" and add "femininity" (woman), you get "queen".

Code (conceptual):

vec_result = vec_king - vec_man + vec_woman
# Look for the word closest to vec_result
# → "queen" is the closest one

Other examples:

Paris - France + Italy ≈ Rome
walking - walk + swim ≈ swimming

Moral: Embeddings capture semantic relationships (gender, geography, verb tense).

---

Application 1: Semantic Search

Problem: Searching documents by meaning, not just keywords.

Example:

Query: "How to fix a leaky faucet"
Documents:
1. "Repairing dripping taps" (it contains neither "leaky" nor "faucet", but it's relevant)
2. "Installing kitchen sinks"
3. "Painting walls"

Keyword search (traditional): It doesn't find document 1 (it doesn't have the exact keywords).

Semantic search (with embeddings):

1. Generate an embedding of the query.
2. Generate embeddings of all the documents.
3. Compute the cosine similarity between the query and each document.
4. Rank by similarity.

Result: Document 1 has high similarity (the same meaning even with different words).

---

Application 2: RAG (Retrieval-Augmented Generation)

Problem: The LLM has outdated knowledge or doesn't know information specific to your domain.

Solution (RAG):

1. Index the documents: Generate embeddings of your documents (e.g. internal company PDFs).
2. Query: The user asks "What is the vacation policy?"
3. Search: Generate an embedding of the query, look for the most similar documents (semantic search).
4. Generate: Pass the relevant documents + the query to the LLM → the LLM generates an answer based on the documents.

Advantage: The LLM has access to up-to-date/specific information without retraining.

---

Application 3: Clustering

Problem: Grouping 10,000 product reviews into topics.

Process:

1. Generate embeddings of each review.
2. Apply K-means clustering in the embedding space.
3. Identify the clusters (e.g. Cluster 1: quality, Cluster 2: price, Cluster 3: shipping).

---

Why this matters for an AI Engineer

1. RAG is a key technique

Almost every enterprise LLM application uses RAG:

• A support chatbot (searching in a knowledge base).
• Document analysis (searching for relevant sections).
• Q&A over internal documents.

Without understanding embeddings: You can't implement RAG effectively.

---

2. Search optimization

Problem: Semantic search across 1M documents is slow (computing similarity with 1M vectors).

Solution: Vector databases (Pinecone, Weaviate, Qdrant) optimize search with ANN (Approximate Nearest Neighbors).

Result: Search in milliseconds (vs seconds with linear search).

---

3. Choosing an embedding model

Trade-offs:

Model	Dimensions	Quality	Cost	Speed
OpenAI ada-002	1,536	Very good	$0.0001/1K tokens	Fast (API)
Sentence Transformers	384	Good	Free (local)	Very fast (local)
Cohere embed-v3	1,024	Excellent	$0.0001/1K tokens	Fast (API)

Decision: It depends on budget, required quality, latency.

---

Common Mistakes

1. Comparing embeddings from different models

Mistake: Generating an embedding with OpenAI, comparing it with an embedding from Sentence Transformers.

Problem: Different models → different vector spaces → the similarity is meaningless.

Solution: Use the same model to generate all the embeddings.

---

2. Not normalizing the vectors

Mistake: Comparing vectors without normalizing them.

Problem: Cosine similarity assumes normalized vectors (length 1).

Solution: Normalize before comparing (libraries do it automatically).

---

Summary

Embeddings:

• Vectors (lists of numbers) that represent the meaning of text.
• Similar words → nearby vectors.

Cosine similarity:

• A metric for measuring closeness between vectors (0-1).
• High similarity → similar meanings.

Applications:

• Semantic search: Searching by meaning.
• RAG: An LLM + a knowledge base.
• Clustering: Grouping similar texts.

Why it matters:

• RAG is a key technique for enterprise applications.
• Vector databases optimize search.
• Model choice affects quality, cost, latency.

Next step: Lesson 05: Context Window — LLMs' memory limit, why it exists, solutions.