Vector Properties of Embeddings

Capsule overview

Embeddings are not just arbitrary lists of numbers—they are vectors with specific mathematical properties that make them incredibly useful for AI systems.

In this capsule you'll learn the key vector properties of embeddings: semantic similarity (how "close" two vectors are), directionality (what direction represents in the space), and magnitude vs direction (which matters more for semantic search).

You'll also see practical code using numpy to calculate cosine similarity and understand why it's the standard metric in AI.

Embeddings as vectors in space

Reminder: A vector is an arrow in space

2D Vector (visualizable):
      ^
      | (3, 4)  ← End point
      |     /
      |   /
      | /
      └────────>
   (0,0) Origin

An embedding is a vector in a high-dimensional space:

# Embedding = a 1536-dimensional vector
embedding = [0.023, -0.145, 0.892, ..., 0.567]
            └─────────────────────────────────┘
                     1536 dimensions

We can't visualize 1536D, but the math is the same as in 2D/3D.

Property 1: Semantic Similarity

Core concept:

Texts with similar meaning have close embeddings in the vector space.

Conceptual visualization (projected to 2D):

           Python ●
                 /   JavaScript ●
               /
             /
  Ruby ●   /
          /
         /
        /      
  Dog ●      
       \
        \  Cat ●
         \

• "Python", "JavaScript", "Ruby" are grouped (programming languages)
• "Dog", "Cat" are grouped (animals)
• The two groups are separated (different domains)

Measuring distance: Cosine Similarity

The standard metric in AI:

import numpy as np

def cosine_similarity(vec_a, vec_b):
    """
    Calculates similarity between two vectors using the cosine of the angle between them.
    
    Returns a value between -1 and 1:
    - 1.0 = identical (angle 0°)
    - 0.0 = orthogonal (angle 90°)
    - -1.0 = opposite (angle 180°)
    """
    dot_product = np.dot(vec_a, vec_b)
    norm_a = np.linalg.norm(vec_a)
    norm_b = np.linalg.norm(vec_b)
    
    return dot_product / (norm_a * norm_b)


# Example with small vectors (conceptual):
vec_python = np.array([0.5, 0.3, 0.8])
vec_javascript = np.array([0.4, 0.3, 0.7])
vec_cat = np.array([-0.2, 0.9, -0.1])

print(cosine_similarity(vec_python, vec_javascript))  # ~0.99 (very similar)
print(cosine_similarity(vec_python, vec_cat))         # ~0.15 (unrelated)

Why cosine similarity and not euclidean distance?

Comparison:

import numpy as np

vec_a = np.array([1.0, 0.0])
vec_b = np.array([2.0, 0.0])  # Same direction, double magnitude
vec_c = np.array([0.0, 1.0])  # Perpendicular direction

# Euclidean distance (geometric distance)
dist_ab = np.linalg.norm(vec_a - vec_b)  # 1.0
dist_ac = np.linalg.norm(vec_a - vec_c)  # 1.41

# Cosine similarity (angle)
cos_ab = cosine_similarity(vec_a, vec_b)  # 1.0  ← Identical (same angle)
cos_ac = cosine_similarity(vec_a, vec_c)  # 0.0  ← Orthogonal (90°)

Advantage of cosine:

• Ignores magnitude (vector length)
• Only measures direction (orientation in the space)
• For embeddings, direction = semantic meaning

In semantic search, we care about meaning (direction), not "intensity" (magnitude).

Real example with OpenAI embeddings

Comparing similar texts:

from openai import OpenAI
import numpy as np
import os
from dotenv import load_dotenv

# Setup
load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def get_embedding(text):
    """Helper to generate an embedding"""
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return np.array(response.data[0].embedding)

def cosine_similarity(vec_a, vec_b):
    """Calculates cosine similarity"""
    return np.dot(vec_a, vec_b) / (np.linalg.norm(vec_a) * np.linalg.norm(vec_b))

# Texts to compare
text_1 = "Python is a programming language"
text_2 = "Python is a language for programming"  # Paraphrase
text_3 = "JavaScript is a programming language"  # Similar (another language)
text_4 = "The cat sleeps on the couch"  # Unrelated

# Generate embeddings
emb_1 = get_embedding(text_1)
emb_2 = get_embedding(text_2)
emb_3 = get_embedding(text_3)
emb_4 = get_embedding(text_4)

# Calculate similarities
print(f"Python vs Python (paraphrase): {cosine_similarity(emb_1, emb_2):.4f}")
# → ~0.95-0.98 (very similar)

print(f"Python vs JavaScript: {cosine_similarity(emb_1, emb_3):.4f}")
# → ~0.85-0.90 (similar, both languages)

print(f"Python vs Cat: {cosine_similarity(emb_1, emb_4):.4f}")
# → ~0.60-0.70 (unrelated)

Interpreting scores:

• 0.95-1.00: Paraphrase or almost identical
• 0.85-0.95: Semantically related (same domain)
• 0.70-0.85: Somewhat related
• <0.70: Unrelated or minimally related

Property 2: Directionality

Concept:

The direction of the vector in the space represents the semantic meaning.

Conceptual visualization:

       Technology
            ^
            |
         Python ●
            |  \   JavaScript ●
            |    \
            |      \ Ruby ●
────────────┼──────────────────> Formality
            |
         Pizza ●
            |
            v
         Food

Different directions = different meanings:

• "Python", "JavaScript", "Ruby" → "Technology" direction
• "Pizza", "Hamburger" → "Food" direction

Vector arithmetic (conceptual):

With word embeddings (Word2Vec) it worked:

# Conceptual (it doesn't work this way with modern sentence embeddings):
vec_king = embed("king")
vec_queen = embed("queen")
vec_man = embed("man")
vec_woman = embed("woman")

# "king" - "man" + "woman" ≈ "queen"
result = vec_king - vec_man + vec_woman
# cosine_similarity(result, vec_queen) → ~0.90

With modern sentence embeddings (OpenAI, SBERT):

• This arithmetic is less predictable
• The models are more contextual and complex
• Don't rely on embedding arithmetic in production

Takeaway: Direction matters, but don't manipulate embeddings arithmetically in production.

Property 3: Magnitude vs Direction

Which matters more?

For semantic search: DIRECTION.

import numpy as np

# Two vectors with the same direction but different magnitude
vec_a = np.array([1.0, 1.0])  # Magnitude = sqrt(2) ≈ 1.41
vec_b = np.array([2.0, 2.0])  # Magnitude = sqrt(8) ≈ 2.83 (double)

# Cosine similarity ignores magnitude:
cosine = np.dot(vec_a, vec_b) / (np.linalg.norm(vec_a) * np.linalg.norm(vec_b))
print(cosine)  # 1.0 (identical according to cosine)

# Euclidean distance DOES consider magnitude:
distance = np.linalg.norm(vec_a - vec_b)
print(distance)  # ~1.41 (different according to euclidean)

Implication: Normalizing embeddings (L2 normalization) is common so that only direction matters.

Normalizing embeddings:

def normalize_embedding(embedding):
    """Normalizes an embedding to unit length (magnitude = 1.0)"""
    return embedding / np.linalg.norm(embedding)

# Example:
emb = np.array([3.0, 4.0])  # Magnitude = 5.0
emb_normalized = normalize_embedding(emb)

print(f"Original: {emb}")           # [3.0, 4.0]
print(f"Normalized: {emb_normalized}")  # [0.6, 0.8]
print(f"Normalized magnitude: {np.linalg.norm(emb_normalized)}")  # 1.0

Advantage: With normalized embeddings, dot product = cosine similarity (more efficient).

# With normalized embeddings:
dot_product = np.dot(emb_a_normalized, emb_b_normalized)
# dot_product == cosine_similarity(emb_a, emb_b)  ✅

Note: OpenAI embeddings are NOT normalized by default. Sentence-BERT does normalize them.

Property 4: Natural Clustering

Concept:

Embeddings of related texts naturally group into clusters.

Conceptual visualization (projected to 2D):

          Cluster "Languages"
    ┌─────────────────────────┐
    │ Python ●  JavaScript ●  │
    │                         │
    │ Ruby ●      Java ●      │
    └─────────────────────────┘


          Cluster "Animals"
    ┌─────────────────────────┐
    │ Dog ●       Cat ●       │
    │                         │
    │ Horse ●     Mouse ●     │
    └─────────────────────────┘

You don't need to label manually: Embeddings group themselves by semantic meaning.

Practical example: Simple clustering

from sklearn.cluster import KMeans
import numpy as np

# Texts from two domains:
texts = [
    # Technology (cluster 1):
    "Python is a programming language",
    "JavaScript runs in the browser",
    "Ruby is good for web development",
    
    # Animals (cluster 2):
    "The dog barks",
    "The cat meows",
    "The horse gallops"
]

# Generate embeddings
embeddings = [get_embedding(text) for text in texts]
embeddings_array = np.array(embeddings)

# Clustering with K-Means (2 clusters)
kmeans = KMeans(n_clusters=2, random_state=42)
clusters = kmeans.fit_predict(embeddings_array)

# Show results
for i, (text, cluster) in enumerate(zip(texts, clusters)):
    print(f"Cluster {cluster}: {text}")

Expected output:

Cluster 0: Python is a programming language
Cluster 0: JavaScript runs in the browser
Cluster 0: Ruby is good for web development
Cluster 1: The dog barks
Cluster 1: The cat meows
Cluster 1: The horse gallops

Automatic unsupervised clustering: embeddings group by topic.

Property 5: Similarity Transitivity

Concept:

If A is similar to B, and B is similar to C, then A is somewhat similar to C (approximately).

# Suppose:
similarity(A, B) = 0.95  # A and B very similar
similarity(B, C) = 0.90  # B and C very similar

# We can infer:
similarity(A, C) ≈ 0.85-0.90  # A and C probably similar

Application: Query expansion in search.

# User searches "Python"
# Similar embeddings: "Python", "py", "python3", "CPython", "Jython"
# → Automatically expands the search to synonyms/variants

Property 6: Smoothness of the Space

Concept:

Small changes in the text → small changes in the embedding.

# Texts with minimal changes:
texts = [
    "The cat sleeps",
    "The cat rests",     # Change: sleeps → rests (synonym)
    "The dog rests",     # Change: cat → dog
    "The pizza is good"  # Total change (different domain)
]

# Expected embeddings:
# emb_1 vs emb_2: ~0.90 (minimal change, synonym)
# emb_2 vs emb_3: ~0.85 (medium change, same verb)
# emb_3 vs emb_4: ~0.65 (total change, different domain)

Useful property: Lets you find "almost identical" texts (paraphrases, near-duplicates).

Comparing distance metrics

Cosine Similarity vs Euclidean Distance vs Dot Product:

import numpy as np

# Two example vectors
vec_a = np.array([1.0, 2.0, 3.0])
vec_b = np.array([2.0, 4.0, 6.0])  # Same direction, double magnitude

# 1. Cosine Similarity (angle)
cos_sim = np.dot(vec_a, vec_b) / (np.linalg.norm(vec_a) * np.linalg.norm(vec_b))
print(f"Cosine similarity: {cos_sim}")  # 1.0 (identical in direction)

# 2. Euclidean Distance (geometric distance)
euclidean = np.linalg.norm(vec_a - vec_b)
print(f"Euclidean distance: {euclidean}")  # ~3.74 (different in magnitude)

# 3. Dot Product
dot_prod = np.dot(vec_a, vec_b)
print(f"Dot product: {dot_prod}")  # 28.0 (considers magnitude)

Interpretation:

Recommendation: Cosine similarity for semantic search (99% of cases).

Dimensionality and compression

Trade-off: More dimensions = more information

# OpenAI models:
# text-embedding-3-small: 1536 dims → Cost/quality balance
# text-embedding-3-large: 3072 dims → Better quality, more expensive

Can we reduce dimensions?

from sklearn.decomposition import PCA
import numpy as np

# Original embeddings (1536 dims)
embeddings = np.array([get_embedding(text) for text in texts])

# Reduce to 128 dims with PCA
pca = PCA(n_components=128)
embeddings_compressed = pca.fit_transform(embeddings)

print(f"Original: {embeddings.shape}")         # (N, 1536)
print(f"Compressed: {embeddings_compressed.shape}")  # (N, 128)

# Trade-off: You lose ~10-20% of semantic information

When to compress:

• Storing millions of embeddings (save memory)
• Faster search (fewer dimensions = faster computation)

When NOT to compress:

• Precision is critical
• Small corpus (<100K docs)

Exercises

Exercise 1: Calculate cosine similarity

Calculate the cosine similarity between these two vectors:

import numpy as np

vec_a = np.array([3.0, 4.0, 0.0])
vec_b = np.array([4.0, 3.0, 0.0])

# What is the cosine similarity?

def cosine_similarity(vec_a, vec_b):
    dot_product = np.dot(vec_a, vec_b)
    norm_a = np.linalg.norm(vec_a)
    norm_b = np.linalg.norm(vec_b)
    return dot_product / (norm_a * norm_b)

vec_a = np.array([3.0, 4.0, 0.0])
vec_b = np.array([4.0, 3.0, 0.0])

similarity = cosine_similarity(vec_a, vec_b)
print(f"Cosine similarity: {similarity}")  # 0.96

Exercise 2: Normalize embeddings

Normalize this embedding to unit length:

import numpy as np

embedding = np.array([6.0, 8.0])
# Current magnitude: sqrt(6² + 8²) = 10.0

# Normalize to magnitude = 1.0

def normalize(vec):
    return vec / np.linalg.norm(vec)

embedding = np.array([6.0, 8.0])
embedding_normalized = normalize(embedding)

print(f"Original: {embedding}")
print(f"Normalized: {embedding_normalized}")
print(f"Normalized magnitude: {np.linalg.norm(embedding_normalized)}")

Original: [6. 8.]
Normalized: [0.6 0.8]
Normalized magnitude: 1.0

Exercise 3: Find the most similar

Given a query embedding, find the most similar document:

import numpy as np

query_emb = np.array([0.5, 0.3, 0.8])

docs_emb = {
    "doc_1": np.array([0.5, 0.3, 0.7]),
    "doc_2": np.array([0.1, 0.9, -0.2]),
    "doc_3": np.array([0.4, 0.3, 0.8])
}

# Which document is most similar to the query?

def cosine_similarity(vec_a, vec_b):
    return np.dot(vec_a, vec_b) / (np.linalg.norm(vec_a) * np.linalg.norm(vec_b))

query_emb = np.array([0.5, 0.3, 0.8])
docs_emb = {
    "doc_1": np.array([0.5, 0.3, 0.7]),
    "doc_2": np.array([0.1, 0.9, -0.2]),
    "doc_3": np.array([0.4, 0.3, 0.8])
}

# Calculate similarities
similarities = {}
for doc_id, doc_emb in docs_emb.items():
    sim = cosine_similarity(query_emb, doc_emb)
    similarities[doc_id] = sim
    print(f"{doc_id}: {sim:.4f}")

# Find the most similar
most_similar = max(similarities, key=similarities.get)
print(f"\nMost similar: {most_similar}")

doc_1: 0.9960
doc_2: 0.3511
doc_3: 0.9980

Most similar: doc_3

Exercise 4: Predict similarity

Which pair should have the higher cosine similarity?

Pair A:

• "The dog barks loudly"
• "The hound barks noisily"

Pair B:

• "Python is popular"
• "JavaScript is popular"

Common troubleshooting

Problem 1: Unexpectedly low similarities

# You expect 0.90+ but you get 0.70
similarity = cosine_similarity(emb_1, emb_2)  # → 0.70

Possible causes:

1. The texts are NOT as similar as you think (the embeddings are correct)
2. Different models used (incomparable)
3. Typos or noise in the text (affects embeddings)

Solution: Inspect the original texts.

Problem 2: All embeddings are similar

# All scores between 0.80-0.85
for doc_emb in docs:
    print(cosine_similarity(query_emb, doc_emb))  # 0.82, 0.83, 0.84...

Cause: A very homogeneous corpus (all docs on the same topic).

Solution: Normal if your corpus is specialized (e.g., all technical Python docs).

Problem 3: Negative similarity

similarity = cosine_similarity(emb_a, emb_b)  # → -0.15

Interpretation: The vectors point in opposite directions (angle >90°).

In practice: Rare with modern embeddings (OpenAI, SBERT).

If it happens: The texts are VERY conceptually different (e.g., "love" vs "hate").

Summary

What you learned:

• ✅ Semantic similarity: Similar texts → close embeddings
• ✅ Cosine similarity: Standard metric (measures angle, ignores magnitude)
• ✅ Directionality: The direction of the vector = semantic meaning
• ✅ Magnitude vs Direction: For semantic search, direction matters (normalize embeddings)
• ✅ Natural clustering: Embeddings group themselves by topic
• ✅ Metrics compared: Cosine > Euclidean for semantic search

Key concepts:

1. Cosine similarity returns values between -1 and 1 (typically 0.6-1.0 for related texts)
2. Normalizing embeddings makes dot product = cosine similarity (more efficient)
3. Embeddings create natural clustering without supervision

Additional resources

1. Cosine Similarity Explained - Visual explanation
2. Vector Space Models - Stanford NLP
3. Similarity Metrics - Complete comparison
4. Clustering with Embeddings - Sentence-BERT
5. OpenAI Embeddings Best Practices - Use cases

In the next capsule

Capsule 04: Vector Spaces

You'll learn:

• High-dimensional spaces (1536 dimensions)
• Why embeddings need so many dimensions
• Proximity in vector space = semantic similarity
• Natural clustering in high dimensionality

From vector properties to the geometry of spaces.

Module 1 - Embeddings Deep Dive Guide Vectors that capture meaning