High-Dimensional Vector Spaces

Capsule overview

Embeddings don't live in a 2D or 3D space we can visualize—they live in high-dimensional spaces (typically 300-3072 dimensions) that have counterintuitive and fascinating properties.

In this capsule you'll learn what a high-dimensional vector space is, why we need so many dimensions to capture semantic meaning, how natural clustering works in these spaces, and why 2D/3D intuition doesn't always apply.

You'll also see practical examples of how real embeddings are distributed in 1536-dimensional spaces.

What is a vector space?

Mathematical definition (simplified):

A vector space is a set of vectors where you can add vectors and multiply them by scalars.

2D visualization:

      y (dimension 2)
      ^
      |
      |  Vector B (3, 4)
      |    ●
      |  /
      | /
      |/
      └──────────────> x (dimension 1)
    Origin

Each point in the space is a vector:

• Vector A = (2, 3) → 2 units in x, 3 in y
• Vector B = (3, 4) → 3 units in x, 4 in y

Embedding space (1536D):

# An embedding is a point in 1536-dimensional space:
embedding = [x₁, x₂, x₃, ..., x₁₅₃₆]
            └────────────────────────┘
                1536 coordinates

We can't visualize it, but the math works the same:

• Distance between points (cosine similarity)
• Neighborhood (close embeddings)
• Clustering (groups of points)

Why so many dimensions

Why not 2D or 3D?

Problem: The semantic complexity of language

Text: "Python is an interpreted, high-level, object-oriented 
        programming language with dynamic typing..."

How much information do you need to capture?
- Domain: Technology
- Type: Programming language
- Characteristics: Interpreted, high-level, OOP, dynamic typing
- Context: Could be backend, data science, AI, etc.
- Formality: Technical
- Sentiment: Neutral
- ... (many more properties)

2D-3D: Insufficient to capture all this semantic richness.

1536D: Each dimension captures a different aspect (learned automatically).

Trade-off: Dimensions vs Information

More dimensions = more semantic nuances captured, but higher computational cost.

The curse of dimensionality

Concept:

As dimensions increase, spaces become "empty" and 2D/3D intuition fails.

Counterintuitive example:

import numpy as np

# In 2D: Volume of sphere vs cube
radius_2d = 1.0
volume_sphere_2d = np.pi * radius_2d**2  # π ≈ 3.14
volume_cube_2d = (2 * radius_2d)**2      # 4

ratio_2d = volume_sphere_2d / volume_cube_2d  # ~0.785 (78.5%)

# In 10D: The sphere takes up LESS of the cube
def sphere_volume_nd(n_dims, radius=1.0):
    from scipy.special import gamma
    return (np.pi**(n_dims/2) * radius**n_dims) / gamma(n_dims/2 + 1)

def cube_volume_nd(n_dims, radius=1.0):
    return (2 * radius)**n_dims

ratio_10d = sphere_volume_nd(10) / cube_volume_nd(10)
print(f"2D: Sphere takes up {ratio_2d:.2%} of the cube")
print(f"10D: Sphere takes up {ratio_10d:.2%} of the cube")
# → In 10D: only ~0.25% (mostly empty space!)

Implication: In high dimensionality, points tend to be in the "corners" of the space.

Distances in high dimensionality:

import numpy as np

# Generate random points in different dimensions
def avg_distance(n_dims, n_points=1000):
    """Calculates the average distance between random points"""
    points = np.random.randn(n_points, n_dims)
    
    # Calculate distance between point 1 and all the others
    distances = [np.linalg.norm(points[0] - points[i]) 
                 for i in range(1, n_points)]
    
    return np.mean(distances), np.std(distances)

# Compare dimensions
for n_dims in [2, 10, 100, 1000]:
    avg, std = avg_distance(n_dims)
    print(f"{n_dims}D: Average distance = {avg:.2f} (std = {std:.2f})")

Output (conceptual):

2D:    Average distance = 1.13 (std = 0.52)
10D:   Average distance = 3.22 (std = 0.41)
100D:  Average distance = 10.05 (std = 0.32)
1000D: Average distance = 31.68 (std = 0.18)

Notice: With more dimensions, distances increase BUT variance decreases.

All points are approximately the same distance apart from each other (counterintuitive).

Natural clustering in high dimensionality

Key concept:

Even though all points are "far", semantically related points are RELATIVELY closer.

Conceptual visualization (projected to 2D):

        Cluster "Technology"
     ┌────────────────────────┐
     │   Python ●             │
     │      Ruby ●  Java ●    │
     │   JavaScript ●         │
     └────────────────────────┘
              .
              .  (a lot of distance)
              .
     ┌────────────────────────┐
     │  Dog ●      Cat ●      │
     │      Horse ●           │
     │   Lion ●               │
     └────────────────────────┘
        Cluster "Animals"

In 1536D:

• Intra-cluster distance: ~0.10-0.20 (close)
• Inter-cluster distance: ~0.40-0.60 (far)
• The relative ratio enables clustering, even though absolutely everything is "far"

Practical example: Distribution of embeddings

from openai import OpenAI
import numpy as np
import os
from dotenv import load_dotenv

load_dotenv()
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def get_embedding(text):
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return np.array(response.data[0].embedding)

# Texts from 3 clusters
texts = {
    "Technology": [
        "Python is a programming language",
        "JavaScript runs in the browser",
        "Ruby is good for web development"
    ],
    "Animals": [
        "The dog barks loudly",
        "The cat meows softly",
        "The horse gallops fast"
    ],
    "Food": [
        "The pizza is delicious",
        "The hamburger has cheese",
        "Tacos are Mexican"
    ]
}

# Generate embeddings
embeddings_by_cluster = {}
for cluster_name, cluster_texts in texts.items():
    embeddings_by_cluster[cluster_name] = [
        get_embedding(text) for text in cluster_texts
    ]

# Calculate average INTRA-cluster distance
print("INTRA-cluster distances (similar):")
for cluster_name, cluster_embs in embeddings_by_cluster.items():
    distances = []
    for i in range(len(cluster_embs)):
        for j in range(i+1, len(cluster_embs)):
            dist = np.linalg.norm(cluster_embs[i] - cluster_embs[j])
            distances.append(dist)
    avg_dist = np.mean(distances)
    print(f"  {cluster_name}: {avg_dist:.4f}")

# Calculate average INTER-cluster distance
print("\nINTER-cluster distances (different):")
cluster_names = list(embeddings_by_cluster.keys())
for i in range(len(cluster_names)):
    for j in range(i+1, len(cluster_names)):
        cluster_1 = embeddings_by_cluster[cluster_names[i]]
        cluster_2 = embeddings_by_cluster[cluster_names[j]]
        
        distances = []
        for emb_1 in cluster_1:
            for emb_2 in cluster_2:
                dist = np.linalg.norm(emb_1 - emb_2)
                distances.append(dist)
        
        avg_dist = np.mean(distances)
        print(f"  {cluster_names[i]} vs {cluster_names[j]}: {avg_dist:.4f}")

Expected output (conceptual):

INTRA-cluster distances (similar):
  Technology: 0.45
  Animals: 0.42
  Food: 0.48

INTER-cluster distances (different):
  Technology vs Animals: 0.78
  Technology vs Food: 0.82
  Animals vs Food: 0.80

Interpretation: Intra-cluster distances (~0.45) << inter-cluster (~0.80).

Subspaces: Embeddings organized into regions

Concept:

The 1536D space is not uniform. Embeddings of similar topics occupy "subspaces" or regions of the total space.

Analogy with cities:

1536D space = The whole world

Subspace "Technology" = Continent Europe
  → Python, JavaScript, Ruby are in nearby "countries"

Subspace "Animals" = Continent Africa
  → Dog, Cat, Horse are in nearby "countries"

The continents are separated (high inter-cluster distance)
But within each continent there's structure (nearby cities)

Hierarchy of subspaces:

# Full embedding space (1536D)
└── Subspace "Technology" (occupies ~dimensions 1-500)
    ├── Sub-subspace "Languages" (dim 1-200)
    │   ├── Python, JavaScript, Ruby
    ├── Sub-subspace "Frameworks" (dim 200-400)
    │   ├── React, Django, Rails
    └── Sub-subspace "Tools" (dim 400-500)
        ├── Git, Docker, Kubernetes

Note: This is conceptual. The dimensions are NOT assigned manually—the model learns this structure automatically.

Proximity in vector space

Operational definition:

Two embeddings are "proximate" if their cosine similarity is high (>0.80).

def are_proximate(emb_a, emb_b, threshold=0.80):
    """
    Determines whether two embeddings are proximate
    
    Args:
        emb_a, emb_b: Embeddings to compare
        threshold: Similarity threshold (default 0.80)
        
    Returns:
        True if cosine similarity > threshold
    """
    cos_sim = np.dot(emb_a, emb_b) / (
        np.linalg.norm(emb_a) * np.linalg.norm(emb_b)
    )
    return cos_sim > threshold

# Example
emb_python = get_embedding("Python")
emb_javascript = get_embedding("JavaScript")
emb_cat = get_embedding("The cat")

print(are_proximate(emb_python, emb_javascript))  # True (both languages)
print(are_proximate(emb_python, emb_cat))         # False (different domains)

Density of the embedding space

Question: What percentage of the 1536D space is occupied by real embeddings?

Answer: Very little (< 0.001%).

# Total volume of the 1536D space (conceptual):
# Infinite (continuous space)

# Real embeddings (e.g., 1 million documents):
# Occupy specific regions (clusters)

# Analogy:
# 1536D space = Pacific Ocean
# Embeddings = Scattered islands
# Semantic search = Navigating from island to island

Implication: Efficient search requires indexing (vector databases), not linear search.

Projection to low-dimensional spaces

Problem: We can't visualize 1536D.

Solution: Project to 2D/3D for visualization (with loss of information).

from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
import numpy as np

# Generate embeddings (example with texts from 3 clusters)
texts = [
    "Python", "JavaScript", "Ruby",  # Technology
    "Dog", "Cat", "Horse",           # Animals
    "Pizza", "Hamburger", "Taco"     # Food
]

embeddings = np.array([get_embedding(text) for text in texts])

# Project from 1536D to 2D with PCA
pca = PCA(n_components=2)
embeddings_2d = pca.fit_transform(embeddings)

# Visualize
plt.figure(figsize=(10, 6))

# Plot by clusters
colors = ['red', 'blue', 'green']
labels = ['Technology', 'Animals', 'Food']

for i, color, label in zip(range(0, 9, 3), colors, labels):
    plt.scatter(
        embeddings_2d[i:i+3, 0], 
        embeddings_2d[i:i+3, 1], 
        c=color, 
        label=label,
        s=100
    )
    
    # Annotate points
    for j in range(3):
        plt.annotate(
            texts[i+j], 
            (embeddings_2d[i+j, 0], embeddings_2d[i+j, 1]),
            fontsize=8
        )

plt.xlabel('PC1 (First principal component)')
plt.ylabel('PC2 (Second principal component)')
plt.title('Embeddings projected from 1536D to 2D')
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.savefig('embeddings_2d_projection.png')
plt.show()

# Explained variance
print(f"Variance explained by PC1: {pca.explained_variance_ratio_[0]:.2%}")
print(f"Variance explained by PC2: {pca.explained_variance_ratio_[1]:.2%}")
print(f"Total variance explained (2D): {sum(pca.explained_variance_ratio_):.2%}")

Expected output:

Variance explained by PC1: 12.5%
Variance explained by PC2: 8.3%
Total variance explained (2D): 20.8%

Interpretation: You only capture ~20% of the information in 2D. The other 1534 dimensions matter.

"Useful" dimensions vs "noise" dimensions

Not all dimensions contribute equally:

from sklearn.decomposition import PCA

# Generate embeddings of 100 documents
embeddings = np.array([get_embedding(text) for text in documents])  # (100, 1536)

# PCA to analyze variance per dimension
pca = PCA(n_components=1536)
pca.fit(embeddings)

# Plot cumulative explained variance
cumulative_variance = np.cumsum(pca.explained_variance_ratio_)

# How many dimensions explain 95% of the variance?
n_dims_95 = np.argmax(cumulative_variance >= 0.95) + 1
print(f"Dimensions for 95% variance: {n_dims_95}")
# → Typically ~400-600 dimensions

# How many dimensions explain 99% of the variance?
n_dims_99 = np.argmax(cumulative_variance >= 0.99) + 1
print(f"Dimensions for 99% variance: {n_dims_99}")
# → Typically ~800-1000 dimensions

Implication: You could compress from 1536D → 400D with minimal loss (<5% information).

Trade-off: You save memory/compute but lose semantic nuances.

The geometry of semantic search

How search works in vector space:

1. The user makes a query: "Python tutorial"
   ↓
2. Embed the query → query_embedding (point in 1536D)
   ↓
3. Calculate the distance to ALL docs in the corpus:
   distances = [dist(query_emb, doc_emb) for doc_emb in corpus_embs]
   ↓
4. Sort by distance (ascending)
   ↓
5. Return the top-K closest (e.g., top-5)

Conceptual visualization (2D):

         Query ●
           / | \
         /   |   \
       /     |     \
     /       |       \
  Doc1 ●   Doc2 ●   Doc3 ●
  (0.92)   (0.88)   (0.75)
   
  ↓ Returns Doc1, Doc2 (top-2)

Complexity: O(N) for linear search (N = corpus size).

Optimization: Vector databases (HNSW, FAISS) reduce it to O(log N).

Sparse vs Dense embeddings

Dense embeddings (what we've seen):

# Dense: Almost all values != 0
dense_emb = [0.023, -0.145, 0.892, -0.234, 0.567, ...]  # 1536 values
non_zero_count = np.count_nonzero(dense_emb)  # ~1530 (>99%)

Advantage: Captures rich semantic nuances.

Disadvantage: Computationally costly (1536 dimensions).

Sparse embeddings (alternative):

# Sparse: Most values = 0
sparse_emb = [0, 0, 0.5, 0, 0, 0, 0.8, 0, ...]  # 10000 dimensions
non_zero_count = np.count_nonzero(sparse_emb)  # ~50 (0.5%)

Example: SPLADE (Sparse Lexical and Expansion)

Advantage: More efficient in memory/compute.

Disadvantage: Fewer semantic nuances captured.

Use: Hybrid dense + sparse (the best of both worlds).

Exercises

Exercise 1: Calculate density

Given an embedding, calculate what percentage of the values are non-zero:

import numpy as np

embedding = np.random.randn(1536)  # Simulation

# Calculate density (% of values != 0)

import numpy as np

embedding = np.random.randn(1536)

# Count non-zero values
non_zero_count = np.count_nonzero(embedding)

# Calculate percentage
density = (non_zero_count / len(embedding)) * 100

print(f"Density: {density:.2f}%")
print(f"Non-zero values: {non_zero_count}/{len(embedding)}")

Density: 100.00%
Non-zero values: 1536/1536

Exercise 2: PCA projection

Reduce a set of embeddings from 1536D to 2D and calculate the explained variance:

from sklearn.decomposition import PCA
import numpy as np

# Simulate 100 embeddings
embeddings = np.random.randn(100, 1536)

# Project to 2D with PCA
# What % of variance does 2D capture?

from sklearn.decomposition import PCA
import numpy as np

# Simulate embeddings
embeddings = np.random.randn(100, 1536)

# PCA to 2D
pca = PCA(n_components=2)
embeddings_2d = pca.fit_transform(embeddings)

# Explained variance
variance_explained = sum(pca.explained_variance_ratio_)

print(f"Variance explained by 2D: {variance_explained:.2%}")
print(f"PC1: {pca.explained_variance_ratio_[0]:.2%}")
print(f"PC2: {pca.explained_variance_ratio_[1]:.2%}")

Variance explained by 2D: ~1.5-2.5%
PC1: ~0.8-1.3%
PC2: ~0.7-1.2%

Exercise 3: Find K nearest neighbors

Given a query embedding, find the 3 most similar documents:

import numpy as np

query_emb = np.random.randn(1536)
doc_embs = np.random.randn(10, 1536)  # 10 documents

# Find the 3 most similar (top-3)

import numpy as np

def cosine_similarity(vec_a, vec_b):
    return np.dot(vec_a, vec_b) / (np.linalg.norm(vec_a) * np.linalg.norm(vec_b))

query_emb = np.random.randn(1536)
doc_embs = np.random.randn(10, 1536)

# Calculate similarities
similarities = []
for i, doc_emb in enumerate(doc_embs):
    sim = cosine_similarity(query_emb, doc_emb)
    similarities.append((i, sim))

# Sort by similarity (descending)
similarities.sort(key=lambda x: x[1], reverse=True)

# Top-3
print("Top-3 most similar documents:")
for rank, (doc_id, sim) in enumerate(similarities[:3], 1):
    print(f"  {rank}. Doc {doc_id}: {sim:.4f}")

Top-3 most similar documents:
  1. Doc 7: 0.0523
  2. Doc 2: 0.0312
  3. Doc 9: 0.0145

Exercise 4: Compare intra vs inter-cluster distances

Calculate average distances within clusters vs between clusters:

# Cluster 1: Technology
cluster_1 = [emb_python, emb_javascript, emb_ruby]

# Cluster 2: Animals
cluster_2 = [emb_dog, emb_cat, emb_horse]

# Calculate:
# 1. Average INTRA-cluster 1 distance
# 2. Average INTER-cluster (1 vs 2) distance

import numpy as np

def avg_intra_distance(cluster):
    """Average distance within the cluster"""
    distances = []
    for i in range(len(cluster)):
        for j in range(i+1, len(cluster)):
            dist = np.linalg.norm(cluster[i] - cluster[j])
            distances.append(dist)
    return np.mean(distances)

def avg_inter_distance(cluster_1, cluster_2):
    """Average distance between clusters"""
    distances = []
    for emb_1 in cluster_1:
        for emb_2 in cluster_2:
            dist = np.linalg.norm(emb_1 - emb_2)
            distances.append(dist)
    return np.mean(distances)

# Simulate clusters (in production, you'd use get_embedding())
cluster_1 = [np.random.randn(1536) for _ in range(3)]
cluster_2 = [np.random.randn(1536) for _ in range(3)]

intra_dist = avg_intra_distance(cluster_1)
inter_dist = avg_inter_distance(cluster_1, cluster_2)

print(f"INTRA-cluster distance: {intra_dist:.4f}")
print(f"INTER-cluster distance: {inter_dist:.4f}")
print(f"Ratio (inter/intra): {inter_dist/intra_dist:.2f}x")

INTRA-cluster distance: 0.45
INTER-cluster distance: 0.78
Ratio (inter/intra): 1.73x

Common troubleshooting

Problem 1: PCA projection loses a lot of information

# Only 15% of variance explained in 2D
pca = PCA(n_components=2)
embeddings_2d = pca.fit_transform(embeddings)
print(f"Variance explained: {sum(pca.explained_variance_ratio_):.2%}")  # 15%

Cause: Normal. 2D can't capture the complexity of 1536D.

Solution: Use 3D or even 50D for intermediate visualization (not for production).

Problem 2: All points seem equally distant

# Similar distances between all points
for doc_emb in doc_embs:
    dist = np.linalg.norm(query_emb - doc_emb)
    print(dist)  # 15.2, 15.8, 16.1, 15.9... (all ~15-16)

Cause: The curse of dimensionality (high dim → distances converge).

Solution: Use cosine similarity (angle) instead of euclidean distance (magnitude).

Summary

What you learned:

• ✅ Vector space: A set of N-dimensional vectors
• ✅ High dimensionality: 1536D needed to capture semantic complexity
• ✅ Curse of dimensionality: 2D/3D intuition doesn't apply, distances converge
• ✅ Natural clustering: Related embeddings form clusters (subspaces)
• ✅ Proximity: Cosine similarity defines neighborhood in the space
• ✅ Projection: PCA reduces dimensions for visualization (with loss)

Key concepts:

1. More dimensions = more semantic information captured
2. Natural clustering emerges without supervision
3. Projection to 2D loses ~80% of the information

Additional resources

1. Curse of Dimensionality - Mathematical explanation
2. Visualizing High-Dimensional Data - t-SNE explained
3. PCA Explained - Principal Component Analysis
4. Vector Spaces in NLP - Stanford paper
5. UMAP for Visualization - Alternative to PCA

In the next capsule

Capsule 05: Real-World Use Cases

You'll learn:

• Semantic search in production
• RAG (Retrieval-Augmented Generation)
• Recommendation systems
• Document classification and clustering
• Concrete examples of real applications

From spatial theory to practical applications.

Module 1 - Embeddings Deep Dive Guide High-dimensional spaces where meaning lives