Introduction to Module 5: Distance Metrics Deep Dive

Module welcome

You've learned embeddings, chunking, and RAG. Now we go deeper into distance metrics: the mathematical formulas that determine "similarity" between vectors. Cosine similarity is the standard, but why? When should you use Euclidean, dot product, or Manhattan?

In this module you'll master the main distance metrics, their mathematical and computational trade-offs, when to use each one, and how to optimize vector search. By the end, you'll be able to choose the right metric for your use case.

---

Module objectives

1. ✅ Cosine similarity: Why it's the standard for embeddings
2. ✅ Euclidean distance: When to use it vs cosine
3. ✅ Dot product: Equivalence with cosine (normalized)
4. ✅ Manhattan distance: Specific use cases
5. ✅ Performance: Speed benchmarks
6. ✅ Optimizations: FAISS, approximations (ANN)
7. ✅ Trade-offs: Accuracy vs speed
8. ✅ Project: A metrics comparator

---

Module roadmap

Phase 1: Main metrics (Capsules 01-04)

Capsule 01: Introduction (this capsule) Capsule 02: Cosine Similarity (the standard) Capsule 03: Euclidean Distance Capsule 04: Dot Product & Others

Phase 2: Performance & Optimization (Capsules 05-07)

Capsule 05: Performance Benchmarks Capsule 06: Approximate Nearest Neighbors (ANN) Capsule 07: FAISS Introduction

Phase 3: Project (Capsule 08)

Capsule 08: Project - Distance Metrics Comparator

---

Why distance metrics matter

# Embeddings (vectors):
emb_a = [0.5, 0.3, 0.8]
emb_b = [0.6, 0.2, 0.9]

# How "similar" are they?
# It depends on the metric:

cosine_sim = 0.9918   # Very similar (nearly identical directions)
euclidean_dist = 0.1732  # Close in space
dot_product = 1.08  # Strong alignment

# Different metrics → different results!

---

Main metrics (overview)

1. Cosine Similarity (the standard):

# Measures the angle between vectors (direction, not magnitude)
cos_sim = dot(a, b) / (||a|| × ||b||)

# Range: [-1, 1]
# 1.0 = Identical (same angle)
# 0.0 = Orthogonal (perpendicular)
# -1.0 = Opposite

# Why it's the standard:
# - Normalizes magnitude (only direction matters)
# - Robust to different scales

---

2. Euclidean Distance:

# "Straight-line" distance in space
euclidean = sqrt(Σ(a_i - b_i)²)

# Range: [0, ∞)
# 0 = Identical
# Larger value = Farther apart

# When to use it:
# - Magnitude matters (not just direction)
# - Already-normalized embeddings

---

3. Dot Product:

# Scalar product
dot_prod = Σ(a_i × b_i)

# Range: [-∞, ∞)
# Larger value = More similar

# Advantage:
# - FASTER than cosine (no normalization required)
# - Equivalent to cosine IF embeddings are normalized

---

4. Manhattan Distance:

# "Taxicab" distance (sum of absolute differences)
manhattan = Σ|a_i - b_i|

# Range: [0, ∞)
# 0 = Identical

# When to use it:
# - High dimensionality (curse of dimensionality)
# - Faster than Euclidean

---

Main trade-offs

Metric	Speed	Accuracy	Normalized?	Typical use
Cosine	Medium	High	Yes	Embeddings (default)
Dot Product	Fast	High*	No**	Normalized embeddings
Euclidean	Medium	Medium	No	Non-normalized embeddings
Manhattan	Fast	Medium	No	High dimensionality

*If embeddings are normalized
**Requires normalization for equivalence

---

Comparative example

import numpy as np

# Vectors
a = np.array([3, 4, 0])
b = np.array([4, 3, 0])

# Cosine
cos = np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
print(f"Cosine: {cos:.4f}")  # 0.9600

# Euclidean
euc = np.linalg.norm(a - b)
print(f"Euclidean: {euc:.4f}")  # 1.4142

# Dot product
dot = np.dot(a, b)
print(f"Dot: {dot:.4f}")  # 24.0000

# Manhattan
man = np.sum(np.abs(a - b))
print(f"Manhattan: {man:.4f}")  # 2.0000

Different metrics → different values!

---

When to use each metric

Cosine Similarity:

✅ Text embeddings (OpenAI, SBERT)
✅ When magnitude doesn't matter (semantics only)
✅ Default for RAG systems
✅ Embeddings from different models (different scales)

Example: Semantic search

Dot Product:

✅ Already-normalized embeddings (magnitude = 1.0)
✅ Performance-critical (3x faster than cosine)
✅ Large-scale search (millions of vectors)

Example: Production RAG (with prior normalization)

Euclidean Distance:

✅ Magnitude matters (not just direction)
✅ Specific embeddings (images, audio)
✅ Clustering (K-means uses Euclidean)

Example: Image similarity

Manhattan Distance:

✅ High dimensionality (curse of dimensionality)
✅ Performance-critical
✅ Outliers (more robust than Euclidean)

Example: High-dim feature vectors

---

Theory/practice balance (40/60)

Theory (40%):

• Mathematical formulas
• Geometric properties
• Conceptual trade-offs

Practice (60%):

• numpy implementation
• Speed benchmarks
• Empirical comparison
• Comparator project

---

What you WON'T learn (out of scope)

❌ Advanced metrics (Mahalanobis, Minkowski):

Reason: Rarely used for embeddings.
Coverage: Only the 4 main ones.

❌ Vector database internals:

Reason: A dedicated module comes next.
Coverage: Only in-memory numpy.

❌ Deep math (proofs, derivations):

Reason: Practical focus (AI Engineering).
Coverage: Intuition + implementation.

---

Module tools

# NumPy (vectors)
import numpy as np

# SciPy (distance functions)
from scipy.spatial.distance import cosine, euclidean

# Scikit-learn (metrics)
from sklearn.metrics.pairwise import cosine_similarity

# FAISS (optimization - optional)
import faiss

---

Module structure

module-05-distance-metrics/
└── es/
    ├── 01-module-introduction-5.md      ← You are here
    ├── 02-cosine-similarity.md
    ├── 03-euclidean-distance.md
    ├── 04-dot-product-others.md
    ├── 05-performance-benchmarks.md
    ├── 06-approximate-nn.md
    ├── 07-intro-to-faiss.md
    └── 08-project-metrics-comparator-2.md

---

Connection with the AI Engineering Path

M4 (Chunking + Evaluation)
          ↓
M5 (Distance Metrics) ← YOU ARE HERE
          ↓
M6 (Embedding Operations)
          ↓
Vector Databases Guide

---

In the next capsule

Capsule 02: Cosine Similarity

You'll learn:

• The mathematical formula
• Why it's the standard for embeddings
• numpy implementation
• Properties (normalization)
• Production-ready code

From introduction to a deep dive into cosine.

---

Module 5 - Embeddings Deep Dive Guide Distance Metrics: the heart of vector search