Mini-Project: Distance Metrics Comparator

Description

You'll build a comparator that evaluates the 4 main metrics (cosine, Euclidean, dot product, Manhattan) on speed and accuracy. You'll compare exact vs approximate search (FAISS).

---

Project

comparator.py:

import numpy as np
import time
from typing import Dict, List
import faiss

class MetricsComparator:
    """Compare distance metrics"""
    
    def __init__(self, embeddings: np.ndarray, ground_truth_indices: List[int]):
        """
        Args:
            embeddings: (N, D) array
            ground_truth_indices: True top-K indices (exact search)
        """
        self.embeddings = embeddings.astype('float32')
        self.ground_truth = ground_truth_indices
        self.n, self.d = embeddings.shape
    
    def benchmark_metric(self, query: np.ndarray, metric: str, k: int = 5) -> Dict:
        """Benchmark a single metric"""
        query = query.astype('float32')
        
        start = time.time()
        
        if metric == 'cosine':
            # Normalize
            emb_norm = self.embeddings / np.linalg.norm(self.embeddings, axis=1, keepdims=True)
            query_norm = query / np.linalg.norm(query)
            scores = np.dot(emb_norm, query_norm)
            top_k = np.argsort(scores)[::-1][:k]
        
        elif metric == 'dot':
            scores = np.dot(self.embeddings, query)
            top_k = np.argsort(scores)[::-1][:k]
        
        elif metric == 'euclidean':
            dists = np.linalg.norm(self.embeddings - query, axis=1)
            top_k = np.argsort(dists)[:k]
        
        elif metric == 'manhattan':
            dists = np.sum(np.abs(self.embeddings - query), axis=1)
            top_k = np.argsort(dists)[:k]
        
        latency = time.time() - start
        
        # Accuracy (overlap with ground truth)
        accuracy = len(set(top_k) & set(self.ground_truth[:k])) / k
        
        return {
            'metric': metric,
            'latency_ms': latency * 1000,
            'accuracy': accuracy,
            'top_k': top_k.tolist()
        }
    
    def benchmark_faiss(self, query: np.ndarray, k: int = 5) -> Dict:
        """Benchmark FAISS (HNSW)"""
        query = query.reshape(1, -1).astype('float32')
        
        # Build HNSW index
        index = faiss.IndexHNSWFlat(self.d, 32)
        
        start_build = time.time()
        index.add(self.embeddings)
        build_time = time.time() - start_build
        
        # Search
        start_search = time.time()
        distances, indices = index.search(query, k)
        search_time = time.time() - start_search
        
        # Accuracy
        accuracy = len(set(indices[0]) & set(self.ground_truth[:k])) / k
        
        return {
            'metric': 'FAISS-HNSW',
            'build_time_ms': build_time * 1000,
            'latency_ms': search_time * 1000,
            'accuracy': accuracy,
            'top_k': indices[0].tolist()
        }
    
    def compare_all(self, query: np.ndarray, k: int = 5) -> List[Dict]:
        """Compare all metrics"""
        results = []
        
        # Exact metrics
        for metric in ['cosine', 'dot', 'euclidean', 'manhattan']:
            result = self.benchmark_metric(query, metric, k)
            results.append(result)
        
        # FAISS
        result_faiss = self.benchmark_faiss(query, k)
        results.append(result_faiss)
        
        return results

# Demo
if __name__ == "__main__":
    # Generate data
    np.random.seed(42)  # Reproducibility: same numbers on every run
    n = 10000
    d = 1536
    embeddings = np.random.randn(n, d)
    query = np.random.randn(d)
    
    # Ground truth (exact search with cosine)
    emb_norm = embeddings / np.linalg.norm(embeddings, axis=1, keepdims=True)
    query_norm = query / np.linalg.norm(query)
    scores = np.dot(emb_norm, query_norm)
    ground_truth = np.argsort(scores)[::-1][:5]
    
    # Compare
    comparator = MetricsComparator(embeddings, ground_truth)
    results = comparator.compare_all(query, k=5)
    
    # Report
    print("=== Distance Metrics Comparison ===\n")
    print(f"{'Metric':<15} {'Latency (ms)':<15} {'Accuracy':<10}")
    print("-" * 40)
    
    for result in results:
        metric = result['metric']
        latency = result['latency_ms']
        accuracy = result['accuracy']
        print(f"{metric:<15} {latency:<15.2f} {accuracy:<10.2%}")

Output (with seed=42; latency times vary by machine):

=== Distance Metrics Comparison ===

Metric          Latency (ms)    Accuracy  
----------------------------------------
cosine          19.54           100.00%
dot             1.80            100.00%
euclidean       23.00           20.00%
manhattan       16.94           40.00%
FAISS-HNSW      0.76            0.00%

Why euclidean, manhattan, and FAISS drop so low here: the ground truth is defined with cosine over normalized vectors, but this data is np.random.randn — pure noise, with no semantic structure. Euclidean and Manhattan are sensitive to magnitude (which is random here), so their top-5 barely overlaps with cosine's. And FAISS-HNSW uses L2 distance over non-normalized vectors, so its top-5 doesn't match the cosine ground truth either. With real embeddings (which do have semantic structure) that are normalized, dot product and FAISS-HNSW align almost 100% with cosine — which is exactly why in production you pre-normalize and use dot/inner-product. This demo, on random data, measures how much the metrics agree with each other, not a real semantic accuracy.

---

Module 5 summary

What you learned:

• ✅ Cosine similarity (the standard)
• ✅ Euclidean distance (magnitude matters)
• ✅ Dot product (3x faster)
• ✅ Performance benchmarks
• ✅ ANN (100x speedup)
• ✅ FAISS (production-ready)

Key insights:

1. Cosine: Default for embeddings
2. Dot product: Production optimization (if normalized)
3. FAISS HNSW: 100x faster, 99% accuracy

---

Next module

Module 6: Embedding Operations

You'll learn:

• Arithmetic (A + B, A - B)
• Interpolation (blend embeddings)
• Dimensionality reduction (PCA, UMAP)
• Embedding composition

---

Module 5 complete ✅ Distance Metrics: the vector search engine