Module 2: Vector Spaces

1. Module Introduction: Vector Spaces - The Universe of Meaning

Overview

This is the module where you go from understanding individual vectors (Module 1) to understanding the space where all the vectors live: the vector space. If in Module 1 you learned what a point on a map is, here you learn what the whole map is and how it's structured.

A vector space is the "geometric universe" where every word, document or concept has a unique location. Understanding how that space works (its dimensions, bases, subspaces, transformations) is key to understanding how LLMs organize meaning and how semantic search finds relevant documents.

This module is 100% conceptual (like Module 1): there's no code, no formal linear algebra. Just geometric intuition applied to high-dimensional spaces. When you finish, you'll understand why an LLM's "embedding space" is a structured space where geometry captures complex semantic relationships.

Why this module?

The problem:

After Module 1, you understand individual vectors:

Vector("dog") = [0.23, -0.45, ..., -0.34] (1536 numbers)
Vector("cat") = [0.25, -0.43, ..., -0.32] (1536 numbers)

But questions remain:

Do all of these vectors "live" in the same "place"?
What defines that "place"? Why does it have 1536 dimensions and not 500 or 5000?
Are there "regions" of the space with special meaning? (e.g. are all the animals in one zone?)
How do we visualize 1536D if we can only draw 2D/3D?

This module answers those questions. You'll see that all vectors live in a structured vector space, with dimensions, axes, subspaces, and geometric properties that capture meaning.

The module's key intuition:

"A vector space is the geometric universe where every concept has a unique location. The geometry of that space (distances, directions, regions) reflects the structure of meaning."

Analogy: If vectors are "houses" in a city, the vector space is the whole city: its streets (axes), its neighborhoods (subspaces), its extent (dimensions). Understanding the city lets you navigate between houses (semantic search), find themed neighborhoods (clustering), and transform maps (2D projections).

Module Objectives

By completing this module, you'll be able to:

Explain what a vector space is (the "container" where all vectors live)
Understand bases and dimensions (how the axes structure the space)
Identify subspaces (regions of the space with special properties)
Grasp projections (how to reduce 1536D to 2D for visualization)
Apply normalization (why you make all vectors have magnitude 1)
Reason about semantic spaces (how LLMs organize meaning geometrically)
Visualize complex spaces (using projections, analogies, abstractions)

Key competency gained: You'll understand that an LLM's "embedding space" isn't a random set of vectors, but a structured space where geometry (distances, directions, regions) captures complex semantic relationships.

Module roadmap: the 7 capsules

#	Capsule	What you'll see	What you'll gain	Approx. duration
01	Module introduction	The full roadmap, why vector spaces matter	An overview of the module	15 min
02	What is a vector space?	The intuitive definition, basic axioms without excessive formalism	The concept of a "universe of vectors"	20 min
03	Bases and dimensions	Axes as "pure directions", dimensionality as capacity	Understanding what defines the "size" of the space	25 min
04	Subspaces and projections	Themed regions, dimensionality reduction (t-SNE, UMAP conceptually)	Visualizing high dimensions in 2D/3D	25 min
05	Normalization and transformations	Why magnitude = 1, rotations, conceptual translations	Operations that preserve or change meaning	20 min
06	Semantic spaces in AI	How LLMs organize meaning, clusters, semantic directions	Direct application to embeddings	25 min
07	Capstone exercise	Designing a 2D vector space for concepts, identifying subspaces	Consolidating the intuition with practice	30 min

Total: ~2.5 hours

Connection to Module 1

In Module 1 you learned:

✅ What a vector is (direction + magnitude)
✅ How to visualize it (2D, 3D, high dimensions)
✅ How to operate on it (addition, subtraction, scaling)
✅ Why AI uses vectors (semantic search, embeddings)

In Module 2 you'll extend that:

🎯 From "one vector" → "all the vectors together" (the space)
🎯 From "coordinates" → "axes and bases" (what defines the coordinates)
🎯 From "closeness" → "the structure of the space" (subspaces, regions)
🎯 From "2D intuition" → "projections from 1536D" (dimensionality reduction)

Analogy: Module 1 taught you how to read a map (coordinates, distances). Module 2 teaches you how the map is built (the geographic projection, meridians, parallels, regions, scales).

Pedagogical order: how the module is structured

Capsules 02-03: Abstract fundamentals

Capsule 02 (What is a vector space):

The intuitive definition without excessive formalism
Basic axioms (addition, scaling) in plain language
Examples: 2D space, 1536D space

Capsule 03 (Bases and dimensions):

What "axes" are (the bases of the space)
Why 1536D has 1536 axes
Linear independence (the concept, without heavy algebra)

Goal: Understanding the formal (but intuitive) structure of the space.

Capsules 04-05: Operations and transformations

Capsule 04 (Subspaces and projections):

Subspaces as "themed regions" (all the animals, all the vehicles)
Projections: how to reduce 1536D to 2D (t-SNE, UMAP, PCA conceptually)
Why projections lose information but preserve intuition

Capsule 05 (Normalization and transformations):

Why normalize (magnitude = 1, only direction matters)
Rotations and translations (transforming the space)
Which operations preserve meaning and which don't

Goal: Seeing how we operate on and visualize high-dimensional spaces.

Capsules 06-07: AI application and practice

Capsule 06 (Semantic spaces in AI):

How LLMs organize the "embedding space"
Semantic clusters (animals, vehicles, emotions)
Semantic directions (gender, size, time)
Why king - man + woman = queen works (the geometry of the space)

Capsule 07 (Capstone exercise):

Designing a 2D space for concepts
Identifying subspaces (themed regions)
Projecting 3D concepts to 2D
Reflecting on how it scales to 1536D

Goal: Applying everything conceptually and consolidating the intuition.

What you will NOT see in this module

To keep the focus conceptual (without technical overload), this module does NOT cover:

❌ Formal linear algebra: You won't see matrices, determinants, eigenvalues, mathematical SVD
❌ Code: You won't implement projections or transformations (that's for later guides)
❌ Model details: You won't see how BERT or GPT build their space internally (that's Guide #6)
❌ Vector databases: You won't see HNSW, IVF indexes, etc. (that's Module 4 and Guide #7)

This is 100% conceptual: The geometry of the space, the intuition of its structure, visualization without implementation.

Prerequisites

Prior knowledge:

Module 1 completed: What a vector is, basic operations, why AI uses vectors
2D/3D geometry: Knowing how to draw points on a Cartesian plane
Basic math: Addition, subtraction, square roots

You do not need:

❌ Linear algebra (we won't use matrices or determinants)
❌ Programming (there's no code in this module)
❌ Experience with PCA, t-SNE, UMAP (we'll cover them conceptually)

Level: Foundational. If you completed Module 1 and you can explain what a vector is, you're ready.

What sets this module apart

vs traditional linear algebra courses:

Aspect	Traditional courses	This module
Approach	Formal axioms, abstract spaces	Geometric intuition applied to AI
Context	Pure mathematics	Embeddings, semantic search, RAG
Code	NumPy/SciPy implementations	There is NO code (concepts only)
Visualization	Few images	Many diagrams, 2D projections
Exercises	Matrix calculations	Conceptual design, reflection

Our unique promise: Finish understanding how LLMs organize meaning geometrically, without writing code or solving matrix equations.

Connection to semantic search and RAG

Everything you'll see in this module is used directly in semantic search and RAG:

1. Vector space = Embedding space

When you use OpenAI's text-embedding-3-small, each text is mapped to a point in a 1536D vector space. That space has:

Dimensions: 1536 axes (Capsule 03)
Structure: Semantic regions (animals, vehicles, emotions) (Capsule 06)
Operations: Normalization (magnitude 1), projections (visualization) (Capsules 04-05)

2. Subspaces = Semantic clusters

In the embedding space, there are themed "regions":

All the animals are in one subspace
All the vehicles are in another subspace
Words about emotions are in another subspace

Semantic search exploits that structure: searching for "domestic animal" finds vectors in the animal subspace.

3. Projections = Embedding visualization

When you see visualizations of embeddings (e.g. in blogs, papers), they're projections from 1536D to 2D (t-SNE, UMAP). Understanding projections (Capsule 04) lets you interpret those visualizations without thinking you're "losing critical information" (you do lose info, but the intuition is preserved).

4. Normalization = Cosine similarity

When you use cosine similarity (Module 3), you're internally normalizing vectors (magnitude = 1) so that only direction matters. Understanding normalization (Capsule 05) explains why cosine is superior to Euclidean distance in high dimensions.

Concrete examples you'll understand after the module

By the end of Module 2, you'll be able to answer questions like:

Q1: Why are OpenAI's embeddings 1536D and not 500D or 5000D?

A: Because 1536D is the size of the vector space OpenAI chose (a balance between capacity and compute). More dimensions = more capacity to capture nuance, but more cost. You'll understand that dimensionality is an architectural decision of the model.

Q2: Why do t-SNE visualizations of embeddings look "clustered"?

A: Because t-SNE projects the 1536D space to 2D while preserving local closeness. The clusters you see (animals together, vehicles together) reflect the structure of the original space (semantic subspaces). You'll understand that it's a projection, not the real space.

Q3: Why normalize embeddings before computing similarity?

A: Because in high dimensions, magnitude can be noise (it depends on word frequency, not meaning). Normalizing (magnitude = 1) removes that noise and makes only direction matter (what kind of concept it is, not how "big" the vector is).

Q4: What does "GPT-4's embedding space has semantic structure" mean?

A: It means it isn't a random set of vectors; there are regions (subspaces) where related concepts cluster, and directions (e.g. gender, size) that capture relationships. That geometric structure reflects the semantic structure of language.

Conceptual map of the module

MODULE 2: Vector Spaces
│
├─ 02. What is a vector space?
│   ├─ The intuitive definition (a universe of vectors)
│   ├─ Basic axioms (addition, scaling)
│   └─ Examples: 2D space, 1536D space
│
├─ 03. Bases and dimensions
│   ├─ Axes as "pure directions"
│   ├─ Dimensionality = the number of axes
│   └─ Linear independence (a simple concept)
│
├─ 04. Subspaces and projections
│   ├─ Subspaces = themed regions
│   ├─ Projections: 1536D → 2D (t-SNE, UMAP)
│   └─ What is lost and what is preserved
│
├─ 05. Normalization and transformations
│   ├─ Normalization (magnitude = 1)
│   ├─ Rotations, translations
│   └─ What preserves meaning
│
├─ 06. Semantic spaces in AI
│   ├─ How LLMs organize the embedding space
│   ├─ Clusters and semantic directions
│   └─ Why `king - man + woman = queen`
│
└─ 07. Capstone exercise
    ├─ Design a 2D space for concepts
    ├─ Identify subspaces
    └─ Project and reflect

How to use this module

Recommended order:

Read Capsule 01 (this one) — 15 min to understand the roadmap
Work through Capsules 02-06 in order — Don't skip; each one builds on the previous
Take your time with the Capstone Exercise (Capsule 07) — It consolidates EVERYTHING
Review the additional resources if you want to go deeper (optional)

Recommended tools:

Paper and pencil — For drawing 2D spaces, subspaces, projections
TensorFlow's Embedding Projector (optional) — For seeing real projections of embeddings
GeoGebra 3D (optional) — For visualizing subspaces in 3D

You don't need to install anything. Everything works with pencil and paper.

Study strategy:

Session 1 (1 hour): Capsules 02-03 (fundamentals: space, bases, dimensions)

Session 2 (1 hour): Capsules 04-05 (operations: subspaces, projections, normalization)

Session 3 (30 min): Capsule 06 (application: semantic spaces in AI)

Session 4 (30 min): Capsule 07 (capstone exercise)

Don't do it all in one session. The intuition of a "space" is more abstract than that of a "vector"; it needs time to settle.

Success criteria

You'll know you completed the module successfully if you can:

✅ Explain what a vector space is (the universe where all vectors live)
✅ Draw a 2D space with axes and explain what they represent
✅ Identify a subspace in a diagram (e.g. all the points on a line)
✅ Explain why projections (t-SNE) lose information but preserve intuition
✅ Justify why you normalize embeddings (removing magnitude noise)
✅ Describe how LLMs organize meaning geometrically (clusters, directions)
✅ Reflect on how high dimensionality (1536D) allows complex relationships to be captured

Quick test: If you can explain "Why does an LLM's embedding space have 'semantic structure'?" without hesitating, you're ready for Module 3.

Common mistakes when learning vector spaces

Mistake 1: "A vector space is just a set of vectors"

Reality: A vector space has structure: dimensions, axes, closed operations (addition, scaling). It isn't just any set of vectors; it has specific geometric and algebraic properties.

Mistake 2: "I can't understand spaces without seeing matrices"

Reality: You can understand spaces conceptually without matrices. Matrices are tools for computing in spaces; they aren't the spaces themselves. This module uses pure geometry, not matrices.

Mistake 3: "Projections (t-SNE) are the real space"

Reality: 2D projections are approximations of the 1536D space. They preserve local closeness but lose global information. They're useful for visualizing, not for reasoning mathematically.

Mistake 4: "Normalization changes the meaning of the vector"

Reality: Normalization does not change the direction (the "kind of concept"). It only changes the magnitude. If the magnitude is noise (frequencies), normalizing improves the representation.

Complementary resources

During the module:

3Blue1Brown: "Linear transformations and matrices" — A 10-minute video on linear transformations (rotations, projections). It complements Capsule 05.
Khan Academy: "Vector spaces" — A series of videos on vector spaces. Useful if you want more formal examples.

After the module:

Embedding Projector (TensorFlow) — A web tool for visualizing real embeddings in 3D with t-SNE/UMAP. To explore after Capsule 04.
Distill.pub: "Visualizing High-Dimensional Space" — An interactive article on how to understand t-SNE and projections. For after Capsule 04.

Next Module

Module 3: Similarity and Distance — You'll formalize how to measure "closeness" between vectors:

Euclidean distance vs cosine
When to use each metric
Why cosine is superior in high dimensions
How to compute similarity efficiently

Prerequisite: Having completed Modules 1-2. If you don't understand what a vector space is or how it's structured, Module 3 will be confusing (it talks about "angles between vectors" and "normalization", concepts from Module 2).

FAQs

Q: Do I need linear algebra for this module?
A: No. This module uses intuitive geometry, not formal algebra. If you completed Module 1, you have everything you need.

Q: Can I skip this module if I already know vectors (Module 1)?
A: Not recommended. Module 1 teaches you individual vectors; Module 2 teaches you how all the vectors form a structured space. That intuition is crucial for Modules 3-8.

Q: Does this module cover PCA, t-SNE, UMAP technically?
A: No. We'll cover them conceptually (what they do, why they're useful) without implementation. The code comes in later guides.

Q: How much time should I spend on it?
A: 2-2.5 hours spread across 2-4 sessions. Don't do it all in one go; the intuition of a "space" needs time to settle.

Q: What if I don't understand Capsule 03 (bases and dimensions)?
A: It's the most abstract capsule. If you get stuck, continue with Capsules 04-06 (applications) and come back to 03 afterward. Sometimes seeing the applications helps you understand the theory.

Q: Are there exercises with answers?
A: Yes. Every capsule has exercises with solutions. The Capstone Exercise (Capsule 07) is the most complete.

Final notes

This module extends your intuition from "vectors" to "spaces of vectors". It's a conceptual leap (from individual points to entire universes), but it's essential for understanding how LLMs organize meaning and how semantic search navigates that meaning.

The key intuition to consolidate:

"The embedding space isn't a random set of points; it's a structured geometric universe where regions capture topics (animals, vehicles) and directions capture relationships (gender, size). Semantic search navigates that universe using distance/similarity."

If you only do one thing in this module: Complete the Capstone Exercise (Capsule 07). Designing a 2D space for concepts, identifying subspaces, and reflecting on how it scales to 1536D will consolidate EVERYTHING.

Enjoy the module! The geometry of vector spaces is elegant and powerful. When you finish, you'll see embeddings not as "lists of numbers" but as "points in a geometric universe that reflects meaning".

Next capsule: 02-what-is-a-vector-space.md — The intuitive definition, basic axioms, 2D and 1536D examples.

← Previous7. Capstone Exercise: Mapping Concepts to Vectors Next →2. What Is a Vector Space?