Module 2: Vector Spaces
2. What Is a Vector Space?
Capsule overview
In this capsule you're going to understand what a vector space is: the "geometric universe" where all vectors live. If in Module 1 you learned what an individual point is, here you learn what the complete map containing all the points is.
A vector space isn't just "a pile of loose vectors". It's a structured system with rules: you can add vectors, scale them, and the result always stays inside the same space. That structure is what makes semantic search work: the space has geometric properties (distances, angles, regions) that capture meaning.
When you finish, you'll understand why OpenAI's 1536D "embedding space" isn't a random set of numbers, but a space with mathematical structure where geometry reflects semantics.
Intuitive definition: The universe of vectors
A vector space is:
A set of vectors where you can add any pair of vectors and scale any vector, and the result is always inside the same set.
Analogy 1: A city with streets
Imagine a city with a perfect grid:
- Each location is a vector (coordinates from the origin)
- You can "add" displacements: go 3 blocks east + 2 north = arrive at a point in the city
- You can "scale": go 2× farther in the same direction = another location in the city
- You never leave the city (there's always a valid location)
That city is a 2D vector space.
Analogy 2: An infinite spreadsheet
Imagine a spreadsheet with 1536 columns (and infinite rows):
- Each row is a vector (1536 numbers)
- You can add two rows: each cell adds with its counterpart
- You can multiply a row by a number: each cell gets multiplied
- The result is always a valid row (another combination of 1536 numbers)
That spreadsheet is a 1536D vector space.
Fundamental properties (without excessive formalism)
A vector space has two closed operations (the result always stays inside the space):
1. Vector addition
If A and B are in the space → A + B is also in the space
Example in 2D:
A = [3, 2] (it's in the space)
B = [1, 1] (it's in the space)
A + B = [4, 3] (it's also in the space)
Interpretation: If two words have embeddings in the space, their "combination" (the sum) is also a valid point in the space.
2. Multiplication by a scalar
If A is in the space and k is a number → k × A is also in the space
Example in 2D:
A = [3, 2] (it's in the space)
k = 2
2 × A = [6, 4] (it's also in the space)
Interpretation: You can "lengthen" or "shorten" a vector (change its magnitude) and it's still valid in the space.
3. The zero vector
The space always contains the zero vector [0, 0, ..., 0]:
Origin = [0, 0] in 2D
Origin = [0, 0, 0] in 3D
Origin = [0, 0, ..., 0] in 1536D
Interpretation: The zero vector is the "reference point" (the origin). In embeddings, the zero vector doesn't represent any real word (models learn non-zero vectors).
Examples of vector spaces
2D space (the Cartesian plane)
Vectors: All the pairs [x, y] where x, y are real numbers
Examples:
- [3, 2] ✅
- [-5, 7] ✅
- [0, 0] ✅
- [π, √2] ✅
Operations:
[3, 2] + [1, 1] = [4, 3] ✅ (inside the space)
2 × [3, 2] = [6, 4] ✅ (inside the space)
Properties:
- Dimensionality: 2D (2 axes: X, Y)
- Size: Infinite (infinitely many possible points)
- Visualizable: Yes (you can draw it)
3D space (physical space)
Vectors: All the triples [x, y, z]
Examples:
- [2, 3, 1] ✅
- [0, 0, 5] ✅
- [-1, 2, -3] ✅
Properties:
- Dimensionality: 3D (3 axes: X, Y, Z)
- Size: Infinite
- Visualizable: Partially (isometric projections)
1536D space (OpenAI embeddings)
Vectors: All the arrays of 1536 real numbers
Examples:
- [0.23, -0.45, 0.12, ..., -0.34] ✅ (1536 numbers)
- [0.25, -0.43, 0.10, ..., -0.32] ✅ (1536 numbers)
Operations (the same as in 2D, just more coordinates):
A + B = [a₁+b₁, a₂+b₂, ..., a₁₅₃₆+b₁₅₃₆] ✅
k × A = [k×a₁, k×a₂, ..., k×a₁₅₃₆] ✅
Properties:
- Dimensionality: 1536D (1536 axes)
- Size: Infinite (infinitely many possible points)
- Visualizable: Not directly (you need projections to 2D/3D)
What is NOT a vector space
Counterexample 1: Only positive numbers
Set: Only vectors [x, y] where x ≥ 0, y ≥ 0
Is it a vector space? ❌ No.
Why: If you scale by -1, you leave the set:
A = [3, 2] (it's in the set)
-1 × A = [-3, -2] (it is NOT in the set)
Conclusion: It isn't closed under scaling → it isn't a vector space.
Counterexample 2: Only vectors with magnitude = 1
Set: Only vectors [x, y] with √(x² + y²) = 1
Is it a vector space? ❌ No.
Why: If you add two unit vectors, the magnitude changes:
A = [1, 0] (magnitude = 1)
B = [0, 1] (magnitude = 1)
A + B = [1, 1] (magnitude = √2 ≠ 1)
Conclusion: It isn't closed under addition → it isn't a vector space.
Note: Normalized vectors (magnitude = 1) are useful in AI, but they do NOT form a vector space on their own. The complete vector space includes all vectors (of any magnitude).
Vector spaces in AI: The embedding space
When you use an embedding model (OpenAI, BERT, etc.), each text is mapped to a point in a vector space:
Text: "The dog is a domestic animal"
↓ Embedding model
Vector: [0.23, -0.45, 0.12, ..., -0.34] (1536D)
That vector lives in a 1536D vector space:
- You can add it to another vector → a valid result in the space
- You can scale it → a valid result in the space
- You can measure its distance to other vectors (they all live in the same space)
Why it matters: Because the space has geometric structure (distances, angles, regions) that captures semantic structure (similarity of meaning).
Useful properties of vector spaces
1. Linearity: Operations distribute
k × (A + B) = k×A + k×B
Example:
2 × ([3,2] + [1,1]) = 2×[4,3] = [8,6]
2×[3,2] + 2×[1,1] = [6,4] + [2,2] = [8,6]
Use in AI: It lets you manipulate embeddings algebraically (e.g. king - man + woman).
2. Associativity: The grouping order doesn't matter
(A + B) + C = A + (B + C)
Example:
([3,2] + [1,1]) + [0,2] = [4,3] + [0,2] = [4,5]
[3,2] + ([1,1] + [0,2]) = [3,2] + [1,3] = [4,5]
3. Commutativity (addition): The order doesn't matter
A + B = B + A
Example:
[3,2] + [1,1] = [4,3]
[1,1] + [3,2] = [4,3]
4. Existence of an opposite: Every vector has a negative
For every A, there exists -A such that A + (-A) = [0,0,...,0]
Example:
A = [3, 2]
-A = [-3, -2]
A + (-A) = [0, 0] (the zero vector)
Comparison: Set vs Vector Space
| Aspect | A simple set | A vector space |
|---|---|---|
| Definition | A collection of objects | A set with operations (addition, scalar) |
| Structure | Has none | Has one (closed under addition and scaling) |
| Operations | Not defined | Addition and scaling are always valid |
| Example | {dog, cat, car} | All the 2D vectors |
Conclusion: A vector space is a set with algebraic and geometric structure.
Visualization: 2D space as an example
Y ↑
|
10 |
|
5 | • Point [3, 5]
|
0 |•──────────────→ X
0 5 10
This entire plane is a vector space:
- You can place points anywhere (infinitely many points)
- Add two points → another valid point
- Scale a point → another valid point
- There's structure: axes (X, Y), origin (0, 0), distances, angles
Connection to semantic search
The problem:
If embeddings did NOT live in a structured vector space:
- You couldn't add vectors (there'd be no "king - man + woman")
- You couldn't measure distances consistently
- There'd be no geometry to reflect semantics
The solution:
Embeddings live in a 1536D vector space:
- Guaranteed structure (addition and scaling are always valid)
- Well-defined geometry (distances, angles)
- Algebraic properties (linearity, associativity)
Result: You can use geometry (distance, similarity) to measure semantics (how alike two texts are).
Exercises
Exercise 1: Is it a vector space?
For each set, determine whether it's a vector space:
a) All the 3D vectors
b) Only the 2D vectors where x = y
c) Only the 2D vectors with x ≥ 0
See solution
a) All the 3D vectors: ✅ Yes
- Addition:
[1,2,3] + [4,5,6] = [5,7,9]✅ - Scalar:
2 × [1,2,3] = [2,4,6]✅ - Zero vector:
[0,0,0]✅
b) Only the 2D vectors where x = y: ✅ Yes (it's a subspace)
Examples: [1,1], [2,2], [3,3]
Addition: [1,1] + [2,2] = [3,3] ✅ (x=y holds)
Scalar: 2 × [1,1] = [2,2] ✅ (x=y holds)
Zero vector: [0,0] ✅
c) Only the 2D vectors with x ≥ 0: ❌ No
Counterexample:
A = [3, 2] (x ≥ 0 ✅)
-1 × A = [-3, -2] (x < 0 ❌)
It isn't closed under scaling → it isn't a vector space.
Exercise 2: Operations in the space
Given these vectors in a 2D space:
A = [2, 3]
B = [-1, 4]
Verify that these operations give results inside the space:
a) A + B
b) 3 × A
c) A - B (which is A + (-1)×B)
See solution
a) A + B:
[2, 3] + [-1, 4] = [1, 7] ✅ (inside the space)
b) 3 × A:
3 × [2, 3] = [6, 9] ✅ (inside the space)
c) A - B:
[2, 3] - [-1, 4] = [2, 3] + [1, -4] = [3, -1] ✅ (inside the space)
Conclusion: All the operations give valid results → this confirms it's a vector space.
Exercise 3: The embedding space
If an embedding model maps texts to 768D vectors:
a) Do those vectors form a vector space?
b) Can you add two embeddings?
c) Does the result of adding two embeddings represent "something" semantically?
See reflection guide
a) Do they form a vector space? ✅ Yes
All the 768D vectors form a vector space (closed under addition and scaling).
b) Can you add them? ✅ Yes
Mathematically, you can always add two vectors from the space. The result is another valid vector in the space.
c) Does it make semantic sense?
It depends:
- If you add embeddings of related concepts ("dog" + "cat"), the result can end up near a generic concept ("pet")
- If you add embeddings of unrelated concepts ("dog" + "car"), the result may have no clear interpretation
Practical use: In models, embeddings are sometimes averaged to represent a "combined concept". E.g. averaging the embeddings of the sentences in a document to get an embedding of the whole document.
Note: Direct addition doesn't always make semantic sense. The useful operations are usually king - man + woman (capturing directions) or weighted averages.
Summary
In one sentence: A vector space is a structured set of vectors where addition and scaling always give results inside the same set, with algebraic properties (linearity, associativity) that allow consistent geometric manipulation.
Key points:
- Vector space: A set + closed operations (addition, scalar)
- Properties: Linearity, associativity, commutativity (addition), the zero vector, opposites
- Examples: 2D, 3D, 1536D space (all the possible vectors)
- Non-examples: Only positives, only magnitude = 1 (not closed)
- In AI: Embeddings live in a 768D-1536D vector space
- The advantage: Structure guarantees consistent geometry → consistent semantics
- Semantic search: Uses the geometry of the space (distances, angles) to measure semantics
Connection to the next capsule
In Capsule 03 (Bases and dimensions), you'll see how the space is structured internally: what the "axes" (bases) are, why the space has 1536 dimensions, and what "linear independence" means. If the vector space is the "universe", the bases are the "coordinate system" we use to locate points.
Additional resources
-
3Blue1Brown: "What is a vector space?" — A 12-minute video on vector spaces with visualizations. In English.
-
Khan Academy: "Vector spaces" — A series on vector spaces with exercises. In English.
-
Wikipedia: "Vector space" — The formal definition with the complete axioms. For going deeper. In English.
Next capsule: 03-basics-and-dimensions.md — What are the "axes"? Why 1536D? Linear independence, conceptually.