Module 2: Vector Spaces
5. Normalization and Transformations
Capsule overview
In this capsule you're going to understand normalization (why you make all vectors have magnitude 1) and transformations of the space (rotations, translations). These operations are fundamental in semantic search: normalization is the basis of cosine similarity.
Normalization: Magnitude = 1
Normalizing a vector means scaling it so its magnitude is exactly 1:
Normalized vector = Vector / Magnitude(Vector)
Example in 2D:
Original vector: [3, 4]
Magnitude: √(9+16) = 5
Normalized vector: [3/5, 4/5] = [0.6, 0.8]
Magnitude of the normalized one: √(0.36+0.64) = 1 ✅
Why normalize in AI?
Reason 1: Removing magnitude noise
In embeddings, magnitude can capture:
- The frequency of words in the training corpus
- The length of the text (long documents → large vectors)
That's noise if all that matters is the "kind of concept".
Normalizing removes that noise:
[3, 4](magnitude 5) and[6, 8](magnitude 10) have the same direction- Normalized: both are
[0.6, 0.8]→ identical
Reason 2: It enables cosine similarity
Cosine similarity measures the angle between vectors:
cos(A, B) = (A · B) / (||A|| × ||B||)
If A and B are normalized (magnitude = 1):
cos(A, B) = A · B (a simple dot product)
Result: A more efficient computation, and only direction matters.
Reason 3: Numerical stability
With millions of vectors, keeping magnitudes consistent (all = 1) avoids overflow/underflow problems.
Transformations of the space
1. Translation (moving the origin)
This is NOT a valid transformation in vector spaces (it doesn't preserve the origin).
In embeddings, the origin has meaning (the zero vector = the absence of a concept). Translating would change that meaning.
2. Rotation (changing the orientation of the axes)
Rotating the space does NOT change the distances or the angles between vectors.
Use in AI: Some models rotate the embedding space internally during fine-tuning. The geometric relationships are preserved.
3. Scaling (uniformly enlarging/shrinking)
Multiplying all the vectors by the same constant:
New space: k × Vector for every vector
Preserves: Angles, the proportions between distances
Changes: Absolute magnitudes
Use: Normalization is a special case (scaling to magnitude = 1).
Which operations preserve meaning
| Operation | Preserves distances | Preserves angles | Use in AI |
|---|---|---|---|
| Normalization | No | Yes | Cosine similarity |
| Rotation | Yes | Yes | Fine-tuning, alignment |
| Uniform scaling | No (absolute ones) | Yes | Normalization |
| Translation | — | — | Not used (it breaks the origin) |
Summary
Key points:
- Normalization: Scaling vectors to magnitude = 1
- Why: It removes magnitude noise, enables cosine, and gives numerical stability
- Rotation: Preserves distances and angles (used in fine-tuning)
- Scaling: Changes magnitude but preserves angles
- Translation: Not used (it breaks the meaning of the origin)
Next capsule: 06-semantic-spaces-in-ai.md — How LLMs organize meaning geometrically.