5. Basic Vector Operations

Capsule overview

In this capsule you're going to learn the three fundamental vector operations: addition, subtraction and scaling. We're not going to get into formal linear algebra or matrix multiplication; only the basic operations that let you understand how vectors are manipulated and combined in AI.

These operations are the foundation of everything that follows: in semantic search, queries are compared against documents (an implicit subtraction to measure distance); in embedding manipulation, you can perform operations like Vector("king") - Vector("man") + Vector("woman") ≈ Vector("queen") (addition and subtraction); in normalization, you scale vectors so they have magnitude 1.

When you finish, you'll know how to add, subtract and scale vectors (with geometric intuition and numerical examples), and you'll understand why those operations are useful in AI. This capsule is 100% conceptual; you won't implement code, but you'll gain the intuition you need to understand how language models manipulate embeddings internally.

Vector addition

Definition:

Adding two vectors means adding their coordinates axis by axis.

Formula:

Vector A + Vector B = [a₁ + b₁, a₂ + b₂, ..., aₙ + bₙ]

Example in 2D:

Vector A = [3, 2]
Vector B = [1, 1]

A + B = [3+1, 2+1] = [4, 3]

Geometric interpretation:

Adding vectors is like combining displacements.

Visualization:

      ↑ Y
      |
    3 |         •(4,3) ← Result A+B
      |        ╱|
    2 |   •───╱ | ← B = [1,1]
      |   |╱    |
    1 | A |     |
      |   |     |
    0 •───────────────→ X
      0   1  2  3  4

Step by step:

1. Draw the vector A [3, 2] from the origin
2. From the end point of A (3, 2), draw the vector B [1, 1]
3. The result is the vector from the origin to the final point: [4, 3]

Physical analogy: If you walk 3 blocks east and 2 north (A), and then 1 block east and 1 north (B), in total you've walked 4 blocks east and 3 north (A+B).

Example in 3D:

Vector A = [2, 3, 1]
Vector B = [1, -1, 2]

A + B = [2+1, 3+(-1), 1+2] = [3, 2, 3]

Interpretation: Combining two 3D displacements. If A moves you to (2, 3, 1) and then B moves you (1, -1, 2) from there, the total displacement from the origin is (3, 2, 3).

Properties of addition:

1. Commutative: A + B = B + A

[3, 2] + [1, 1] = [4, 3]
[1, 1] + [3, 2] = [4, 3]

Order doesn't matter.

2. Associative: (A + B) + C = A + (B + C)

([3, 2] + [1, 1]) + [0, 2] = [4, 3] + [0, 2] = [4, 5]
[3, 2] + ([1, 1] + [0, 2]) = [3, 2] + [1, 3] = [4, 5]

You can group them in any order.

3. Identity element: A + [0, 0] = A

[3, 2] + [0, 0] = [3, 2]

Adding the zero vector changes nothing.

Addition in high dimensions:

Vector A (1536D): [0.23, -0.45, 0.12, ..., -0.34]
Vector B (1536D): [0.02, 0.05, -0.01, ..., 0.06]

A + B = [0.23+0.02, -0.45+0.05, 0.12+(-0.01), ..., -0.34+0.06]
      = [0.25, -0.40, 0.11, ..., -0.28]

Interpretation: Combining two "meanings" in 1536D space. For example, if A is the embedding of "king" and B is a small adjustment, A+B is a vector "near king" with some variation.

In AI: Adding embeddings is used for operations like:

• Combining contexts (e.g. document + query)
• Adjusting embeddings (e.g. base embedding + correction)

Vector subtraction

Definition:

Subtracting two vectors means subtracting their coordinates axis by axis.

Formula:

Vector A - Vector B = [a₁ - b₁, a₂ - b₂, ..., aₙ - bₙ]

Example in 2D:

Vector A = [5, 3]
Vector B = [2, 1]

A - B = [5-2, 3-1] = [3, 2]

Geometric interpretation:

Subtracting vectors gives you the displacement from B to A.

Visualization:

      ↑ Y
      |
    3 |          •A(5,3)
      |         ╱|
    2 |        ╱ | ← A-B = [3,2]
      |       ╱  |
    1 |   •──────| B(2,1)
      |
    0 •───────────────→ X
      0  1  2  3  4  5

Interpretation: A - B is the vector that goes from point B to point A.

Analogy: If you're at (2, 1) (point B) and you want to get to (5, 3) (point A), the displacement you need is [3, 2] (A - B).

Example in 3D:

Vector A = [4, 5, 2]
Vector B = [1, 2, -1]

A - B = [4-1, 5-2, 2-(-1)] = [3, 3, 3]

Interpretation: To go from B to A, move 3 units on each axis.

Using subtraction to measure difference:

In AI, subtraction is used to calculate how different two vectors are:

Vector "dog" = [0.23, -0.45, ..., -0.34]
Vector "cat" = [0.25, -0.43, ..., -0.32]

Difference = dog - cat = [0.23-0.25, -0.45-(-0.43), ..., -0.34-(-0.32)]
           = [-0.02, -0.02, ..., -0.02]

If the difference has very small values (close to 0), the vectors are very similar.

The famous example: Vector arithmetic

The most famous operation in embeddings:

Vector("king") - Vector("man") + Vector("woman") ≈ Vector("queen")

Step by step:

1. Vector("king") - Vector("man") = "royalty" without the masculine gender
2. Adding Vector("woman") = "royalty" with the feminine gender
3. Result ≈ Vector("queen")

Interpretation: The subtraction captures "which direction in the space represents gender". Then you add "woman" to apply that change to "king".

Simplified visualization (2D):

      Royalty ↑
              |
    "queen" • | • "king"
              |
    "woman" • | • "man"
              |───────────→ Gender (masculine-feminine)

The "gender direction" is king - man ≈ queen - woman. Therefore king - man + woman ≈ queen.

This works because embeddings capture geometric relationships. In high dimensions (1536D), there are "directions" that represent concepts like gender, size, time, etc.

Scaling vectors (multiplication by a scalar)

Definition:

Multiplying a vector by a number (a scalar) multiplies each coordinate:

Formula:

k × Vector A = [k × a₁, k × a₂, ..., k × aₙ]

Example in 2D:

Vector A = [3, 2]
Scalar k = 2

2 × A = [2×3, 2×2] = [6, 4]

Geometric interpretation:

Scaling a vector changes its magnitude but not its direction.

Visualization:

      ↑ Y
      |
    4 |           •(6,4) ← 2×A (twice as long)
      |          ╱
    2 |     •───╱ ← A = [3,2]
      |    ╱
    0 •───────────────→ X
      0  2  4  6

Interpretation:

• 2 × A points in the same direction as A (northeast)
• But it has double the magnitude (twice as long)

Analogy: If the vector A represents "walk 3 steps east, 2 north", then 2 × A is "walk 6 steps east, 4 north". Same direction, double the distance.

Scaling with negative numbers:

Vector A = [3, 2]
Scalar k = -1

-1 × A = [-3, -2]

Interpretation: It flips to the opposite direction (from northeast to southwest).

Visualization:

      ↑ Y
      |
    2 |     •(3,2) ← A
      |    ╱
    0 •───────────→ X
      | ╲
   -2 |  •(-3,-2) ← -A (opposite)

Rule:

• k > 1: Lengthens the vector (same direction, greater magnitude)
• 0 < k < 1: Shortens the vector (same direction, smaller magnitude)
• k = 0: The result is the zero vector [0, 0]
• k < 0: Reverses the direction

Example in 3D:

Vector A = [2, 3, 1]
Scalar k = 0.5

0.5 × A = [0.5×2, 0.5×3, 0.5×1] = [1, 1.5, 0.5]

Interpretation: A vector with half the magnitude, same direction.

Scaling in high dimensions:

Vector A (1536D): [0.23, -0.45, 0.12, ..., -0.34]
Scalar k = 2

2 × A = [0.46, -0.90, 0.24, ..., -0.68]

Use in AI: Scaling embeddings is used for:

• Normalization: Making all vectors have magnitude 1 (dividing by their magnitude)
• Weight adjustments: Giving more or less "importance" to an embedding in combinations

Combining operations

You can combine addition, subtraction and scaling in a single expression:

Example:

A = [3, 2]
B = [1, 1]
C = [0, 2]

Result = 2×A - B + 0.5×C
       = [6, 4] - [1, 1] + [0, 1]
       = [6-1+0, 4-1+1]
       = [5, 4]

Interpretation: Combining multiple displacements with different "weights" (scalars).

In AI: Operations like this are used in language models to combine contexts with different importances:

final_context = 0.7 × document_embedding + 0.3 × query_embedding

That gives more "weight" to the document than to the query in the combined embedding.

Normalization: Making unit vectors

Normalization is scaling a vector so that its magnitude is 1 (a unit vector).

Formula:

Normalized vector = Vector / Magnitude(Vector)

Example in 2D:

Vector A = [3, 4]
Magnitude = √(3² + 4²) = √25 = 5

A normalized = [3/5, 4/5] = [0.6, 0.8]

Check: Magnitude([0.6, 0.8]) = √(0.36 + 0.64) = √1 = 1 ✓

Interpretation: The normalized vector points in the same direction as A, but has magnitude 1.

Visualization:

      ↑ Y
      |
    4 |       •A(3,4) ← Magnitude = 5
      |      ╱|
    3 |     ╱ |
      |    ╱  |
    2 |   ╱   |
      |  ╱    |
    1 | •(0.6,0.8) ← A normalized, magnitude = 1
      |╱
    0 •───────────────→ X
      0  1  2  3

Both point in the same direction (northeast), but the normalized one has magnitude 1.

Why normalize?

In AI, normalization is crucial because:

1. It removes the effect of magnitude:

If you use Euclidean distance, vectors with greater magnitude can skew the results. Normalizing makes only the direction matter (angular similarity), not the "size" of the vector.

Example:

• Vector A = [3, 4] (magnitude = 5)
• Vector B = [6, 8] (magnitude = 10, same direction as A)

Without normalizing, they look different (different distances). Normalized:

• A normalized = [0.6, 0.8]
• B normalized = [0.6, 0.8]

They're identical → same direction → perfectly similar.

2. Numerical stability:

In systems with millions of vectors, keeping magnitudes consistent (all = 1) avoids overflow or underflow problems in calculations.

3. Comparison with cosine:

When you use cosine similarity (which you'll see in Module 3), it normalizes internally. If you pre-normalize your vectors, the cosine reduces to a simple dot product (more efficient).

Cosine formula:

cos(A, B) = (A · B) / (||A|| × ||B||)

If A and B are normalized (||A|| = ||B|| = 1):
cos(A, B) = A · B  (a simple dot product)

That's why many vector databases (Pinecone, Weaviate) normalize embeddings automatically before storing them.

Operations in high dimensions: Practical summary

Key point: The operations work exactly the same in high dimensions. There are just more coordinates to process.

Exercises

Exercise 1: Adding 2D vectors

Calculate:

a) [2, 3] + [1, -1]
b) [5, 0] + [0, 5]
c) [3, 2] + [-3, -2]

[2+1, 3+(-1)] = [3, 2]

[5+0, 0+5] = [5, 5]

[3+(-3), 2+(-2)] = [0, 0]

Exercise 2: Subtracting 2D vectors

Calculate:

a) [5, 4] - [2, 1]
b) [0, 5] - [0, 2]
c) [3, 2] - [3, 2]

[5-2, 4-1] = [3, 3]

[0-0, 5-2] = [0, 3]

[3-3, 2-2] = [0, 0]

Exercise 3: Scaling vectors

Calculate:

a) 3 × [2, 1]
b) 0.5 × [4, 6]
c) -1 × [3, 2]

[3×2, 3×1] = [6, 3]

[0.5×4, 0.5×6] = [2, 3]

[-1×3, -1×2] = [-3, -2]

Exercise 4: Normalization

Normalize the vector [3, 4].

Magnitude = √(3² + 4²) = √(9 + 16) = √25 = 5

Normalized vector = [3/5, 4/5] = [0.6, 0.8]

Magnitude([0.6, 0.8]) = √(0.36 + 0.64) = √1 = 1 ✓

Exercise 5: Combining operations

Calculate: 2 × [3, 1] - [1, 2] + 0.5 × [4, 0]

2 × [3, 1] = [6, 2]
[1, 2] = [1, 2]
0.5 × [4, 0] = [2, 0]

[6, 2] - [1, 2] + [2, 0]
= [6-1, 2-2] + [2, 0]
= [5, 0] + [2, 0]
= [7, 0]

Exercise 6: Vector arithmetic (conceptual)

Suppose that in an embedding space:

Vector("king") ≈ [10, 8]  (simplified to 2D for illustration)
Vector("man") ≈ [5, 2]
Vector("woman") ≈ [5, 8]

Calculate: Vector("king") - Vector("man") + Vector("woman") ≈ ?

[10, 8] - [5, 2] + [5, 8]
= [10-5, 8-2] + [5, 8]
= [5, 6] + [5, 8]
= [10, 14]

Troubleshooting

Problem 1: "I don't understand why A + B ≠ B + A visually"

Symptom: When you draw A + B and B + A, you get confused because it seems like order matters.

Solution: Order does NOT matter for the final result. Both give the same resulting vector (the same end point), even though the visual path is different:

A + B:
1. Draw A from the origin
2. From the end of A, draw B
3. You arrive at the end point

B + A:
1. Draw B from the origin
2. From the end of B, draw A
3. You arrive at the same end point

Analogy: If you walk 3 blocks east and then 2 north, you end up in the same place as if you walk 2 north and then 3 east. The order of the steps doesn't matter; the final destination is the same.

Problem 2: "Why is A - B different from B - A?"

Symptom: You get confused because subtraction is NOT commutative.

Solution: A - B ≠ B - A (except when A = B).

Example:

• A - B = [5, 3] - [2, 1] = [3, 2] (the displacement from B to A)
• B - A = [2, 1] - [5, 3] = [-3, -2] (the displacement from A to B, the opposite)

Interpretation: A - B goes "from B toward A"; B - A goes "from A toward B" (the opposite direction).

Rule: B - A = -(A - B) (opposite vectors).

Problem 3: "When do I use addition vs subtraction?"

Symptom: You don't know when to use each operation.

Practical guide:

Use addition when:

• You want to combine two vectors (e.g. document + query)
• You want to accumulate displacements (e.g. A + B + C)

Use subtraction when:

• You want to measure the difference between vectors (e.g. distance = ||A - B||)
• You want to extract a conceptual "direction" (e.g. king - man = "royalty without the masculine gender")

Problem 4: "Why normalize if it changes the coordinates?"

Symptom: You're afraid of losing information by normalizing.

Solution: Normalization only changes magnitude, not direction. In many applications (e.g. cosine similarity), only the direction matters (how alike the vectors are), not the magnitude.

Example:

• [3, 4] (magnitude = 5) and [6, 8] (magnitude = 10) have the same direction.
• Normalized: both [0.6, 0.8] → identical.

If what matters is "which concepts are alike" (direction), normalizing removes the noise of inconsistent magnitudes.

When NOT to normalize: If the magnitude carries meaning (e.g. the frequency of a word in TF-IDF), don't normalize without thinking. But in modern embeddings (OpenAI, BERT), the magnitude is usually noise and gets normalized.

Connection to semantic search

The operations you saw here are the foundation of semantic search:

1. Generate embeddings (1536D vectors):

Document: "The dog is a domestic animal"
→ Vector doc = [0.23, -0.45, ..., -0.34]

Query: "Which animals are pets?"
→ Vector query = [0.25, -0.43, ..., -0.32]

2. Calculate the difference (subtraction):

Difference = doc - query = [-0.02, -0.02, ..., -0.02]

If the difference is small (many values close to 0), the vectors are similar.

3. Calculate the distance (the magnitude of the difference):

Distance = ||doc - query|| = √((-0.02)² + (-0.02)² + ... + (-0.02)²)
         ≈ small

A small distance → nearby vectors → similar concepts → a relevant document.

4. Normalize (optional):

Many vector databases normalize automatically so that only the direction matters (angular similarity, not absolute distance).

Result: The document with the smallest distance (or the highest cosine, which you'll see in Module 3) is the most relevant one for the query. All of this is based on the operations you saw here (addition, subtraction, scaling).

Summary

In one sentence: The basic vector operations (addition = combine, subtraction = difference, scaling = change magnitude, normalization = make magnitude 1) work the same in 2D, 3D and high dimensions, and they're the foundation of how embeddings are manipulated in AI.

Key points:

• Addition: A + B = add coordinates axis by axis → combine displacements
• Subtraction: A - B = subtract coordinates axis by axis → the difference from B to A
• Scaling: k × A = multiply each coordinate by k → change magnitude, same direction
• Normalization: A / ||A|| = divide by the magnitude → a vector with magnitude 1 (only direction matters)
• Properties: Addition is commutative and associative; subtraction is NOT commutative
• Vector arithmetic: king - man + woman ≈ queen works thanks to geometric operations in high dimensions
• Use in AI: Combining embeddings (addition), measuring difference (subtraction), normalizing for comparison (scaling)
• High dimensions: The operations work the same (just more coordinates to process)

Connection to the next capsule

In Capsule 06 (Why vectors in AI), you'll see how everything you learned (what a vector is, 2D/3D visualization, high dimensions, operations) applies specifically in AI: why language models use vectors to represent meaning, how embeddings are trained, and why "nearby vectors = similar meanings" is the foundation of semantic search and RAG. The operations in this capsule (addition, subtraction) are the foundation of embedding manipulation in language models.

Additional resources

1. 3Blue1Brown: "Linear combinations, span, and basis vectors" — A 10-minute video on linear combinations (vector addition and scaling). Excellent visualizations. In English.

2. Khan Academy: "Vector addition and subtraction" — A series of short videos on geometric addition and subtraction. In English with subtitles.

3. Jay Alammar: "The Illustrated Word2vec" — A blog with diagrams on embedding operations (king - man + woman = queen). Excellent for seeing the operations in a real context. In English.

4. DeepLearning.AI: "Vector Embeddings Visualized" — Resources on embedding visualization and vector operations in NLP. In English.

5. Pinecone: "Vector Operations" — A blog aimed at vector databases that explains common operations (addition, normalization, distance). In English.

6. Better Explained: "Vector Calculus: Understanding the Dot Product" — An intuitive explanation of the dot product (not covered in this capsule, but useful for going deeper). In English.

Next capsule: 06-why-vectors-in-ai.md — Why do language models use vectors? How embeddings are trained, why nearby vectors = similar meanings, and the direct connection to semantic search and RAG.