6. Semantic Spaces in AI: How LLMs Organize Meaning

Capsule overview

In this capsule you're going to see how LLMs organize the embedding space: semantic clusters (all the animals together), semantic directions (gender, size), and why operations like king - man + woman = queen work. This is the direct application of everything you saw in the module.

The embedding space as a semantic map

When a model (OpenAI, BERT, GPT) learns embeddings, the resulting space has semantic structure:

1. Clusters (groupings)

Related concepts group together in regions:

Region 1 (Domestic animals):
- "dog", "cat", "hamster"

Region 2 (Wild animals):
- "lion", "tiger", "elephant"

Region 3 (Vehicles):
- "car", "airplane", "boat"

Conceptual visualization (simplified 2D):

        ↑
        |
   •cat •dog       (Cluster: domestic animals)
        |
        |     •lion •tiger  (Cluster: wild animals)
        |
        |────────────────→
        |
        |  •car •airplane  (Cluster: vehicles)
        |

2. Semantic directions

There are "directions" in the space that capture relationships:

The gender direction:

Vector("king") - Vector("man") ≈ Vector("queen") - Vector("woman")

That "direction" captures the concept of "masculine → feminine gender".

The size direction:

Vector("elephant") - Vector("mouse") ≈ "the size direction"

The tense direction:

Vector("run") - Vector("ran") ≈ "the present → past direction"

Why `king - man + woman = queen` works

Step 1: king - man captures "royalty without the masculine gender"

king = [royalty + masculine]
man = [masculine]

king - man ≈ [royalty]

Step 2: + woman adds "the feminine gender"

[royalty] + [feminine] ≈ [royalty + feminine] ≈ queen

Geometry:

        Royalty ↑
                |
    queen •     | • king
                |
    woman •     | • man
                |───────→ Gender

The "gender direction" is horizontal. The operation king - man + woman navigates that direction.

Semantic subspaces

In the 1536D embedding space, there are subspaces that capture topics:

The "animals" subspace:

Dimensions that capture "animalness", "size", "habitat"
All the animals have high values on those dimensions

The "emotions" subspace:

Dimensions that capture "valence" (positive/negative), "intensity"
All the emotions have high values on those dimensions

Use in semantic search: When you search for "domestic animal", the system searches within the animal subspace.

How that structure forms

During training:

The model sees millions of sentences
It adjusts vectors so that words in similar contexts end up close together
Result: The structure emerges naturally (it isn't programmed by hand)

Analogy: It's like learning a language. You don't memorize rules; after hearing millions of sentences, your brain organizes words by topic and relationship automatically.

Applications in AI Engineering

1. Semantic search with thematic filters

Query: "fast animal"
→ Search within the animal subspace
→ Filter by high speed
→ Result: "cheetah", "falcon"

2. Automatic clustering

Grouping documents by topic using the structure of the space:

Cluster 1: Documents in the "technology" subspace
Cluster 2: Documents in the "medicine" subspace

3. Semantic analogies

Using directions to find analogies:

"Python" is to "programming" as "MySQL" is to ___?

Vector("MySQL") + (Vector("programming") - Vector("Python"))
≈ Vector("databases")

Limitations

1. The structure isn't perfect

Even though there are clusters and directions, they aren't mathematically perfect. king - man + woman gives a vector that is close to queen, not identical to it.

2. Bias in the space

If the training text has biases (gender, race), the space will reflect them:

Vector("doctor") - Vector("man") + Vector("woman")

Sometimes it gives "nurse" instead of "doctor" (a historical bias in the corpus).

3. Polysemy

Contextual embeddings (BERT, GPT) help, but "bank" (riverbank vs institution) can have vectors that partially overlap.

Summary

Key points:

Clusters: Related concepts group together (animals, vehicles, emotions)
Directions: Relationships captured geometrically (gender, size, tense)
king - man + woman = queen: Navigating semantic directions
Subspaces: Themed regions in high dimensions
Formation: It emerges during training (it isn't programmed)
Applications: Semantic search, clustering, analogies
Limitations: Not perfect, bias, polysemy

Next capsule: 07-capstone-exercise-2.md — Designing a 2D space for concepts, identifying subspaces.