Module 1: What Are Embeddings?

Welcome to Module 1: What Are Embeddings?

Next →

Overview

Welcome to the first module of Embeddings Deep Dive, where you'll learn the fundamental concept that powers 70% of modern AI applications: vector embeddings.

If you already completed the AI Semantics Guide (#5 in the path), you'll remember concepts of vectors, similarity, and vector spaces. In this module we'll dig into how those concepts apply specifically to embeddings for text, and why they are the technical core of RAG (Retrieval-Augmented Generation) and semantic search.

By the end of this module, you'll understand what embeddings are, their vector properties, real-world use cases, and you'll build your first working similarity calculator with OpenAI embeddings.

Why is this module critical?

For modern AI systems:

Embeddings are the underlying technology of:

  • RAG (Retrieval-Augmented Generation): 70% of AI jobs require RAG, and RAG requires embeddings
  • Semantic search: Searching by meaning, not by keywords
  • Recommendation systems: Suggesting similar content
  • Clustering and classification: Grouping documents by topic

Without embeddings, LLMs can only respond with their pre-trained knowledge (limited and outdated). With embeddings, you give them access to external knowledge specific to your domain.

For your career:

Understanding embeddings sets you apart:

  • ✅ 90% of developers just use langchain.embeddings without understanding what it does
  • ✅ 80% don't know how to evaluate whether their embeddings are good
  • ✅ 70% can't explain the difference between cosine similarity and dot product

This module gives you that deep understanding.

For the rest of this guide:

This module lays the foundation for:

  • Module 2: How embeddings work internally (architecture)
  • Module 3: Model comparison (OpenAI vs SBERT vs BGE)
  • Module 4: Implementing semantic search from scratch with numpy
  • Modules 5-8: Production patterns, evaluation, final project

If the fundamentals aren't clear, the advanced modules will be confusing.

Technology versions

This module uses:

  • OpenAI API: text-embedding-3-small (January 2026)
  • Python 3.10+
  • NumPy 1.24+ for vector operations
  • Requests / httpx for HTTP calls

Note: 95% of the content is provider-agnostic. The concepts apply to any embedding model (OpenAI, HuggingFace, Cohere, etc.).

Objectives of this module

By completing this module, you'll be able to:

Conceptual level (Capsules 02-04):

  • ✅ Define what embeddings are and why they exist
  • ✅ Explain the text → numeric vector transformation
  • ✅ Understand vector properties (similarity, directionality, magnitude)
  • ✅ Understand high-dimensional vector spaces

Practical level (Capsules 05-07):

  • ✅ Identify real-world use cases (RAG, search, recommendations)
  • ✅ Differentiate embeddings vs keyword search (BM25 vs semantic)
  • ✅ Understand high-level architecture (Transformers, encoders, pooling)

Implementation level (Capsule 08):

  • ✅ Consume the OpenAI Embeddings API
  • ✅ Calculate cosine similarity with numpy
  • ✅ Build a working similarity calculator

Module roadmap

Capsule 01 (this one): Module Introduction

Context, objectives, and preparation.

Capsule 02: Defining Embeddings

What they are, text → vectors, dimensionality, numeric representation.

Capsule 03: Vector Properties

Semantic similarity, directionality, magnitude vs direction.

Capsule 04: Vector Spaces

High-dimensional spaces, natural clustering, proximity = similarity.

Capsule 05: Real-World Use Cases

Semantic search, RAG, recommendations, classification, clustering.

Capsule 06: Embeddings vs Keywords

BM25 (keyword-based) vs embeddings (semantic), hybrid search.

Capsule 07: Architecture Overview

High-level Transformers, encoder-only models, pooling strategies.

Capsule 08: Mini-Project - Your First Embedding

Setup OpenAI API, embed 10 docs, calculate similarity, CLI tool.

Estimated duration

ActivityTime
Reading capsules 02-0745-60 min
Practice while reading20-30 min
Mini-project (capsule 08)30-40 min
Total~2 hours

Note: This module is 50% conceptual, 50% practical. You'll spend time understanding concepts before implementing.

Verified prerequisites

Before starting, make sure you have:

Prior knowledge:

✅ AI Semantics Guide (#5) completed:

  • Vectors and numeric representations
  • Similarity and distance (cosine, euclidean)
  • Basic vector spaces

If you did NOT complete the AI Semantics Guide:

  • This module has a brief recap (capsules 03-04)
  • But we recommend doing the AI Semantics Guide first for better understanding

✅ Basic Python:

  • Variables, lists, dictionaries
  • Basic functions
  • Imports and modules

✅ Basic AI concepts (recommended):

  • What an LLM (Large Language Model) is
  • What RAG (Retrieval-Augmented Generation) is - conceptually

Technical setup:

✅ Python 3.10+ installed

python --version  # Must be 3.10 or higher

✅ OpenAI API key (for capsule 08)

  • Create an account at https://platform.openai.com
  • Generate an API key under "API Keys"
  • ~$0.50 USD of credit is enough for the entire module

✅ Basic dependencies:

pip install openai numpy python-dotenv requests

Note: The detailed setup is in capsule 08. You don't need to configure anything now.

Module structure

Phase 1: Conceptual Fundamentals (Capsules 02-04)

You'll learn what embeddings are, vector properties, and high-dimensional spaces.

Result: You'll understand the "what" and the "why" of embeddings.

Phase 2: Practical Application (Capsules 05-07)

You'll learn real use cases, comparison with keyword search, and technical architecture.

Result: You'll understand the "when" and the "how" of using embeddings.

Phase 3: Implementation (Capsule 08)

You'll build a working similarity calculator with OpenAI API + numpy.

Result: A functional system that calculates similarity between texts.

Connection with the AI Engineering Learning Path

Where are you in the path?

AI Engineering Learning Path:
├── #1: Git & GitHub Guide
├── #2: TypeScript Essentials Guide
├── #3: Python + REST APIs for AI Guide
├── #4: AI Fundamentals Guide
├── #5: AI Semantics Guide (prerequisite)
├── #6: Embeddings Deep Dive Guide ← You are here (Module 1)
├── #7: Vector Databases Comparison Guide (next)
└── #8: Advanced RAG Patterns Guide

Direct prerequisite:

AI Semantics Guide (#5) gave you:

  • Fundamentals of vectors
  • Similarity and distance
  • Vector spaces

This module (#6) goes deeper:

  • Embeddings specifically for text
  • Technical architecture (Transformers)
  • Practical implementation with OpenAI

Prepares you for:

Vector Databases Comparison Guide (#7):

  • Storing embeddings efficiently
  • Search at scale (millions of vectors)
  • ChromaDB, Pinecone, Weaviate

Advanced RAG Patterns Guide (#8):

  • Using embeddings in production
  • Chunking strategies
  • Hybrid search

Key differences from the AI Semantics Guide

If you completed the AI Semantics Guide, this is new:

AspectAI Semantics Guide (#5)This Module (#6)
FocusGeneral vector conceptsEmbeddings for text specifically
DepthFoundationalTechnical + practical
CodeConceptualReal implementation with OpenAI
ArchitectureNot coveredTransformers, encoders, pooling
Use casesTheoretical examplesReal RAG, semantic search
ProjectNo projectWorking similarity calculator

No redundancy: AI Semantics is a conceptual prerequisite, this module is practical application.

Pedagogical philosophy of this module

50/50 theory/practice:

This is the most conceptual module of the guide (50% theory), because you need to understand what embeddings are before implementing complex systems.

Balance:

Module 1: 50/50 (solid fundamentals)
Module 2-3: 40/60 (more practice)
Module 4-6: 30/70 (heavy implementation)
Module 7-8: 20/80 (projects)

Gradual progression:

Capsule 02: "An embedding is a numeric vector"
         ↓
Capsule 03: "Similar vectors are close in the space"
         ↓
Capsule 04: "Embedding space of 1536 dimensions"
         ↓
Capsule 05: "We use embeddings in RAG, search, recommendations"
         ↓
Capsule 06: "Embeddings > keywords because they capture meaning"
         ↓
Capsule 07: "Transformers generate contextual embeddings"
         ↓
Capsule 08: "Implement your first similarity calculator"

Each concept builds on the previous one. There are no abrupt jumps.

What you will NOT learn in this module

Clear boundaries to avoid confusion:

❌ You will NOT train embedding models

  • You'll use pre-trained models (OpenAI, SBERT)
  • Training custom embeddings is ML Engineering scope (not AI Engineering)

❌ You will NOT dive into Transformer math

  • You'll see high-level architecture (capsule 07)
  • We won't cover attention mechanisms in detail
  • Focus: practical use, not research

❌ You will NOT implement embeddings from scratch

  • You won't code a Transformer from PyTorch
  • You'll use existing APIs and libraries
  • Focus: integration, not model implementation

❌ You will NOT cover fine-tuning of embeddings

  • Fine-tuning is an advanced module (outside this guide)
  • This module uses out-of-the-box models

❌ You will NOT see advanced visualization (PCA, t-SNE)

  • This is not a Data Science course
  • AI Engineering focus: working code, not visualization
  • We'll eliminate matplotlib except where absolutely necessary

This module is about using embeddings in production, not about research or advanced ML.

Professional skills you'll develop

By completing this module, you'll be able to:

  1. Explain embeddings to technical and non-technical stakeholders

    • "Embeddings turn text into numbers that capture meaning"
    • Clear analogies (coordinates on a map)

  2. Identify when to use embeddings vs keyword search

    • Embeddings: Semantic search, synonyms, context
    • Keywords: Exact search, IDs, codes

  3. Consume embedding APIs correctly

    • OpenAI, Cohere, HuggingFace
    • Rate limiting, error handling, batching

  4. Calculate similarity between texts

    • Cosine similarity with numpy
    • Interpret scores (0.0 = unrelated, 1.0 = identical)

  5. Build functional prototypes

    • CLI tools that use embeddings
    • Foundation for more complex systems

These skills are the foundation for Modules 2-8.

Real use cases you'll understand

This module will prepare you to build:

1. Semantic Search Engine

  • User searches "how to restart my laptop"
  • System finds "reset computer", "reboot PC"
  • Embeddings capture that "restart", "reset", "reboot" are similar

2. RAG System (Retrieval-Augmented Generation)

  • User asks about technical documentation
  • System searches relevant docs with embeddings
  • LLM responds using those docs as context

3. Content Recommendation

  • User reads an article about "Python for AI"
  • System recommends "Machine Learning with Python", "FastAPI tutorial"
  • Embeddings identify related topics

4. Duplicate Detection

  • Detect duplicate questions in support
  • "How to reset password" vs "Forgot my password" → Similar
  • Embeddings > keyword matching

5. Document Clustering

  • Group 1000 support tickets by topic
  • Embeddings naturally group similar tickets
  • You don't need to label manually

All these cases use embeddings as their foundation.

The mental model you'll build

By the end of Module 1, your mental model will be:

EMBEDDINGS = 
  Text → Numeric vector of N dimensions
    ↓
  Captures semantic meaning
    ↓
  Similar vectors = texts with similar meaning
    ↓
  Measure similarity with cosine similarity
    ↓
  Applications: search, RAG, recommendations, clustering

Key components:

  • Embedding model: Transforms text → vector (Transformer)
  • Vector space: High-dimensional space (e.g., 1536D)
  • Similarity metric: Measures how close 2 vectors are (cosine)
  • Use case: Specific application (search, RAG, etc.)

This mental model will be refined in Modules 2-8.

Preparation before starting

Before continuing with the following capsules:

1. The right mindset

✅ Yes to:

  • Reading calmly and reflecting
  • Doing the self-assessment questions
  • Experimenting with code in capsule 08
  • Connecting concepts with the AI Semantics Guide

❌ No to:

  • Skipping conceptual capsules (02-04)
  • Memorizing without understanding
  • Getting frustrated by math (it's minimal)
  • Comparing with ML Engineering (different scope)

Solid fundamentals are an investment, not a waste of time.

2. Workspace

We recommend:

  • Notebook or digital notes for key concepts
  • Project folder for code (capsule 08)
  • Clean Python environment (virtualenv recommended)

Example setup:

mkdir embeddings-module-01
cd embeddings-module-01
python -m venv venv
source venv/bin/activate  # macOS/Linux
# venv\Scripts\activate  # Windows
pip install openai numpy python-dotenv requests

3. Optional context resources

If you want additional context (optional, NOT required):

You don't need to read these resources now. The capsules are self-contained.

Module success metrics

By the end of Module 1, you should be able to:

  • ✅ Explain what embeddings are in 2 minutes to someone technical
  • ✅ Identify 3 use cases where embeddings beat keywords
  • ✅ Describe the full flow: Text → Embedding → Similarity → Top-K results
  • ✅ Consume the OpenAI Embeddings API with Python
  • ✅ Calculate cosine similarity with numpy
  • ✅ Build a working similarity calculator (mini-project)

Evidence: You'll be able to complete the mini-project in capsule 08 without assistance.

Philosophy: Understand before implementing

Why we dedicate 6 capsules to concepts before code:

Common problem in embedding courses:

# Capsule 1, line 1:
from langchain.embeddings import OpenAIEmbeddings
embeddings = OpenAIEmbeddings()
result = embeddings.embed_query("Hello")
# "It works, but... what's happening?"

Problem: Students use embeddings without understanding what they are.

Our approach:

Capsule 02: You understand what an embedding is (numeric vector)
Capsule 03: You understand properties (similarity, directionality)
Capsule 04: You understand vector spaces (natural clustering)
Capsule 05: You understand use cases (when to use embeddings)
Capsule 06: You understand advantages vs keywords
Capsule 07: You understand architecture (how they're generated)
         ↓
Capsule 08: NOW you implement with the OpenAI API

Result: When you implement in capsule 08, you'll understand why that code works.

Avoiding common misconceptions

This module will also debunk myths:

❌ Myth 1: "Embeddings are only for searching similar words" ✅ Reality: They capture full semantic meaning, not just synonyms

❌ Myth 2: "I need a PhD in math to understand embeddings" ✅ Reality: Basic vector concepts + practical code is enough

❌ Myth 3: "All embeddings are the same" ✅ Reality: Different models for different use cases (Module 3)

❌ Myth 4: "Embeddings are always better than keyword search" ✅ Reality: Hybrid is better for many cases (capsule 06)

❌ Myth 5: "OpenAI embeddings are the only option" ✅ Reality: Open-source (SBERT, BGE) are competitive (Module 3)

How to get the most out of this module

During the capsules:

  1. Read without rushing (each capsule takes 10-15 min)
  2. Take notes on concepts that resonate with you
  3. Do the reflection questions when they appear
  4. Connect with the AI Semantics Guide (#5)

At the end of each capsule:

  1. Summarize what you learned in 2-3 points
  2. Identify what became clear and what needs review

At the end of the module:

  1. Complete the mini-project (capsule 08)
  2. Review your notes and consolidate the mental model
  3. Get ready for Module 2 (technical architecture)

Structure of this module

8 capsules, each building on the previous one:

01. Module Introduction ← You are here
    ↓
02. Defining Embeddings (What they are)
    ↓
03. Vector Properties (How they work)
    ↓
04. Vector Spaces (Where they live)
    ↓
05. Real-World Use Cases (What they're for)
    ↓
06. Embeddings vs Keywords (When to use them)
    ↓
07. Architecture Overview (How they're generated)
    ↓
08. Mini-Project: Your First Embedding (Implementation)

Each capsule is self-contained but references the previous ones.

Summary

Module 1 is your solid introduction to embeddings.

What you'll learn:

  • What embeddings are (text → vector)
  • Vector properties (similarity, spaces)
  • Real use cases (RAG, search, recommendations)
  • Basic implementation (OpenAI API + numpy)

What you will NOT do:

  • Train models (you'll use pre-trained ones)
  • Advanced math (the minimum necessary)
  • Implement Transformers (you'll use APIs)

What you'll build:

  • A clear mental model of embeddings
  • A working similarity calculator
  • A foundation for Modules 2-8

Signs of success:

  • You'll be able to explain embeddings clearly
  • You'll be able to identify when to use embeddings
  • You'll be able to implement a similarity calculator

Ready to begin?

In the next capsule, you'll answer the fundamental question:

"What exactly are embeddings?"

You'll see:

  • The text → numeric vector transformation
  • Why 1536 dimensions (OpenAI text-embedding-3-small)
  • How they capture semantic meaning
  • Concrete examples with 2D visualization (projection)

You'll also see why embeddings are different from one-hot encoding, TF-IDF, and other vector representations you may know.

This is the beginning of your mastery of embeddings for AI Engineering. 🚀

Preparation resources (optional)

If you want additional context before starting:

You don't need to read these resources. The capsules are complete and self-contained.

Next capsule: 02-defining-embeddings.md

Next →Defining Embeddings