2. Image Embeddings

Description

An embedding is a numerical vector that captures the semantic meaning of a piece of data. For text, you already know this: "dog" and "hound" have close embeddings. But how do you do the same with an image? How do you turn a photo of a dog into a vector you can compare with the text "dog"? That's the central question of this capsule.

Image embeddings are the fundamental piece of multimodal RAG. Without them, you can't index images or search them. There are two main strategies: generate a textual description of the image with a vision model and then get the embedding of the text, or use a multimodal model like CLIP that generates embeddings directly in a shared text-image space.

Why it matters: Without image embeddings, your RAG is text only. With them, you can search "architecture diagram" and retrieve both paragraphs mentioning architecture and visual diagrams that illustrate it. This capsule gives you the two tools to achieve that.

Connection with the module: The embeddings you generate here are stored in the index (capsule 03), searched with hybrid retrieval (capsule 04), and form the core of the project (capsule 08).

Text Embeddings: The Foundation You Already Know

Before jumping to images, let's review how text embeddings work, because the same logic extends to images.

from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()
client = OpenAI()


def text_embedding(text: str) -> list[float]:
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return response.data[0].embedding


emb_dog = text_embedding("dog")
emb_hound = text_embedding("hound")
emb_car = text_embedding("car")

print(f"Embedding dimension: {len(emb_dog)}")
print(f"First 5 values: {emb_dog[:5]}")

Each text is converted into a 1536-dimension vector (for text-embedding-3-small). Texts with similar meaning produce close vectors.

Cosine similarity

To measure how close two embeddings are, we use cosine similarity: a value between -1 and 1 where 1 = identical, 0 = unrelated, -1 = opposite.

import numpy as np


def cosine_similarity(a: list[float], b: list[float]) -> float:
    vec_a = np.array(a)
    vec_b = np.array(b)
    dot_product = np.dot(vec_a, vec_b)
    norm_product = np.linalg.norm(vec_a) * np.linalg.norm(vec_b)
    if norm_product == 0:
        return 0.0
    return float(dot_product / norm_product)


sim_dog_hound = cosine_similarity(emb_dog, emb_hound)
sim_dog_car = cosine_similarity(emb_dog, emb_car)

print(f"dog ↔ hound: {sim_dog_hound:.4f}")
print(f"dog ↔ car: {sim_dog_car:.4f}")

Expected result: dog ↔ hound will have a high score (~0.85+), while dog ↔ car will be lower (~0.50-0.65).

Strategy 1: Image → Description → Embedding

How it works

This is the simplest strategy and the one that works best with the current OpenAI APIs:

Image
  ↓ (Vision API: gpt-4o-mini)
Textual description: "Architecture diagram showing three microservices..."
  ↓ (Embeddings API: text-embedding-3-small)
Vector [0.02, -0.15, 0.33, ...] (1536 dimensions)

You convert the image into text using a vision-capable model, and then convert that text into an embedding. The resulting embedding is comparable with any other text embedding.

Advantages and limitations

Complete implementation

from openai import OpenAI
import base64
from pathlib import Path
from dotenv import load_dotenv

load_dotenv()
client = OpenAI()

DESCRIPTION_PROMPT = (
    "Describe this image concisely and precisely for indexing "
    "in a search system. Include: what it shows, main elements, "
    "type of content (diagram, photo, chart, table, screenshot). "
    "Maximum 2-3 sentences."
)


def encode_image(image_path: str) -> str:
    path = Path(image_path)
    if not path.exists():
        raise FileNotFoundError(f"Image not found: {image_path}")

    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")


def get_mime_type(image_path: str) -> str:
    ext = Path(image_path).suffix.lower()
    mime_map = {
        ".jpg": "image/jpeg",
        ".jpeg": "image/jpeg",
        ".png": "image/png",
        ".gif": "image/gif",
        ".webp": "image/webp",
    }
    mime = mime_map.get(ext)
    if not mime:
        raise ValueError(f"Unsupported format: {ext}")
    return mime


def describe_image(image_path: str, model: str = "gpt-4o-mini") -> str:
    b64 = encode_image(image_path)
    mime = get_mime_type(image_path)

    response = client.chat.completions.create(
        model=model,
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": DESCRIPTION_PROMPT},
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:{mime};base64,{b64}"}
                }
            ]
        }],
        max_tokens=150
    )
    return response.choices[0].message.content


def image_to_embedding(image_path: str) -> dict:
    description = describe_image(image_path)

    embedding = client.embeddings.create(
        model="text-embedding-3-small",
        input=description
    ).data[0].embedding

    return {
        "embedding": embedding,
        "description": description,
        "source": image_path
    }


result = image_to_embedding("test_docs/diagram.png")
print(f"Description: {result['description']}")
print(f"Embedding dim: {len(result['embedding'])}")

The description prompt

The prompt you use to describe the image directly impacts the quality of the embedding. A generic prompt like "Describe this image" produces vague descriptions. A prompt specific to your use case produces descriptions that index better.

PROMPTS_BY_DOMAIN = {
    "technical": (
        "Describe this technical image: what type of diagram or figure it is, "
        "what components or elements it shows, what relationships exist between them. "
        "Use precise technical terminology."
    ),
    "product": (
        "Describe this product: type, color, shape, apparent material, "
        "usage context. Include visual details relevant for search."
    ),
    "document": (
        "Describe the visual content of this page: tables, charts, "
        "figures, diagrams. What information they convey."
    ),
    "general": DESCRIPTION_PROMPT,
}


def describe_image_for_domain(
    image_path: str,
    domain: str = "general"
) -> str:
    prompt = PROMPTS_BY_DOMAIN.get(domain, PROMPTS_BY_DOMAIN["general"])
    b64 = encode_image(image_path)
    mime = get_mime_type(image_path)

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": prompt},
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:{mime};base64,{b64}"}
                }
            ]
        }],
        max_tokens=200
    )
    return response.choices[0].message.content

Strategy 2: Multimodal Embeddings with CLIP

How CLIP works

CLIP (Contrastive Language-Image Pre-training) is a model from OpenAI (open source via Hugging Face) that was trained on 400 million image-text pairs. It learned to place images and texts in the same vector space: a photo of a cat and the text "cat" produce close vectors.

Text: "orange cat"    → CLIP → vector [0.12, -0.08, ...]  (512 dim)
Image: cat_photo.jpg  → CLIP → vector [0.11, -0.09, ...]  (512 dim)
                                          ↑ close ↑

Unlike strategy 1, here there's no intermediate description step. The image is converted directly into a vector comparable with text.

CLIP setup

pip install transformers torch pillow

Implementation

from transformers import CLIPProcessor, CLIPModel
from PIL import Image
import torch
import numpy as np

CLIP_MODEL_NAME = "openai/clip-vit-base-patch32"

clip_model = CLIPModel.from_pretrained(CLIP_MODEL_NAME)
clip_processor = CLIPProcessor.from_pretrained(CLIP_MODEL_NAME)
clip_model.eval()


def clip_image_embedding(image_path: str) -> list[float]:
    image = Image.open(image_path).convert("RGB")
    inputs = clip_processor(images=image, return_tensors="pt")

    with torch.no_grad():
        features = clip_model.get_image_features(**inputs)

    normalized = features / features.norm(dim=-1, keepdim=True)
    return normalized[0].numpy().tolist()


def clip_text_embedding(text: str) -> list[float]:
    inputs = clip_processor(text=[text], return_tensors="pt", padding=True)

    with torch.no_grad():
        features = clip_model.get_text_features(**inputs)

    normalized = features / features.norm(dim=-1, keepdim=True)
    return normalized[0].numpy().tolist()


img_emb = clip_image_embedding("test_docs/diagram.png")
txt_emb = clip_text_embedding("architecture diagram")

sim = cosine_similarity(img_emb, txt_emb)
print(f"CLIP embedding dim: {len(img_emb)}")
print(f"Similarity image ↔ text: {sim:.4f}")

CLIP comparison against different texts

def compare_image_with_texts(
    image_path: str,
    texts: list[str]
) -> list[tuple[str, float]]:
    img_emb = clip_image_embedding(image_path)
    results = []

    for text in texts:
        txt_emb = clip_text_embedding(text)
        sim = cosine_similarity(img_emb, txt_emb)
        results.append((text, sim))

    results.sort(key=lambda x: x[1], reverse=True)
    return results


texts = [
    "software architecture diagram",
    "photo of a cat",
    "monthly sales chart",
    "Python code in a terminal",
    "mountainous landscape",
]

rankings = compare_image_with_texts("test_docs/diagram.png", texts)
for text, score in rankings:
    print(f"  {score:.4f} — {text}")

Comparison of strategies

When to use each one?

Use Strategy 1 (Vision → Embedding) when:
  - Your corpus is technical documentation with diagrams
  - The important information is CONCEPTUAL (what the diagram represents)
  - You already use OpenAI embeddings for text
  - You don't want to install local models

Use Strategy 2 (CLIP) when:
  - Your corpus is visual (catalogs, photos, art)
  - The important information is VISUAL (colors, shapes, composition)
  - You need low cost and low latency
  - You want image → image search

Batch Embeddings

When you have many images (or many texts), processing one by one is inefficient. Here are utilities for batch processing.

Batch text embeddings (OpenAI)

def batch_text_embeddings(
    texts: list[str],
    batch_size: int = 100
) -> list[list[float]]:
    all_embeddings = []

    for i in range(0, len(texts), batch_size):
        batch = texts[i:i + batch_size]
        response = client.embeddings.create(
            model="text-embedding-3-small",
            input=batch
        )
        batch_embs = [item.embedding for item in response.data]
        all_embeddings.extend(batch_embs)

    return all_embeddings


descriptions = ["Image 1 description", "Image 2 description", "..."]
embeddings = batch_text_embeddings(descriptions)
print(f"Generated {len(embeddings)} embeddings")

Batch CLIP embeddings

def batch_clip_image_embeddings(
    image_paths: list[str]
) -> list[list[float]]:
    images = [Image.open(p).convert("RGB") for p in image_paths]
    inputs = clip_processor(images=images, return_tensors="pt", padding=True)

    with torch.no_grad():
        features = clip_model.get_image_features(**inputs)

    normalized = features / features.norm(dim=-1, keepdim=True)
    return normalized.numpy().tolist()


paths = ["img1.png", "img2.png", "img3.png"]
clip_embs = batch_clip_image_embeddings(paths)
print(f"Generated {len(clip_embs)} CLIP embeddings")

Batch descriptions with Vision (async)

import asyncio
from openai import AsyncOpenAI


async def describe_image_async(
    async_client: AsyncOpenAI,
    image_path: str
) -> dict:
    b64 = encode_image(image_path)
    mime = get_mime_type(image_path)

    response = await async_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": DESCRIPTION_PROMPT},
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:{mime};base64,{b64}"}
                }
            ]
        }],
        max_tokens=150
    )
    return {
        "path": image_path,
        "description": response.choices[0].message.content
    }


async def batch_describe_images(
    image_paths: list[str],
    max_concurrent: int = 5
) -> list[dict]:
    async_client = AsyncOpenAI()
    semaphore = asyncio.Semaphore(max_concurrent)

    async def limited_describe(path: str) -> dict:
        async with semaphore:
            return await describe_image_async(async_client, path)

    tasks = [limited_describe(p) for p in image_paths]
    return await asyncio.gather(*tasks)


# results = asyncio.run(batch_describe_images(["img1.png", "img2.png"]))

Vector Databases: Intro to ChromaDB

Embeddings on their own are just vectors in memory. To search efficiently among thousands or millions of embeddings, you need a vector store: a database optimized for vector similarity search.

ChromaDB: why we use it

ChromaDB is an open source, lightweight vector store that runs locally with no external server. It's the ideal choice for development and prototyping.

First steps with ChromaDB

import chromadb
from chromadb.utils import embedding_functions

openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    model_name="text-embedding-3-small"
)

chroma_client = chromadb.Client()

collection = chroma_client.create_collection(
    name="test_embeddings",
    embedding_function=openai_ef,
    metadata={"hnsw:space": "cosine"}
)

collection.add(
    documents=[
        "Python is a versatile programming language",
        "JavaScript is mainly used for web development",
        "Architecture diagram with three microservices",
    ],
    ids=["doc_1", "doc_2", "doc_3"],
    metadatas=[
        {"type": "text", "topic": "python"},
        {"type": "text", "topic": "javascript"},
        {"type": "image_desc", "topic": "architecture"},
    ]
)

results = collection.query(
    query_texts=["microservices"],
    n_results=2
)

for doc, meta, dist in zip(
    results["documents"][0],
    results["metadatas"][0],
    results["distances"][0]
):
    print(f"  [{dist:.4f}] ({meta['type']}) {doc[:60]}...")

chroma_client.delete_collection("test_embeddings")

Store image embeddings in ChromaDB

def store_image_embeddings(
    collection,
    image_paths: list[str]
) -> None:
    for i, path in enumerate(image_paths):
        result = image_to_embedding(path)
        collection.add(
            documents=[result["description"]],
            embeddings=[result["embedding"]],
            ids=[f"img_{i}"],
            metadatas=[{
                "type": "image",
                "source": path,
                "description": result["description"]
            }]
        )
        print(f"Indexed: {path} → {result['description'][:50]}...")

Similarity Search: Text ↔ Image

With text and image embeddings in the same index, you can search in both directions.

Search images with text

def search_images_by_text(
    collection,
    query: str,
    n: int = 5
) -> list[dict]:
    results = collection.query(
        query_texts=[query],
        n_results=n,
        where={"type": "image"}
    )

    matches = []
    for doc, meta, dist in zip(
        results["documents"][0],
        results["metadatas"][0],
        results["distances"][0]
    ):
        matches.append({
            "description": doc,
            "source": meta.get("source", "unknown"),
            "distance": dist,
            "similarity": 1 - dist / 2,
        })
    return matches

Search text with an image

def search_text_by_image(
    collection,
    image_path: str,
    n: int = 5
) -> list[dict]:
    desc = describe_image(image_path)

    results = collection.query(
        query_texts=[desc],
        n_results=n,
        where={"type": "text"}
    )

    matches = []
    for doc, meta, dist in zip(
        results["documents"][0],
        results["metadatas"][0],
        results["distances"][0]
    ):
        matches.append({
            "text": doc,
            "metadata": meta,
            "distance": dist,
        })
    return matches

Combined search (no type filter)

def search_all(
    collection,
    query: str,
    n: int = 10
) -> list[dict]:
    results = collection.query(
        query_texts=[query],
        n_results=n
    )

    matches = []
    for doc, meta, dist in zip(
        results["documents"][0],
        results["metadatas"][0],
        results["distances"][0]
    ):
        matches.append({
            "content": doc,
            "type": meta.get("type", "unknown"),
            "distance": dist,
        })
    return matches


all_results = search_all(collection, "microservices architecture")
for r in all_results:
    print(f"  [{r['type']}] {r['distance']:.4f} — {r['content'][:60]}...")

Troubleshooting

Error: "Invalid image format"

SUPPORTED_FORMATS = {".jpg", ".jpeg", ".png", ".gif", ".webp"}

def validate_image(image_path: str) -> bool:
    path = Path(image_path)
    if not path.exists():
        print(f"ERROR: File does not exist: {image_path}")
        return False
    if path.suffix.lower() not in SUPPORTED_FORMATS:
        print(f"ERROR: Unsupported format: {path.suffix}")
        return False
    size_mb = path.stat().st_size / (1024 * 1024)
    if size_mb > 20:
        print(f"ERROR: File too large: {size_mb:.1f} MB (max 20 MB)")
        return False
    return True

Error: "Embedding dimension mismatch"

If you mix OpenAI embeddings (1536d) with CLIP (512d), you can't compare them or store them in the same collection.

Solution: Use a single strategy per collection.
  - Collection "openai_index": only text-embedding-3-small embeddings
  - Collection "clip_index": only CLIP embeddings

Vague descriptions from the vision model

If the model describes an image as "An image with text and shapes", the resulting embedding will be of little use.

Solution: Use a prompt specific to your domain (see the prompts section).
  - For diagrams: ask for components and relationships
  - For products: ask for type, color, material
  - For charts: ask for chart type, variables, trend

ChromaDB: "Collection already exists"

collection = chroma_client.get_or_create_collection(
    name="multimodal",
    embedding_function=openai_ef
)

CLIP: slowness on CPU

CLIP on CPU can take ~1-2 seconds per image. With GPU, it drops to ~0.05s.

device = "cuda" if torch.cuda.is_available() else "cpu"
clip_model = CLIPModel.from_pretrained(CLIP_MODEL_NAME).to(device)

def clip_image_embedding_gpu(image_path: str) -> list[float]:
    image = Image.open(image_path).convert("RGB")
    inputs = clip_processor(images=image, return_tensors="pt")
    inputs = {k: v.to(device) for k, v in inputs.items()}

    with torch.no_grad():
        features = clip_model.get_image_features(**inputs)

    normalized = features / features.norm(dim=-1, keepdim=True)
    return normalized[0].cpu().numpy().tolist()

Exercises

Exercise 1: Text embedding for semantic search

Generate embeddings for a list of queries and a list of image descriptions. Find the most similar query-description pair.

def find_best_match(
    queries: list[str],
    descriptions: list[str]
) -> list[tuple[str, str, float]]:
    query_embs = batch_text_embeddings(queries)
    desc_embs = batch_text_embeddings(descriptions)

    matches = []
    for i, q_emb in enumerate(query_embs):
        best_score = -1
        best_desc = ""
        for j, d_emb in enumerate(desc_embs):
            score = cosine_similarity(q_emb, d_emb)
            if score > best_score:
                best_score = score
                best_desc = descriptions[j]
        matches.append((queries[i], best_desc, best_score))

    return matches


queries = [
    "database diagram",
    "photo of a development team",
    "server performance chart",
]
descriptions = [
    "Photograph of a group of people in a modern office",
    "Entity-relationship diagram with users, orders and products tables",
    "Line chart showing server latency over the last 30 days",
]

for query, desc, score in find_best_match(queries, descriptions):
    print(f"  {query} → {desc[:50]}... ({score:.4f})")

Exercise 2: Compare embedding strategies

Given an image of a technical diagram, generate embeddings with both strategies (Vision+Embedding and CLIP). Search against a set of texts and compare which strategy produces better results.

def compare_strategies(
    image_path: str,
    search_texts: list[str]
) -> dict:
    vision_result = image_to_embedding(image_path)
    vision_emb = vision_result["embedding"]

    clip_img_emb = clip_image_embedding(image_path)

    vision_scores = []
    clip_scores = []

    for text in search_texts:
        text_emb_openai = text_embedding(text)
        vision_scores.append({
            "text": text,
            "score": cosine_similarity(vision_emb, text_emb_openai)
        })

        text_emb_clip = clip_text_embedding(text)
        clip_scores.append({
            "text": text,
            "score": cosine_similarity(clip_img_emb, text_emb_clip)
        })

    vision_scores.sort(key=lambda x: x["score"], reverse=True)
    clip_scores.sort(key=lambda x: x["score"], reverse=True)

    return {
        "vision_description": vision_result["description"],
        "vision_ranking": vision_scores,
        "clip_ranking": clip_scores,
    }


texts = [
    "microservices architecture",
    "cooking recipe",
    "data flow diagram",
    "natural landscape",
    "Python source code",
]

comparison = compare_strategies("test_docs/diagram.png", texts)
print(f"Vision description: {comparison['vision_description']}")
print("\nVision+Embedding ranking:")
for r in comparison["vision_ranking"]:
    print(f"  {r['score']:.4f} — {r['text']}")
print("\nCLIP ranking:")
for r in comparison["clip_ranking"]:
    print(f"  {r['score']:.4f} — {r['text']}")

Exercise 3: Search for the most similar image in ChromaDB

Create a collection in ChromaDB with 5+ documents (a mix of text and image descriptions). Implement a function that takes a text query and returns the most similar result, indicating whether it's text or image.

def build_and_search_index(
    text_chunks: list[str],
    image_descriptions: list[dict],
    query: str
) -> dict:
    openai_ef = embedding_functions.OpenAIEmbeddingFunction(
        model_name="text-embedding-3-small"
    )

    chroma = chromadb.Client()
    coll = chroma.get_or_create_collection(
        "search_test", embedding_function=openai_ef
    )

    for i, chunk in enumerate(text_chunks):
        coll.add(
            documents=[chunk],
            ids=[f"text_{i}"],
            metadatas=[{"type": "text"}]
        )

    for i, img in enumerate(image_descriptions):
        coll.add(
            documents=[img["description"]],
            ids=[f"img_{i}"],
            metadatas=[{
                "type": "image",
                "source": img["source"]
            }]
        )

    results = coll.query(query_texts=[query], n_results=1)

    best = {
        "content": results["documents"][0][0],
        "type": results["metadatas"][0][0]["type"],
        "distance": results["distances"][0][0],
    }

    chroma.delete_collection("search_test")
    return best


result = build_and_search_index(
    text_chunks=[
        "FastAPI is a modern web framework for Python",
        "Docker lets you containerize applications",
        "PostgreSQL is a relational database",
    ],
    image_descriptions=[
        {"description": "Architecture diagram with API Gateway and microservices", "source": "arch.png"},
        {"description": "Terminal screenshot showing Docker Compose logs", "source": "docker.png"},
    ],
    query="how do microservices communicate"
)
print(f"Result: [{result['type']}] {result['content']}")

Exercise 4: Top-K with normalized scores

Implement a function that searches the K most relevant results and returns normalized scores between 0 and 1 (where 1 = perfect match).

def search_top_k_normalized(
    collection,
    query: str,
    k: int = 5
) -> list[dict]:
    results = collection.query(
        query_texts=[query],
        n_results=k
    )

    if not results["documents"][0]:
        return []

    distances = results["distances"][0]
    max_dist = max(distances) if distances else 1.0

    matches = []
    for doc, meta, dist in zip(
        results["documents"][0],
        results["metadatas"][0],
        distances
    ):
        normalized_score = 1.0 - (dist / max_dist) if max_dist > 0 else 1.0
        matches.append({
            "content": doc,
            "type": meta.get("type", "unknown"),
            "raw_distance": dist,
            "normalized_score": round(normalized_score, 4),
        })

    return matches

Summary

• Image embeddings convert visual content into vectors comparable with text.
• Strategy 1 (Vision → Embedding): describe the image with an LLM and generate an embedding of the text. Simple, compatible, but loses visual information.
• Strategy 2 (CLIP): generate embeddings directly in a shared text-image space. More accurate for visual search, free and local.
• Cosine similarity measures how close two vectors are (0 to 1 for normalized vectors).
• ChromaDB is a local vector store ideal for development: it stores embeddings with metadata and allows similarity search.
• For large volumes, use batch processing and async to describe images in parallel.
• The description prompt directly impacts the quality of the embedding in Strategy 1.

Additional Resources

1. CLIP Paper (Radford et al. 2021) — The original CLIP paper
2. Hugging Face CLIP — Available CLIP model
3. OpenAI Embeddings — Official embeddings guide
4. ChromaDB Docs — Complete ChromaDB documentation
5. Cosine Similarity Explained — Visual guide to cosine similarity