Capsule 03: Designing the metadata schema — the decision your future self will thank you for (or curse you for)

Capsule description

Pre-filtering only works if your metadata is well designed. And "well designed" means thought through before the first bulk ingest, not after you discover you need a new field when you already have 10M documents indexed. This capsule gives you the playbook for designing the right schema from day 1, avoiding the two opposite traps: a schema that's too minimalist (which forces you to re-index at the first change) and a bloated one (where 80% of the fields are never used but slow everything down).

The secret is the principle "design for the filters you'll need in the next 12 months, not for what you have today". That projection requires a workshop with stakeholders; it isn't the engineer's decision alone. This capsule teaches you how to run that workshop, which fields are universal (they always go in), and how to validate metadata before indexing so the technical debt doesn't pile up.

By the end of this capsule you'll be able to:

• ✅ Identify the 5 universal fields almost every production RAG needs
• ✅ Run a schema design workshop with stakeholders in under 1 hour
• ✅ Apply the four modeling conventions (snake_case, controlled enums, correct types, consistent naming)
• ✅ Implement pre-indexing metadata validation with Pydantic
• ✅ Design a legacy metadata migration with no downtime
• ✅ Anticipate the most expensive mistake: adding a filter field 6 months later and having to re-embed 10M docs

Estimated time: 30-35 minutes

The 5 universal fields

Almost every production-ready RAG system needs these five fields. Start with them, then add domain-specific ones:

Field 1: tenant_id (or workspace_id)

Type: string Mandatory: yes, in multi-tenant systems Reason: security isolation. Covered in capsules 02 and 05. Example: "tenant_id": "acme_corp"

Field 2: doc_id and chunk_index

Type: string + int Mandatory: yes Reason: traceability. When a retrieval returns a chunk, you want to know which original document it came from and what position it occupies. Example: "doc_id": "manual_v3_chapter_4", "chunk_index": 12

Field 3: source

Type: string Mandatory: yes Reason: debugging. When something is wrong, you want to know which original document caused it. Example: "source": "/docs/legal/contracts/v3/section_4.md"

Field 4: created_at (timestamp)

Type: int (Unix epoch seconds) Mandatory: yes Reason: time-based filters. "only docs from the last year", "ignore docs older than version 3.0", retention policies, freshness in ranking. Example: "created_at": 1714521600

Important: Unix epoch as an int, not an ISO string. It enables range filters ($gte, $lt) that strings don't allow.

Field 5: type or category

Type: string (a closed enum) Mandatory: recommended Reason: segmentation by intent. "search only the FAQ", "tutorials only", "ignore marketing". It improves precision dramatically. Example: "type": "tutorial" (from an enum {tutorial, reference, faq, changelog, marketing})

Domain-specific fields

You add these based on your case. Some common ones:

The rule: add a field if you're going to filter on it. Don't add it "just in case" — a field that's never used is permanent overhead.

The schema design workshop

Before indexing 1M documents, run a 60-90 minute workshop with the product's stakeholders:

The questions to ask

1. Do we have multiple customers who must NOT see each other's data? (If yes: tenant_id is mandatory)
2. Are we going to have documents of different types? (If yes: a type enum)
3. Does the document's date matter for some queries? (If yes: created_at)
4. What filters are we going to be asked for in the next 6-12 months? (This is where product thinks)
5. Are there categories/segments of the corpus that get handled differently? (If yes: add a field)
6. Are there regulations that require specific fields? (Compliance: GDPR, HIPAA, SOX)

The workshop's output: a schema contract

# data_contract.py
from pydantic import BaseModel, Field
from typing import Literal
from datetime import datetime


class DocumentMetadata(BaseModel):
    """The metadata contract for every document in the system."""

    # Universals
    tenant_id: str = Field(..., description="The tenant's ID. Multi-tenant isolation.")
    doc_id: str = Field(..., description="A stable ID for the parent document.")
    chunk_index: int = Field(..., ge=0, description="The chunk's position within the doc.")
    source: str = Field(..., description="The original document's path or URI.")
    created_at: int = Field(..., description="Unix timestamp in seconds.")

    # Type (a closed enum)
    type: Literal["tutorial", "reference", "faq", "changelog", "marketing"]

    # Domain-specific
    language: Literal["en", "es", "pt", "fr"]
    product_line: Literal["api", "dashboard", "mobile", "general"]
    version: str = Field(..., regex=r"^\d+\.\d+(\.\d+)?$")  # e.g. "3.1.0"

    # Open tags (a list of normalized strings)
    tags: list[str] = Field(default_factory=list)

    class Config:
        # Forbid extra fields to avoid accidentally bloating the schema
        extra = "forbid"

The benefits of having this contract:

• Every document that gets indexed goes through validation.
• If the contract changes, there's a single place to update it.
• New devs know exactly what metadata is expected.
• Pydantic generates an automatic JSON schema for the documentation.

The four modeling conventions

Convention 1: consistent snake_case

# ✅ Good
{"tenant_id": "acme", "created_at": 1714521600, "product_line": "api"}

# ❌ Bad: mixed casing
{"tenantId": "acme", "createdAt": 1714521600, "ProductLine": "api"}

Why: consistency makes the filters readable and prevents typos. Some vector DBs (ChromaDB included) are case-sensitive on metadata keys.

Convention 2: closed enums for categorical fields

# ✅ Good
ALLOWED_TYPES = {"tutorial", "reference", "faq"}
ALLOWED_LANGUAGES = {"en", "es", "pt"}

# ❌ Bad: an open field
{"type": "Tutorial"}     # same concept, but "Tutorial" vs "tutorial"
{"type": "TutoriaL"}     # a typo
{"type": "tutorial doc"} # a semantic variant

Why: without an enum, you end up with 47 variants of the same concept and the filters lose coverage. Pre-ingest validation forces consistency.

Convention 3: correct types for range filters

# ✅ Good
{"created_at": 1714521600}  # int, allows $gte, $lt
{"price": 99.99}             # float
{"is_published": True}       # bool

# ❌ Bad
{"created_at": "2024-04-30"}  # a string, doesn't support range filters properly
{"price": "99.99"}            # string parsing on every filter

Why: filters like where={"created_at": {"$gte": cutoff}} require comparable numeric types. Strings don't work correctly.

Convention 4: descriptive, prefixed naming

# ✅ Good (descriptive, unambiguous)
{"tenant_id": "X", "user_role": "admin", "doc_visibility": "private"}

# ❌ Bad (ambiguous)
{"id": "X", "role": "admin", "vis": "private"}  # which id? a role of what?

Why: names stick around for years. A couple of extra characters of clarity is worth it.

Pre-indexing validation

Implement validation that fails loudly if the metadata doesn't meet the contract:

# validation.py
from pydantic import ValidationError


def validate_and_normalize(raw_metadata: dict) -> DocumentMetadata:
    """Validates and normalizes metadata. Fails if it doesn't meet the contract."""
    try:
        # Pydantic validates types, required fields, enums, regex
        validated = DocumentMetadata(**raw_metadata)
    except ValidationError as e:
        # Detailed logging for debugging
        print(f"Metadata validation failed: {e.errors()}")
        raise

    # Additional normalization
    metadata_dict = validated.model_dump()
    metadata_dict["tags"] = [t.lower().strip() for t in metadata_dict.get("tags", [])]

    return metadata_dict


# Usage in the ingest pipeline
def ingest_document(content: str, raw_metadata: dict):
    try:
        validated_metadata = validate_and_normalize(raw_metadata)
    except ValidationError:
        # The decision: skip or fail loud
        print(f"Skipping doc due to invalid metadata")
        return None

    collection.add(
        documents=[content],
        metadatas=[validated_metadata],
        ids=[f"{validated_metadata['doc_id']}_chunk_{validated_metadata['chunk_index']}"],
    )

The key decision: what do you do if validation fails?

• Fail loud (raise): better in development. It forces you to fix the problem.
• Skip + log: better in production. Don't block the pipeline over one malformed doc, but log it for investigation.
• Quarantine: move the invalid docs to a separate collection for manual review.

Migrating legacy metadata

If you arrive late and already have 1M docs with no structured metadata, there are options:

Option A: re-index with complete metadata (the right way, but expensive)

def reindex_with_proper_metadata(old_collection, new_collection):
    """Re-indexes the whole corpus with valid metadata."""
    docs = old_collection.get(include=["documents", "metadatas", "embeddings"])

    for doc, raw_meta, emb in zip(docs["documents"], docs["metadatas"], docs["embeddings"]):
        # Infer or add the missing fields
        new_meta = backfill_metadata(raw_meta, doc)
        validated = validate_and_normalize(new_meta)

        new_collection.add(
            documents=[doc],
            embeddings=[emb],  # reuse the existing embeddings (don't re-embed)
            metadatas=[validated],
            ids=[validated["doc_id"]],
        )


def backfill_metadata(raw: dict, content: str) -> dict:
    """Infers the missing fields from the content or from defaults."""
    return {
        "tenant_id": raw.get("tenant_id", "unknown"),  # a default if it doesn't exist
        "doc_id": raw.get("doc_id") or hash(content),   # generate it if it's missing
        "chunk_index": raw.get("chunk_index", 0),
        "source": raw.get("source", raw.get("filename", "legacy")),
        "created_at": raw.get("created_at") or int(datetime.now().timestamp()),
        "type": raw.get("type", "reference"),
        "language": detect_language(content),  # ML detection
        "product_line": raw.get("product_line", "general"),
        "version": raw.get("version", "1.0"),
        "tags": raw.get("tags", []),
    }

The cost: the time to re-index, but not to re-embed (you reuse the embeddings).

Option B: progressive migration (breaks nothing)

def progressive_migration():
    """Marks docs with a schema_version. Processes them in batches."""
    # Process 10K docs per day
    batch = old_collection.get(
        where={"schema_version": None},  # the un-migrated ones only
        limit=10000,
    )

    for item in batch:
        # ... the same backfill ...
        validated["schema_version"] = "v2"
        update_in_place(item, validated)

The benefit: no downtime. The system mixes migrated and un-migrated docs during the transition.

The cost: the filters have to tolerate both schemas for weeks.

Traps and common mistakes

Trap 1: a schema that's too minimalist at the start

The mistake: "let's just add tenant_id for now, we'll add more later if we need to".

The symptom: 6 months later you need to filter by type, language, date. Re-indexing 5M docs costs ~$50-100 + downtime.

How to prevent it: the workshop up front. Think 12 months ahead. Better to add 3 extra fields on day 1 than to migrate later.

Trap 2: a schema bloated with fields that are never used

The mistake: "let's add 30 fields just in case".

The symptom: the RAM used by metadata is 2x larger than necessary. Indexing is slower. New devs get confused about which fields to use.

How to prevent it: balance — only add what you'll use in the next 12 months. The stakeholder workshop defines the cutoff.

Trap 3: fields without a closed enum fragment

The mistake: {"category": <free text>} allows any value at all.

The symptom: after 6 months you have "tutorial", "Tutorial", "tut", "tutoriaL", "tutorial-doc", "tutorials" in the field. Filters lose coverage because they only match one of them.

How to prevent it: Pydantic Literal types, or validation against an ALLOWED_VALUES list. Reject any value outside the list.

Trap 4: timestamps as strings

The mistake: {"created_at": "2024-04-30"}.

The symptom: the filter where={"created_at": {"$gte": "2024-01-01"}} may work by coincidence (string comparison) but it fails with inconsistent formats.

How to prevent it: always a Unix timestamp int. Convert from ISO at ingest time:

"created_at": int(datetime.fromisoformat(date_str).timestamp())

Trap 5: fields with ultra-high cardinality

The mistake: using {"hash": "<a 64-char hash>", "uuid": "<uuid>"} as a filter.

The symptom: ChromaDB/Pinecone don't build an efficient index over fields with 1M+ unique values. The filters get slow.

How to prevent it: fields used as filters should have cardinality <10K. For unique identifiers (hashes, UUIDs), store them in metadata but don't use them as filters.

Trap 6: forgetting to version the schema

The mistake: without versioning, you don't know which docs are on schema v1 vs v2.

The symptom: during the progressive migration you can't tell the migrated docs from the ones still pending.

How to prevent it: add "schema_version": "v2" to every doc. The filter where={"schema_version": {"$ne": "v2"}} shows which ones still need migrating.

Applied exercise

Scenario: you're an AI Engineer at a SaaS company for managing medical clinics. The data:

• 50 clinics (tenants)
• ~200K documents per clinic (patient records, protocols, papers, manuals)
• Languages: Spanish, Portuguese, English
• Compliance: HIPAA + GDPR
• Typical queries: doctors and nurses searching for guidelines and protocols

Your job:

1. Design the complete metadata schema with Pydantic.
2. Justify each field (universal vs domain-specific).
3. Define the enums for the categorical fields.

# clinic_metadata.py
from pydantic import BaseModel, Field
from typing import Literal


class ClinicalDocumentMetadata(BaseModel):
    """The metadata schema for clinical documents."""

    # === UNIVERSALS ===

    tenant_id: str = Field(..., description="The clinic's ID. HIPAA isolation.")
    doc_id: str
    chunk_index: int = Field(..., ge=0)
    source: str = Field(..., description="The path to the original document.")
    created_at: int = Field(..., description="Unix timestamp.")

    # === SPECIFIC TO THE MEDICAL DOMAIN ===

    # Document type (to segment the queries)
    type: Literal[
        "patient_record",
        "protocol",
        "clinical_guideline",
        "research_paper",
        "training_material",
        "policy",
    ]

    # Audience (filters by role)
    audience: Literal[
        "clinical",      # doctors, nurses
        "administrative", # admin, finance
        "patient",       # content for patients
        "all",
    ]

    # Language
    language: Literal["es", "pt", "en"]

    # Medical specialty (a closed enum)
    specialty: Literal[
        "general",
        "cardiology",
        "pediatrics",
        "oncology",
        "neurology",
        "psychiatry",
        "internal_medicine",
        "emergency",
        "surgery",
        "obstetrics_gynecology",
    ]

    # PHI sensitivity (filters by role)
    contains_phi: bool = Field(..., description="Does it contain Protected Health Info?")

    # The document's version
    version: str = Field(..., regex=r"^\d+\.\d+$")

    # Regulatory status
    regulatory_status: Literal["draft", "approved", "deprecated", "under_review"]

    # Open tags for sub-segmentation
    tags: list[str] = Field(default_factory=list)

    # Schema versioning (for migrations)
    schema_version: Literal["v1"] = "v1"

    class Config:
        extra = "forbid"

# A cardiologist searches for approved protocols
where={
    "tenant_id": "clinic_acme",
    "specialty": "cardiology",
    "type": "protocol",
    "regulatory_status": "approved",
    "language": "es",
}

# Admin staff search only non-clinical docs
where={
    "tenant_id": "clinic_acme",
    "audience": "administrative",
    "contains_phi": False,
}

# A researcher searches for recent papers
where={
    "tenant_id": "clinic_acme",
    "type": "research_paper",
    "created_at": {"$gte": one_year_ago_ts},
    "language": {"$in": ["en", "es"]},
}

def ingest_clinical_doc(content: str, raw_metadata: dict):
    try:
        validated = ClinicalDocumentMetadata(**raw_metadata)
    except ValidationError as e:
        # Quarantine for review
        save_to_quarantine(content, raw_metadata, e.errors())
        log_metric("ingest_validation_failed", tenant=raw_metadata.get("tenant_id"))
        return None

    # Extra PHI validation (regulatory)
    if validated.contains_phi and validated.audience == "patient":
        raise SecurityError("A doc with PHI cannot have the 'patient' audience")

    return collection.add(
        documents=[content],
        metadatas=[validated.model_dump()],
        ids=[f"{validated.doc_id}_{validated.chunk_index}"],
    )

Summary and next step

What you learned:

• Five universal fields: tenant_id, doc_id+chunk_index, source, created_at, type.
• Domain-specific fields: language, version, audience, etc.
• A stakeholder workshop up front defines the contract — worth the 60-90 minutes.
• Four conventions: snake_case, closed enums, numeric types for dates, descriptive naming.
• Pre-ingest validation with Pydantic. The decision: fail loud (dev), skip+log (prod), or quarantine.
• Legacy migration: re-index with a backfill, or the progressive option with schema_version.
• The traps: a schema that's too minimalist (a future re-index), a bloated one (overhead), open enums (fragmentation), timestamps as strings.

Checkpoint: before moving on, you should be able to:

• List the 5 universal fields and justify each one.
• Run a schema design workshop in under 90 min.
• Implement validation with Pydantic + appropriate error handling.

Next capsule: 04 — Filters with ChromaDB's where clause.

You have the schema. Now you'll learn the full filter syntax: equality, ranges, IN, AND/OR. Capsule 04 is the concrete practice on top of this capsule's theoretical schema.

Resources

1. Pydantic — Data Validation — Models and validation
2. ChromaDB — Metadata Filtering — The supported operators
3. JSON Schema — Formal validation
4. Martin Fowler — Data Modeling — General patterns
5. Anthropic — Contextual Retrieval — A complementary pattern
6. GDPR Data Mapping Guide — Compliance and metadata

Estimated time: 30-35 minutes Next: 04-filters-with-chromadb-where.md