Capsule 07: Persistence and Durability — what happens when something breaks

Capsule description

Until now you worked as if the data lived forever. You inserted vectors, searched them, tuned configuration. But there's a question we didn't ask: what happens when the machine shuts down, the process crashes, or someone deletes the directory by mistake?

In development, the answer is almost always "I re-run the ingestion, it's 5 minutes". In production, it's the difference between "the system comes back on its own in 30 seconds" and "we lost 2 days of ingestion work and users see errors until we regenerate everything". This capsule teaches you to configure persistence correctly and to design a backup strategy that lets you sleep easy when your RAG is in production.

It's not a glamorous capsule. You won't learn a new algorithm. But the day something breaks in production, this is what separates a "minor incident" from a "post-mortem that lasts three weeks".

By the end of this capsule, you'll be able to:

• ✅ Differentiate EphemeralClient, PersistentClient, and HttpClient by context
• ✅ Understand the structure of ChromaDB's data directory and which file stores what
• ✅ Implement three backup strategies: offline copy, JSON export, incremental snapshot
• ✅ Design a disaster recovery plan with concrete RTO and RPO
• ✅ Recover a corrupt collection without losing data when a backup is available
• ✅ Anticipate the most expensive disaster: thinking you have a backup when you actually never verified it

Estimated time: 25-30 minutes

Three client types, three persistence guarantees

ChromaDB offers three client modes. The operational difference matters more than it seems.

EphemeralClient — everything in RAM

import chromadb

client = chromadb.Client()  # equivalent to chromadb.EphemeralClient()
collection = client.create_collection("temp")
collection.add(documents=["hello"], ids=["1"])
# When the Python process ends, everything is lost

When to use:

• Unit tests (collections that shouldn't persist between tests)
• Exploration notebooks (you'll delete everything anyway)
• Ephemeral demos

When NOT to use:

• Any ingestion that takes more than 5 minutes
• Any system that serves real queries
• Production, almost without exception

Risk: you run chromadb.Client() out of habit, ingest 1M docs over 4 hours, close the terminal — you lost everything. It's happened more times than is publicly admitted.

PersistentClient — data on local disk

client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_or_create_collection("docs")
collection.add(documents=["hello"], ids=["1"])
# Data in ./chroma_db/, survives the process closing

When to use:

• Most projects: a single process that writes and reads
• Setups where you don't need to share data between multiple processes/machines
• Small to medium scale production

When NOT to use:

• Multiple processes accessing simultaneously — SQLite serializes but there's contention
• You need horizontal distribution (several servers writing to the same DB)
• Your ingestion pipeline runs on one machine and the query API on another

Guarantees:

• Data persists across process restarts
• Writes are committed to SQLite + parquet on disk
• ⚠️ But the disk itself isn't backed up — if the machine breaks, you lose everything

HttpClient — centralized server

# 1) On one machine, start the server:
#    chroma run --path /data/chroma_db --port 8000

# 2) On clients, connect via HTTP:
client = chromadb.HttpClient(host="chromadb.mycompany.com", port=8000)
collection = client.get_or_create_collection("docs")
collection.add(documents=["hello"], ids=["1"])

When to use:

• Multiple services that need to share the same collection
• Layer separation: ingestion on one host, API on another
• Medium to large scale production with centralized backups

Guarantees:

• You centralize backups, monitoring, resources
• Cost: network latency between clients and server (~1-5ms LAN, ~50-100ms WAN)

Common trade-off: PersistentClient to start; HttpClient when the system grows or you need multiple consumers.

Structure of the data directory

When you use PersistentClient(path="./chroma_db"), this is what gets created:

./chroma_db/
├── chroma.sqlite3              # ← global metadata (collections, IDs, configurations)
├── <collection-uuid-1>/
│   ├── data_level0.bin         # ← vectors in binary format
│   ├── header.bin              # ← HNSW index header
│   ├── length.bin
│   ├── link_lists.bin          # ← HNSW graph (connections between nodes)
│   └── index_metadata.pickle   # ← HNSW config (M, construction_ef, etc.)
├── <collection-uuid-2>/
│   └── ...

What each thing stores:

Approximate size:

For 1M vectors × 1536 dim × float32:
  data_level0.bin:           ~6 GB (vectors)
  link_lists.bin:            ~2 GB (HNSW graph with M=32)
  chroma.sqlite3:            ~500 MB (metadata + IDs)
  total:                     ~8.5 GB

Implicit plan: any backup strategy has to cover the entire directory. Copying chroma.sqlite3 alone isn't enough — you'd lose the HNSW index.

Three backup strategies

Strategy 1: directory copy (the simplest)

Concept: stop the ChromaDB process, copy the entire directory, restart.

# backup_offline.py
import shutil
import os
from datetime import datetime
from pathlib import Path


def backup_chroma_offline(source_dir: str, backup_root: str) -> str:
    """
    Full backup of the ChromaDB directory.
    REQUIRES the client to be closed (no processes writing).
    """
    source = Path(source_dir)
    if not source.exists():
        raise FileNotFoundError(f"Source not found: {source_dir}")

    timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
    backup_path = Path(backup_root) / f"chroma_backup_{timestamp}"

    print(f"Backing up {source_dir} → {backup_path}")
    shutil.copytree(source, backup_path)

    # Verify the backup has the critical file
    if not (backup_path / "chroma.sqlite3").exists():
        raise RuntimeError("Backup invalid: chroma.sqlite3 not found")

    size_mb = sum(f.stat().st_size for f in backup_path.rglob("*") if f.is_file()) / 1024 / 1024
    print(f"Backup OK: {size_mb:.1f} MB")
    return str(backup_path)


# Usage (make sure no process has the DB open)
backup_path = backup_chroma_offline(
    source_dir="./chroma_db",
    backup_root="./backups",
)

Pros:

• Simple, predictable, atomic (if the copy completes, the backup is valid)
• Recovery is trivial: copy the directory back

Cons:

• Requires downtime — no active writes during the copy
• If you have 50 GB of data, the copy can take minutes

When to use it: systems with a maintenance window, small-medium datasets (<100 GB).

Strategy 2: export to JSON (portable, slow)

Concept: you export the content of each collection to JSON. You import it into another environment when needed.

# backup_export.py
import json
from pathlib import Path
import chromadb


def export_collection_to_json(
    collection: chromadb.Collection,
    output_file: str,
    batch_size: int = 1000,
):
    """
    Exports a complete collection to JSON with its embeddings.
    Useful for: portability between environments, migration between versions, archive.
    """
    output_path = Path(output_file)
    output_path.parent.mkdir(parents=True, exist_ok=True)

    total = collection.count()
    print(f"Exporting {total} docs from '{collection.name}'")

    all_data = {
        "name": collection.name,
        "metadata": collection.metadata,
        "documents": [],
        "metadatas": [],
        "embeddings": [],
        "ids": [],
    }

    # Paginate for large datasets
    offset = 0
    while offset < total:
        batch = collection.get(
            limit=batch_size,
            offset=offset,
            include=["documents", "metadatas", "embeddings"],
        )
        all_data["documents"].extend(batch["documents"])
        all_data["metadatas"].extend(batch["metadatas"])
        all_data["embeddings"].extend(batch["embeddings"])
        all_data["ids"].extend(batch["ids"])
        offset += batch_size

    with open(output_path, "w") as f:
        json.dump(all_data, f)

    size_mb = output_path.stat().st_size / 1024 / 1024
    print(f"Exported to {output_path} ({size_mb:.1f} MB)")


def import_collection_from_json(
    client: chromadb.Client,
    json_file: str,
    new_collection_name: str | None = None,
    batch_size: int = 200,
) -> chromadb.Collection:
    """Restores a collection from a JSON export."""
    with open(json_file) as f:
        data = json.load(f)

    name = new_collection_name or data["name"]
    collection = client.get_or_create_collection(
        name=name,
        metadata=data.get("metadata"),
    )

    # Insert in batches
    total = len(data["ids"])
    print(f"Importing {total} docs into '{name}'")

    for i in range(0, total, batch_size):
        end = min(i + batch_size, total)
        collection.add(
            documents=data["documents"][i:end],
            metadatas=data["metadatas"][i:end],
            embeddings=data["embeddings"][i:end],  # Reuses embeddings, doesn't recompute
            ids=data["ids"][i:end],
        )

    print(f"Restored. Verify: collection.count() = {collection.count()}")
    return collection

Pros:

• Portable: the JSON works on any ChromaDB version
• Reuses embeddings — no cost to re-embed with OpenAI
• Useful for manual versioning with git LFS or for migrating between environments

Cons:

• 2-3x more space than the binary format (JSON is inefficient)
• Restore is slow: it has to rebuild the HNSW index from scratch
• For 1M vectors the JSON can weigh 20+ GB

When to use it: migration between versions, long-term archive, environments where you can't copy the binary format.

Strategy 3: continuous snapshot with time metadata (incremental)

Concept: you add created_at to each chunk in metadata. Incremental backups only cover new chunks.

# Metadata schema with timestamp
import time

def add_with_timestamp(collection, documents, ids, metadatas=None):
    now = int(time.time())
    metas_with_time = []
    for i, m in enumerate(metadatas or [{}] * len(documents)):
        meta_copy = dict(m)
        meta_copy["created_at"] = now
        metas_with_time.append(meta_copy)
    collection.add(documents=documents, ids=ids, metadatas=metas_with_time)


def export_incremental(collection, since_timestamp: int, output_file: str):
    """Exports only documents created after since_timestamp."""
    new_docs = collection.get(
        where={"created_at": {"$gte": since_timestamp}},
        include=["documents", "metadatas", "embeddings"],
    )
    print(f"Found {len(new_docs['ids'])} docs since {since_timestamp}")

    with open(output_file, "w") as f:
        json.dump(new_docs, f)

    return len(new_docs["ids"])

Pros:

• Incremental backups are much faster than full ones
• Useful when the dataset grows but few docs change per day

Cons:

• Requires discipline: ALL inserts have to set created_at
• ChromaDB doesn't have automatic timestamps — you add them
• If you make a mistake in a migration, the incremental doesn't catch those changes

When to use it: systems with a lot of incremental ingestion (chat logs, events, daily new docs).

Disaster recovery: RTO and RPO

Before designing your strategy, define two numbers:

RTO (Recovery Time Objective): how long can you tolerate the system being down? "If the DB gets corrupted, in how many minutes/hours do we have to be back?"

RPO (Recovery Point Objective): how much data can you tolerate losing? "If the DB gets corrupted, is it acceptable to lose the last X minutes/hours of ingestion?"

Examples by system type:

Your strategy derives from these numbers:

• RTO < 1 hour: directory copy + a tested restore script. 30-50 GB restores in ~10-15 minutes.
• RPO < 1 hour: backups every hour, not daily. If your cron is 0 2 * * * (2 AM daily), you have an RPO of 24 hours.
• RTO < 15 min: hot standby — a second node with continuous replication. It goes beyond the scope of basic ChromaDB, requires a distributed setup (covered in M07 and guide #18).

Minimum operational plan for production:

# disaster_recovery_plan.yml (concept)
backup_strategy:
  type: directory_copy_offline
  frequency: every 6 hours
  retention: 7 days
  destination: s3://my-backups/chroma/
  verify: yes (try to open the backup on each run)

monitoring:
  - alert if last backup > 12 hours ago
  - alert if backup size differs ±20% from the previous one
  - alert if verify fails

runbook_recovery:
  - 1. Detect corruption (queries fail, count() inconsistent)
  - 2. Identify the last valid backup (verify passes)
  - 3. Stop service
  - 4. mv chroma_db chroma_db_corrupt_$(date)
  - 5. Restore: aws s3 sync s3://backups/<latest> ./chroma_db
  - 6. Verify count() vs expected
  - 7. Start service
  - 8. Notify data loss after the backup (in the RPO window)

Verifying backups (the part almost nobody does)

The most expensive anti-pattern: assuming you have backups because you run a script. On incident day you discover that:

• The backup is corrupt
• Only part of the directory was backed up
• The backup is 3 weeks old because the cron stopped silently

How to avoid it:

def verify_backup(backup_path: str) -> bool:
    """
    Verifies that a backup is functional by trying to open it and run a query.
    Returns True only if everything works.
    """
    backup = Path(backup_path)

    # 1. Critical files exist
    if not (backup / "chroma.sqlite3").exists():
        print(f"❌ Missing chroma.sqlite3 in {backup_path}")
        return False

    # 2. Reasonable size
    size_mb = sum(f.stat().st_size for f in backup.rglob("*") if f.is_file()) / 1024 / 1024
    if size_mb < 1:
        print(f"❌ Backup suspiciously small ({size_mb:.1f} MB)")
        return False

    # 3. Open as a client
    try:
        client = chromadb.PersistentClient(path=str(backup))
        collections = client.list_collections()
    except Exception as e:
        print(f"❌ Can't open as ChromaDB: {e}")
        return False

    if not collections:
        print(f"⚠️ Backup with no collections")
        return False

    # 4. Run a trivial query on each collection
    for col in collections:
        try:
            count = col.count()
            if count == 0:
                print(f"⚠️ Collection '{col.name}' empty")
                continue
            # Query test
            sample = col.peek(limit=1)
            if not sample["ids"]:
                print(f"❌ Collection '{col.name}': peek returned empty")
                return False
        except Exception as e:
            print(f"❌ Collection '{col.name}': error in peek - {e}")
            return False

    print(f"✅ Valid backup: {len(collections)} collections, {size_mb:.1f} MB")
    return True


# Integrate into the backup cron
def backup_with_verification(source: str, backup_root: str) -> str:
    backup_path = backup_chroma_offline(source, backup_root)
    if not verify_backup(backup_path):
        raise RuntimeError(f"Backup verification FAILED: {backup_path}")
    return backup_path

Recommended verify frequency:

• Every backup: at least a basic check (exists, size OK).
• Weekly: full restore to a staging environment and real queries.
• Quarterly: a disaster recovery drill — simulate an incident and measure the real RTO.

Traps and common mistakes

Trap 1: backup while the process is writing

The mistake:

# While this process is active:
client = chromadb.PersistentClient(path="./chroma_db")
collection = client.get_collection("docs")
# ... inserting data...

# Another process tries to back up:
shutil.copytree("./chroma_db", "./backup_chroma_db_NOW")

Symptom: the backup ends up partially written or corrupt. SQLite may have pending writes.

How to prevent it: stop the client before copying. If the system needs 24/7 availability, use replication (out of scope for basic ChromaDB).

Workaround: a filesystem-level snapshot (LVM snapshot, ZFS snapshot) that captures a consistent point atomically.

Trap 2: backing up only the SQLite file

The mistake: "the important file is chroma.sqlite3, I copy only that".

Symptom: on restore, the collection exists but the HNSW index is empty or inconsistent. Queries return zero results.

How to prevent it: back up the entire directory. The HNSW *.bin files are as critical as chroma.sqlite3.

Trap 3: a backup cron that fails silently

The mistake: you scheduled 0 2 * * * for a daily backup. It worked for 6 months. Then it stopped running because the cron user didn't have permissions on the destination. Nobody noticed.

Symptom: on incident day, the most recent backup is 4 months old.

How to prevent it:

• Health check: the API exposes /last_backup_timestamp. Monitoring alerts if > 26 hours ago (margin over the daily cycle).
• Alerts in the cron: redirect output to a file + review it, or use a tool like healthchecks.io that alerts if it doesn't receive a ping.

Trap 4: not testing the restore

The mistake: you've had daily backups for 2 years. You never tested them. On incident day, you discover the format changed between ChromaDB versions and the old backup can't be opened with the current binary.

How to prevent it:

• Quarterly DR drill: take a random backup, restore it in staging, run queries. Measure the real RTO.
• Document the ChromaDB version of the backup in the file name (chroma_backup_v0.5.3_20260508.tar.gz).

Trap 5: backups on the same machine

The mistake: the machine's disk dies. You had backups in /var/backups/chroma/ — the same machine you lost.

How to prevent it: 3-2-1 rule: 3 copies, on 2 different media, 1 off-site.

• 1 copy in production (the live system)
• 1 backup on a different disk on the same machine (fast to recover from minor errors)
• 1 backup in remote storage (S3, GCS, Backblaze) to recover from a site disaster

Trap 6: infinite retention = infinite cost

The mistake: you keep all backups since day 1. After 2 years with daily backups + a backup weighing 50 GB → 36 TB of storage.

How to prevent it: a clear retention policy:

• Daily backups: keep the last 7 days.
• Weekly backups: keep the last 4 weeks.
• Monthly backups: keep the last 12 months.
• Yearly backups (compliance): keep N years per regulation.

Implement it with an automatic cleanup script:

def cleanup_old_backups(backup_root: str, retention_days: int = 7):
    cutoff = time.time() - (retention_days * 86400)
    for backup_dir in Path(backup_root).iterdir():
        if backup_dir.is_dir() and backup_dir.stat().st_ctime < cutoff:
            shutil.rmtree(backup_dir)
            print(f"Deleted old backup: {backup_dir.name}")

Applied exercise

Scenario: you're an AI Engineer at a legal services company. You operate a RAG with:

• 500K chunks of case law, persistent ChromaDB, ~4 GB on disk
• 50-100 queries/hour, mainly between 9 AM and 7 PM
• Regulatory compliance: they must be able to reconstruct answers from up to 90 days ago
• The stakeholder says: "we need guarantees that we never lose data"

Your task: design the complete backup strategy. Define RTO, RPO, frequency, retention, backup location, monitoring, and a recovery runbook.

RETENTION = {
    "hot_backup": "24 hours",
    "daily_backup_local": "7 days",
    "daily_backup_remote": "90 days",   # compliance
    "weekly_backup": "12 months",
    "monthly_backup": "5 years",        # compliance + audit
}

# /etc/cron.d/chroma_backup

# Hot backup every 30 min during business hours
*/30 8-19 * * 1-5 /usr/local/bin/chroma_hot_backup.sh

# Daily full backup at 3 AM
0 3 * * * /usr/local/bin/chroma_daily_backup.sh

# Weekly backup Sundays at 4 AM
0 4 * * 0 /usr/local/bin/chroma_weekly_backup.sh

# Cleanup old backups daily
0 5 * * * /usr/local/bin/chroma_cleanup_backups.sh

# chroma_daily_backup.sh
#!/bin/bash
set -euo pipefail

DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/var/backups/chroma/daily/chroma_${DATE}"
S3_BUCKET="s3://legal-rag-backups/chroma/daily/"

# 1. Stop service
systemctl stop chroma-rag

# 2. Backup
rsync -a /data/chroma_db/ "${BACKUP_DIR}/"

# 3. Start service
systemctl start chroma-rag

# 4. Verify
python /usr/local/bin/verify_backup.py "${BACKUP_DIR}"

# 5. Compress and upload to S3
tar czf "${BACKUP_DIR}.tar.gz" -C "$(dirname ${BACKUP_DIR})" "$(basename ${BACKUP_DIR})"
aws s3 cp "${BACKUP_DIR}.tar.gz" "${S3_BUCKET}"

# 6. Notify health check
curl -fsS -o /dev/null https://hc-ping.com/{check-uuid}

# 7. Cleanup local archive (keep only the directory, S3 has the tar.gz)
rm "${BACKUP_DIR}.tar.gz"

# Metrics exported by the system
metrics = {
    "chroma_backup_last_success_timestamp": <unix_ts>,
    "chroma_backup_last_size_bytes": <size>,
    "chroma_backup_verify_success": 0 | 1,
}

# Alerts in Datadog/Grafana
alerts = [
    {
        "name": "ChromaBackupStale",
        "condition": "now() - chroma_backup_last_success_timestamp > 90 minutes",
        "severity": "page_oncall",
        "runbook": "https://wiki/runbooks/chroma-backup-stale",
    },
    {
        "name": "ChromaBackupSizeAnomaly",
        "condition": "abs(size_today - size_yesterday) / size_yesterday > 0.20",
        "severity": "ticket",
        "runbook": "https://wiki/runbooks/chroma-backup-size",
    },
    {
        "name": "ChromaBackupVerifyFailed",
        "condition": "chroma_backup_verify_success == 0",
        "severity": "page_oncall",
        "runbook": "https://wiki/runbooks/chroma-backup-verify",
    },
]

# Runbook: Chroma DB corruption / data loss

## Detection
- Queries fail with a schema error
- `collection.count()` returns an unexpected number
- Logs show SQLite or HNSW corruption

## Recovery (target RTO: 1 hour)

1. **Stop service** (5 min)
   ```bash
   systemctl stop chroma-rag

### 7. DR drill (quarterly)

Every 3 months:
1. Take a random backup from the last 90 days.
2. Restore it in a staging environment.
3. Run a predefined query suite.
4. Document the total time and problems found.
5. If the time > target, adjust the pipeline.

**Total estimated cost:**

- S3 storage: $20/month
- Compute for hot backups (incremental rsync): negligible
- Operator time: ~2 hours/month (quarterly drill + adjustments)

**Communication to the stakeholder:**

> *"We implemented a backup every 30 min during active hours + a daily full backup, with 90-day retention in S3 (compliance) and 5 years for weekly/monthly. RTO target 1 hour, RPO 30 min. What we CAN'T guarantee is 'zero loss' — that would require synchronous replication to a hot standby, which triples the infra cost and isn't justified for our case. Our RPO of 30 min means that in the worst-case scenario (corruption right after the last hot backup), we can lose up to 30 minutes of logged queries. For the chunks of the legal dataset, the loss would be zero because they only change with scheduled releases."*

</details>

---

## Summary and next step

**What you learned:**

- Three client modes: `EphemeralClient` (RAM, doesn't persist), `PersistentClient` (local disk), `HttpClient` (centralized server).
- The ChromaDB directory has `chroma.sqlite3` (global metadata) + subdirectories per collection with binary files of the HNSW index.
- Three backup strategies: offline directory copy (simplest), JSON export (portable, slow), incremental with timestamps (efficient for growing datasets).
- RTO and RPO are the two numbers that define your strategy. Agree on them with stakeholders before choosing technology.
- Verifying backups on every run + a periodic DR drill is what separates "having backups" from "having backups that work".
- 3-2-1 rule: 3 copies, 2 media, 1 off-site. A backup on the same machine isn't a real backup.
- An explicit retention policy prevents storage from growing infinitely.

**Checkpoint:** before moving on, you should be able to:

- [ ] Differentiate `EphemeralClient` vs `PersistentClient` vs `HttpClient` and when to use each.
- [ ] Explain why backing up only `chroma.sqlite3` isn't enough.
- [ ] Define realistic RTO and RPO for a given system.
- [ ] Design automatic backup verification in the operational pipeline.

**Next capsule: 08 — Document Search mini-project.**

You close the first arc of M4 (capsules 01-08) by building a complete Document Search System: 10K documents, validated batch ingestion, queries with metadata filtering, persistence, and benchmarks that validate p95 <20ms with throughput >1K docs/sec. It's the practical consolidation of everything you learned before moving on to the RAG arc (capsules 09-11) that covers embeddings with OpenAI, chunking, and an end-to-end RAG pipeline.

---

## Resources

1. [ChromaDB — Persistence](https://docs.trychroma.com/usage-guide#initiating-a-persistent-chroma-client) — Official PersistentClient documentation
2. [ChromaDB — Running as a Server](https://docs.trychroma.com/deployment) — HttpClient setup with a centralized server
3. [SQLite — Backup API](https://www.sqlite.org/backup.html) — How SQLite handles atomic backup
4. [Google SRE — Backup and Recovery](https://sre.google/sre-book/data-integrity/) — General backup principles
5. [The 3-2-1 Backup Rule](https://www.backblaze.com/blog/the-3-2-1-backup-strategy/) — Classic backup rule
6. [Postgres backup-and-restore concepts (applicable to ChromaDB)](https://www.postgresql.org/docs/current/backup.html) — Transferable best practices

---

**Estimated time:** 25-30 minutes
**Next:** [08-mini-project-document-search.md](08-mini-project-document-search.md)