Module 8: Capstone RAG Project with ChromaDB

Capsule 02: Project Architecture

Capsule description

You will define the system architecture before coding. This decision avoids rework and makes testing, monitoring, and maintenance easier. A clear design lets you scale ingestion and serving independently, migrate components without breaking everything, and keep stable contracts between parts.

Main components

The RAG system is made up of five logical blocks:

Component	Responsibility	Input	Output
ingestion	Loading, cleaning, chunking, and embeddings	Raw documents (PDF, TXT, MD)	Vectors + metadata in ChromaDB
vector_store	Persistence and querying of vectors	Vectors, IDs, metadata	Similarity search results
retrieval	Top-k + metadata filters	User query	Ranked documents with scores
generation	Context assembly and LLM answer	Query + retrieved documents	Generated answer + citations
api	Endpoints for ingest, search, ask	HTTP requests	JSON responses

Detailed responsibilities per component

Ingestion

Loading: Supports TXT, MD, RST (and PDF with pypdf). Detects encoding (UTF-8, Latin-1).
Cleaning: Whitespace normalization, handling of special characters.
Chunking: RecursiveCharacterTextSplitter with separators \n\n, \n, . , .
Metadata: Extracts and assigns doc_id, source, chunk_index, title to each chunk.
Embeddings: Calls OpenAI in batches; handles rate limits and retries.
Persistence: Adds to ChromaDB in batches; must not mix retrieval business logic.

Vector Store (ChromaDB)

Storage: On-disk persistence (PersistentClient).
Indexing: HNSW by default; requires no additional configuration for 1K-10K vectors.
Query: Similarity search with query_embeddings, n_results, where for filters.
No business logic: It only receives vectors and returns results; it doesn't decide what is "relevant".

Retrieval

Query embedding: Same function/model as ingestion for consistency.
Search: Calls ChromaDB with top_k and optional filters.
Post-processing: Applies a score threshold; discards results below the threshold.
Output format: A standardized structure so that generation doesn't depend on ChromaDB.

Generation

Context assembly: Concatenates chunks with separators and source metadata.
Prompt engineering: Clear instructions: "answer only with the context", "don't make things up".
LLM call: GPT-3.5-turbo with low temperature (0.2) for consistency.
Post-processing: Formats sources for the response contract; doesn't modify the generated content.

API

Routing: FastAPI with routers per domain (ask, search, ingest, collections).
Validation: Pydantic for request bodies; length limits.
Responses: A stable JSON contract; a trace_id in every response.
Documentation: Automatic Swagger and ReDoc.

Architecture diagram (ASCII)

┌─────────────────────────────────────────────────────────────────────────────────────────┐
│                               RAG SYSTEM - ARCHITECTURE                                  │
└─────────────────────────────────────────────────────────────────────────────────────────┘

  ┌──────────────┐     ┌─────────────────────────────────────────────┐     ┌──────────────┐
  │  Documents   │     │              INGESTION PIPELINE             │     │   ChromaDB   │
  │  (PDF, TXT,  │────▶│  Load → Chunk (512) → Embed → Metadata      │────▶│   (vector    │
  │   MD, etc)   │     │  Batch: 1K-2K docs, progress, error handling│     │   store)     │
  └──────────────┘     └─────────────────────────────────────────────┘     └──────┬───────┘
                                                                                │
  ┌──────────────┐     ┌─────────────────────────────────────────────┐         │
  │     User     │     │              RETRIEVAL PIPELINE               │         │
  │   (client)   │────▶│  Query → Embed → Similarity Search → Top-K   │◀────────┘
  │              │     │  Metadata filtering, score threshold         │
  └──────────────┘     └────────────────────┬────────────────────────┘
                                             │
                                             ▼
  ┌──────────────┐     ┌─────────────────────────────────────────────┐
  │   Response   │     │             GENERATION PIPELINE              │
  │   + sources  │◀────│  Context assembly → Prompt → GPT-3.5-turbo   │
  │   + trace_id │     │  Citation generation, fallback if no context │
  └──────────────┘     └─────────────────────────────────────────────┘
         ▲
         │
  ┌──────┴───────────────────────────────────────────────────────────┐
  │                      FastAPI REST API                             │
  │  POST /ingest | GET /search | POST /ask | GET /collections | ...   │
  └──────────────────────────────────────────────────────────────────┘

Detailed data flow

Ingestion flow (offline or on demand)

Source documents
    → Document Loader (PDF, TXT, MD by type)
    → Text cleaning (normalization, encoding)
    → RecursiveCharacterTextSplitter (chunk_size=512, overlap=50)
    → Chunks with metadata (source, doc_id, chunk_index)
    → OpenAI text-embedding-3-small (batch 100-200)
    → ChromaDB add (batch 1000-2000)
    → Validation: count, mandatory metadata, no duplicates

Query flow (online)

User question
    → Input validation (length, format)
    → Query embedding (OpenAI text-embedding-3-small)
    → ChromaDB query (top_k=5, optional where filters)
    → Score threshold (discard < 0.5 or similar)
    → Context assembly (concatenate chunks)
    → Prompt engineering (instructions + context + question)
    → GPT-3.5-turbo (temperature=0.2)
    → Post-processing: extract citations, format sources
    → JSON response: answer, sources, confidence, trace_id

Design decisions and justification

Why ChromaDB?

No infrastructure cost: Open-source, runs locally.
Fast for prototyping: On-disk persistence with PersistentClient, requires no external server.
Sufficient for 1K-10K documents: HNSW by default, metadata filtering.
A learning base: The concepts (CRUD, similarity search, metadata) transfer to Pinecone, Weaviate, Qdrant.

For large-scale production (millions of vectors, multi-tenant), you would migrate to Pinecone or Weaviate; ChromaDB is optimal for this project.

Quick comparison:

Criterion	ChromaDB	Pinecone	Weaviate
Infra cost	Free (self-hosted)	Per usage (serverless) or plan	Self-hosted or cloud
Typical scale	1K-100K vectors	Millions	Millions
Setup	`pip install`	Account, API key	Docker or cloud
Use in this project	✅ Learning, portfolio	Guide #8	Alternative

Why chunk_size=512 (approximate tokens)?

Context vs granularity balance: Very large chunks dilute relevance; very small ones lose context.
Embedding model: text-embedding-3-small supports up to 8191 tokens; 512 leaves margin and avoids truncation.
Retrieval: 5 chunks × 512 ≈ 2500 tokens of context for the LLM, within GPT-3.5's reasonable limit (4K context window for generation).
Standard in RAG: Many tutorials use 500-1000; 512 is a proven middle point.

Why chunk_overlap=50?

Semantic continuity: Avoids cutting sentences or paragraphs in half; the overlap keeps an idea that crosses the boundary retrievable.
50 tokens: Enough to not lose context, without duplicating too much content (fewer vectors, lower storage cost).

Why GPT-3.5-turbo?

Cost: More economical than GPT-4 for a learning and portfolio project.
Latency: Faster than GPT-4.
Sufficient for RAG: With good retrieval and prompt engineering, GPT-3.5-turbo produces correct answers most of the time.
Upgrade path: You can switch to GPT-4 or GPT-4o via an environment variable without changing the code.

When to use GPT-4: If the domain is very technical or you need deeper reasoning, move up to GPT-4. For most RAG with good retrieval, GPT-3.5-turbo is sufficient.

Why separate ingestion from serving?

Independent scaling: Ingestion can be a heavy batch/job; serving must be low-latency.
Resources: Ingestion consumes CPU/memory for chunking and embedding API calls; serving is more I/O-bound (ChromaDB query + LLM).
Deployment: You can run ingestion as a separate script or worker; the API only serves queries.

Contracts between components

Ingestion → Vector Store

Field	Type	Required	Description
`ids`	list[str]	Yes	Unique, deterministic IDs (e.g. `{doc_id}_{chunk_idx}`)
`documents`	list[str]	Yes	Chunk text
`embeddings`	list[list[float]]	Yes	Vectors from text-embedding-3-small
`metadatas`	list[dict]	Yes	At least `source`, `doc_id`, `chunk_index`

Retrieval → Generation

Field	Type	Description
`documents`	list[str]	Retrieved chunks
`metadatas`	list[dict]	source, doc_id, score
`ids`	list[str]	ChromaDB IDs

API → Client

{
  "answer": "string",
  "sources": [
    {"doc_id": "string", "source": "string", "title": "string", "score": 0.82}
  ],
  "confidence": 0.78,
  "trace_id": "uuid",
  "fallback_reason": null
}

Anti-patterns to avoid

Anti-pattern	Problem	Solution
Mixing retrieval and generation in one monolithic function	Hard to test, coupled	Separate functions: `retrieve()` and `generate()`
Sensitive configuration hardcoded	Security, not portable	Environment variables: `OPENAI_API_KEY`, `CHROMA_PATH`
Changing the metadata schema without versioning	Breaks prior ingestion, inconsistency	Document the schema, use versioning if you change it
Random or non-deterministic IDs	Duplicates on re-ingestion	`f"{doc_id}_{chunk_index}"`
No fallback when retrieval returns little	Made-up answers	Score threshold, explicit "I don't have information" message

Failure points and mitigation

Failure point	Symptom	Mitigation
OpenAI API down	Timeout or 5xx on embeddings/generation	Retry with backoff; clear message to the user
ChromaDB disk full	Error on add	Monitor space; alerts
OpenAI rate limit	429 on embeddings	Adequate batch size; queue with backoff
Corrupt documents	Some empty or invalid chunks	Validate before embedding; skip and log
Very long query	Token limit exceeded	Limit length in the API; truncate if necessary
No documents in the DB	Retrieval always empty	Health check that validates count > 0; message in /ask

Architecture alternatives

Monolithic: A single script that does load → chunk → embed → add. Simpler, but hard to scale.

Microservices: Ingestion, API, and ChromaDB as separate services. Overkill for 1K docs.

Library + CLI + API: Reusable code; a CLI for ingestion; an API for serving. Recommended if you plan to publish.

For this module: Use the 5 components as logical layers within the same repo. You don't need physical microservices.

Testing considerations per component

Component	What to test	How
Ingestion	Correct chunking, complete metadata	Unit test with a sample doc; assert chunk count and keys
Ingestion	Batch add without errors	Integration test with in-memory ChromaDB
Retrieval	Query returns ordered results	Mock ChromaDB or a fixture with known data
Generation	Answer with context, no fabrication	Mock retrieval; assert the answer contains fragments of the context
API	/ask end-to-end	FastAPI test client; mock OpenAI if necessary
API	Fallback when retrieval is empty	Call /ask with an empty DB; assert fallback_reason

Separating components lets you test each one in isolation with mocks, and then an end-to-end integration test with real (or emulated) services.

Configuration per environment

Variable	Development	Production
`CHROMA_PATH`	`./chroma_data` (local)	`/var/data/chroma` or a Docker volume
`OPENAI_API_KEY`	local `.env`	Secrets manager, injected env
`TOP_K`	5-10 (explore more)	5 (recall/latency balance)
`SCORE_THRESHOLD`	0.3 (more permissive)	0.5 (avoid noise)
`LLM_MODEL`	gpt-3.5-turbo	gpt-3.5-turbo or gpt-4 by budget
Log level	DEBUG	INFO or WARN

Centralize everything in a config.py module that reads from os.getenv. Never hardcode values that change between environments.

Decision Log template

To document design decisions, use a structure like this in DESIGN.md:

## Decision Log

### DL-001: ChromaDB as the vector store
**Date:** 2025-03
**Context:** We need a vector store for RAG, limited budget.
**Decision:** Use self-hosted ChromaDB.
**Alternatives:** Pinecone (cost), Weaviate (more setup).
**Consequences:** Free, easy; scaling to 100K+ will require evaluation.

### DL-002: chunk_size=512
**Context:** Balance between granularity and context for the LLM.
**Decision:** 512 characters (~128 tokens).
**Consequences:** Adjust if retrieval is too fragmented or too long.

Keeping a log like this makes it easier to explain "why" to other developers or to your future self.

Project environment variables

Full list of variables you'll use (document in the README):

Variable	Description	Default
`OPENAI_API_KEY`	OpenAI API key	(required)
`CHROMA_PATH`	ChromaDB persistence path	`./chroma_data`
`COLLECTION_NAME`	Collection name	`rag_docs`
`CHUNK_SIZE`	Chunk size in characters	`512`
`CHUNK_OVERLAP`	Overlap between chunks	`50`
`BATCH_SIZE_CHROMADB`	Docs per batch on add	`1000`
`BATCH_SIZE_EMBEDDINGS`	Texts per OpenAI call	`100`
`EMBEDDING_MODEL`	Embedding model	`text-embedding-3-small`
`LLM_MODEL`	Generation model	`gpt-3.5-turbo`
`TOP_K`	Retrieval results	`5`
`SCORE_THRESHOLD`	Minimum score threshold	`0.5`

Design exercises

Exercise 1: Draw your architecture

With pencil and paper or a diagramming tool, draw:

The 5 components (ingestion, vector_store, retrieval, generation, api).
Data flows with arrows (documents → ingestion → ChromaDB; query → retrieval → generation → response).
Failure points (what happens if OpenAI fails? if ChromaDB is down?).
Observability strategy per component (logs, metrics, trace_id).

Solution: Use the ASCII diagram from this capsule as a base. Add:

Failure point: If OpenAI embedding fails → retry with backoff, then fail with a clear message.
ChromaDB down → health check returns 503.
Per component: ingestion logs processed docs/errors; retrieval logs query + top_k; generation logs tokens used; api logs trace_id on each request.

Exercise 2: Define the metadata schema

List the metadata fields you'll store per chunk. Justify each one.

Suggested solution:

source: path or identifier of the original document (for citations).
doc_id: unique document ID (to filter by doc).
chunk_index: index of the chunk within the doc (to order).
title: document title if available (to show in sources).

Optional: created_at, category depending on the domain.

Exercise 3: Design the ID strategy

How do you generate IDs so ingestion is idempotent (re-running doesn't duplicate)?

Solution: doc_id = hash of the path or a stable identifier of the document. chunk_id = f"{doc_id}_chunk_{chunk_index}". On re-ingestion, you use upsert (if ChromaDB supports it) or delete the collection first. Alternatively: check existence before add.

Exercise 4: Fallback when retrieval is weak

Define rules: What to do if the maximum score is < 0.5? If there is only 1 document? If there are 0?

Solution:

Max score < 0.5: don't use context, answer "I didn't find relevant information for your question".
1 document: use it if score > 0.4; indicate low confidence.
0 documents: always fallback, "I don't have data to answer".

Exercise 5: Prioritize observability

Of logs, metrics, and traces, which would you implement first and why?

Solution:

Structured logs (JSON): request_id, component, message. Minimal effort, maximum value for debugging.
trace_id in the response: lets you correlate the backend log with what the user saw.
Latency metrics: p50, p95 per endpoint; typical bottleneck in /ask.
Distributed traces: optional for v2; with a single service, a trace_id in logs may be enough.

Exercise 6: Scaling ingestion

If you had 100,000 documents, how would you adapt the pipeline?

Solution:

Larger batch (2000-5000) to reduce ChromaDB overhead.
Parallelization: multiple workers for embedding (respecting OpenAI rate limits).
Queue (Redis, SQS) for asynchronous ingestion.
Checkpointing: save progress to resume if it fails.
Consider ChromaDB in server mode or migrating to Pinecone for that scale.

Exercise 7: Sequence diagram for `/ask`

Draw a sequence diagram (User → API → Retrieval → ChromaDB; API → Generation → OpenAI) showing the order of calls for a request to /ask. Include what happens when retrieval returns 0 results.

Solution (text):

User sends POST /ask with {"question": "..."}.
API validates input, generates a trace_id.
API calls Retrieval with the query.
Retrieval embeds the query (OpenAI).
Retrieval calls ChromaDB.query with the embedding.
ChromaDB returns results (or empty).
If empty: Retrieval returns []; API returns fallback without calling Generation.
If there are results: Retrieval returns docs; API calls Generation with the docs.
Generation builds the prompt, calls OpenAI, receives the answer.
API formats the response with sources, trace_id, and returns it to the user.

Exercise 8: Abstract interface for the Vector Store

Define an interface (ABC or Protocol in Python) that encapsulates the vector store operations you need: add(ids, documents, embeddings, metadatas) and query(embedding, top_k, where). What would change if tomorrow you use Pinecone?

Solution:

from abc import ABC, abstractmethod
from typing import List, Optional

class VectorStoreProtocol(ABC):
    @abstractmethod
    def add(self, ids: List[str], documents: List[str], embeddings: List[List[float]], metadatas: List[dict]) -> None: ...

    @abstractmethod
    def query(self, embedding: List[float], top_k: int = 5, where: Optional[dict] = None) -> dict: ...

You implement ChromaDBStore(VectorStoreProtocol) for ChromaDB. For Pinecone, you implement PineconeStore(VectorStoreProtocol). Retrieval only depends on the interface, not the concrete backend.

Architecture troubleshooting

"Everything works, but we don't know where it fails"

Cause: Lack of separation of responsibilities and instrumentation per stage.

Solution: Clearly separate ingestion, retrieval, and generation. Add logs at each boundary (e.g. "retrieve returned 5 docs", "generate called with 3 chunks"). Use a trace_id to follow a request through the whole pipeline.

"Every change breaks another part"

Cause: Implicit contracts, high coupling.

Solution: Define explicit contracts (schemas, types). Write integration tests per flow (ingestion → retrieval → generate). If you change a contract, update tests and documentation.

"We don't know how to scale ingestion"

Cause: Ingestion coupled to serving, without batches or parallelism.

Solution: Decouple: ingestion as an independent script/job. Use batches of 1K-2K, measure throughput. Consider parallel workers for embeddings (with rate limiting).

"ChromaDB fills up or is slow"

Cause: Too many vectors, heavy metadata, or without adequate indexes.

Solution: Review chunk_size (larger = fewer vectors). Reduce metadata to the essentials. ChromaDB uses HNSW by default; for millions of vectors, consider migrating to Pinecone/Weaviate.

"Made-up answers when there is no context"

Cause: Generation doesn't validate retrieval quality, keeps generating without evidence.

Solution: Score threshold in retrieval (e.g. discard < 0.5). In generation, if there are no valid chunks, return an explicit message and don't call the LLM to make things up.

Summary

The architecture has 5 components: ingestion, vector_store, retrieval, generation, api.
The data flow is linear: documents → ingestion → ChromaDB → retrieval → generation → API response.
Key decisions: ChromaDB (free, learning), chunk_size=512 (balance), chunk_overlap=50 (continuity), GPT-3.5-turbo (cost/latency).
Explicit contracts between components make testing and evolution easier.
Avoid anti-patterns: monolithic functions, hardcoded config, non-deterministic IDs, answers without a fallback.
Document the design to ease migration to Guide #8 and onboarding of other developers.
Consider testing per component: each layer is tested in isolation before the end-to-end test.
Use per-environment configuration (dev vs prod) and a Decision Log for traceability.

Checklist before implementing

Before moving to capsule 03 (ingestion), verify:

You have the architecture diagram (paper or digital).
You defined the metadata schema (doc_id, source, chunk_index, title).
You know what to do when retrieval returns 0 or low scores.
You have externalized config (environment variables).
You documented at least 2-3 decisions in a DESIGN.md or README.

If you meet the 5 points, you're ready to implement the ingestion pipeline.

Dependency diagram between Python modules

api/
  main.py          → routes (ask, search, ingest, collections)
  routes/
    ask.py         → services.retrieval, services.generation
    search.py      → services.retrieval
    ingest.py      → ingestion.pipeline
  services/
    retrieval.py   → chromadb, openai
    generation.py  → openai

ingestion/
  pipeline.py      → loaders, chunking, embeddings, chromadb
  loaders.py       → (only pathlib, pypdf if PDF)
  chunking.py      → langchain_text_splitters
  embeddings.py    → openai

Rule: The services (retrieval, generation) don't import from ingestion. The API orchestrates both. This way ingestion and serving can evolve independently.

Example of a `/ask` request flow (pseudocode)

request = {"question": "What is X?"}
trace_id = uuid4()

# 1. Validate
validate_length(question, max=2000)

# 2. Retrieval
query_embedding = openai.embeddings.create(input=[question])
results = chroma.query(query_embedding[0], n_results=5)

# 3. Filter by score
filtered = [r for r in results if r.score >= 0.5]
if not filtered:
    return fallback_response(trace_id)

# 4. Generation
context = build_context(filtered.documents, filtered.metadatas)
prompt = f"Context:\n{context}\n\nQuestion: {question}"
answer = openai.chat.completions.create(messages=[...], prompt=prompt)

# 5. Response
return {
  "answer": answer,
  "sources": format_sources(filtered),
  "confidence": avg(filtered.scores),
  "trace_id": trace_id,
}

This pseudocode summarizes what you'll implement in capsule 04. The separation into clear steps makes debugging and tests easier.

Next steps

In capsule 03 you'll implement the ingestion pipeline: loaders for TXT/MD/RST, chunking with RecursiveCharacterTextSplitter, embeddings with OpenAI, and storage in ChromaDB. Use the architecture you defined here as a guide; if some component doesn't fit, adjust the design before over-coding. Coherence between design and implementation avoids costly refactors.

Additional resources

Martin Fowler - Microservices — Decoupled design principles
ChromaDB Collections — Data structure
OpenAI Embeddings — Models and limits
RAG Best Practices — RAG patterns
FastAPI Project Structure — Modular organization
C4 Model — Architecture diagrams by levels
Twelve-Factor App — Configuration, logs, deployment
Vector DB Comparison (this guide)

Estimated time: 25-30 minutes
Next: 03-pipeline-ingestion-1000-docs.md

← PreviousModule 8: Capstone RAG Project with ChromaDB Next →Capsule 03: Ingestion Pipeline for 1,000+ documents

Capsule 02: Project Architecture

Capsule description

Main components

Detailed responsibilities per component

Ingestion

Vector Store (ChromaDB)

Retrieval

Generation

API

Architecture diagram (ASCII)

Detailed data flow

Ingestion flow (offline or on demand)

Query flow (online)

Design decisions and justification

Why ChromaDB?

Why chunk_size=512 (approximate tokens)?

Why chunk_overlap=50?

Why GPT-3.5-turbo?

Why separate ingestion from serving?

Contracts between components

Ingestion → Vector Store

Retrieval → Generation

API → Client

Anti-patterns to avoid

Failure points and mitigation

Architecture alternatives

Testing considerations per component

Configuration per environment

Decision Log template

Project environment variables

Design exercises

Exercise 1: Draw your architecture

Exercise 2: Define the metadata schema

Exercise 3: Design the ID strategy

Exercise 4: Fallback when retrieval is weak

Exercise 5: Prioritize observability

Exercise 6: Scaling ingestion

Exercise 7: Sequence diagram for /ask

Exercise 8: Abstract interface for the Vector Store

Architecture troubleshooting

"Everything works, but we don't know where it fails"

"Every change breaks another part"

"We don't know how to scale ingestion"

"ChromaDB fills up or is slow"

"Made-up answers when there is no context"

Summary

Checklist before implementing

Dependency diagram between Python modules

Example of a /ask request flow (pseudocode)

Next steps

Additional resources

Exercise 7: Sequence diagram for `/ask`

Example of a `/ask` request flow (pseudocode)