Module 8: The Capstone Final Project
5. Stage 4: Cost Estimation
Overview
In this stage you calculate the system's complete monthly cost: the initial setup, infrastructure, queries, and maintenance. The budget is $2000/month.
The initial setup cost (one time)
1. Hardware (if buying)
GPU server (for the embeddings):
- NVIDIA T4 (16GB)
- 32GB RAM
- 500GB SSD
Cost: $3000 (to buy) or $0.35/hour in the cloud ($250/month)
Weaviate server:
- 16 vCPUs
- 64GB RAM
- 500GB SSD
Cost: $2000 (to buy) or $0.34/hour in the cloud ($245/month)
→ The recommended option: Cloud (no CAPEX, more flexible)
2. The initial indexing
Embedding 500K chunks (local Sentence-BERT):
- Time: 8 hours
- GPU cost: $0.35/hour × 8 = $2.80
Weaviate indexing:
- Time: 2 hours
- Server cost: $0.34/hour × 2 = $0.68
Total setup: $3.48 (negligible)
The recurring monthly cost
Category 1: Infrastructure
GPU server (embeddings):
$0.35/hour × 730 hours/month = $255/month
Weaviate server:
$0.34/hour × 730 hours/month = $248/month
API server (FastAPI on AWS Lambda):
50K requests/month × $0.0000002 = $0.01/month (negligible)
+ $5/month (API Gateway)
Infrastructure subtotal: $508/month
Category 2: The LLM (GPT-3.5 Turbo)
Expected queries:
Total queries: 50K/month
Of which:
- 40% are simple searches (no LLM): 20K
- 60% are Q&A (with an LLM): 30K
LLM cost:
Q&A queries: 30K/month
Per query:
- Input: 2500 tokens (the query + 5 chunks × 500 tokens)
- Output: 300 tokens (the answer)
- Input cost: 2500 × $0.0005/1K = $0.00125
- Output cost: 300 × $0.0015/1K = $0.00045
- Total cost/query: $0.0017
Monthly LLM cost:
30K × $0.0017 = $51/month
Category 3: Storage
The vector DB (Weaviate):
- 500K vectors × 768D × 4 bytes = 1.5GB
- The HNSW graph: ~2GB
- Metadata + text: ~3GB
Total: ~6.5GB
AWS EBS (SSD): $0.10/GB/month
→ 6.5GB × $0.10 = $0.65/month (negligible)
Category 4: Monitoring and logs
CloudWatch (AWS):
- Logs: ~10GB/month
- Metrics: Standard
Cost: ~$10/month
Category 5: Incremental re-indexing
New documents: +5K/year = ~417/month
Re-indexing:
- Embedding: 417 docs × 5 chunks = 2085 chunks/month
- GPU time: ~3 minutes/month
- Cost: $0.35/hour × (3/60) = $0.018/month (negligible)
A summary of the monthly costs
| Category | Cost/month |
|---|---|
| GPU server (embeddings) | $255 |
| Weaviate server | $248 |
| API server (Lambda + API Gateway) | $5 |
| LLM (GPT-3.5 Turbo) | $51 |
| Storage (EBS) | $1 |
| Monitoring (CloudWatch) | $10 |
| Re-indexing | <$1 |
| Total | $570/month |
Analysis vs the budget
Budget: $2000/month
Actual cost: $570/month
Headroom: $1430/month (71% under budget) ✅
Cost optimizations
If the budget were tight ($500/month):
Option 1: Drop the dedicated GPU (use CPU)
Sentence-BERT on CPU:
- Latency: 200ms/batch (vs 50ms on GPU)
- Impact on a query: +150ms
- The new total latency: 380ms (search), 1590ms (Q&A)
→ Still < the 2s target ✅
Savings: $255/month
The new total: $315/month
Option 2: Weaviate on a smaller server
c5.xlarge (4 vCPUs, 8GB RAM):
- Cost: $0.17/hour = $124/month (vs the current $248)
- Trade-off: kNN latency +20ms (80ms → 100ms)
Savings: $124/month
The new total: $446/month
Option 3: Use a local Llama 3 (dropping GPT-3.5)
Llama 3 8B (self-hosted):
- It requires an additional GPU (NVIDIA A10): $0.60/hour = $438/month
- But: it eliminates the GPT-3.5 cost ($51/month)
Net: +$387/month (more expensive) ❌
→ NOT recommended for this case
Growth projection
Year 1:
Queries: 50K/month (constant)
New docs: +5K/year
Cost: $570/month × 12 = $6,840/year
Year 2:
Queries: 75K/month (+50% users)
LLM cost: 45K × $0.0017 = $76.5/month (+$25.5)
The new total: $595.5/month × 12 = $7,146/year
Year 5:
Docs: 100K + (5K × 5) = 125K docs (+25%)
Vectors: 625K (vs the initial 500K)
Weaviate: It needs an upgrade → c5.4xlarge ($0.68/hour = $496/month)
Queries: 100K/month (2x the initial)
LLM cost: 60K × $0.0017 = $102/month
The new total: $255 + $496 + $5 + $102 + $1 + $10 = $869/month
→ Even in year 5, the cost ($869/month) < the budget ($2000/month) ✅
Sensitivity analysis
What happens if the queries increase 10x?
Queries: 500K/month (10x)
Q&A queries: 300K/month
LLM cost: 300K × $0.0017 = $510/month (vs the current $51)
The new total: $255 + $248 + $5 + $510 + $1 + $10 = $1,029/month
→ Still < the $2000 budget ✅
But: The latency can degrade (a saturated GPU)
→ The solution: Add a second GPU server (+$255/month)
→ Total: $1,284/month (still OK)
What happens if we use GPT-4 instead of GPT-3.5?
GPT-4 cost/query: $0.022 (vs GPT-3.5's $0.0017)
→ 13x more expensive
LLM cost: 30K × $0.022 = $660/month (vs the current $51)
The new total: $255 + $248 + $5 + $660 + $1 + $10 = $1,179/month
→ Still < the $2000 budget ✅
The trade-off: +$609/month for better quality (it may be worth it)
Cost recommendations
-
Monitor the monthly usage:
Alerts if the cost > $700/month (50% under budget) -
Cache frequent queries:
The top-100 queries (30% of the traffic) → savings of $15/month on the LLM -
Tune top-K if possible:
If precision@10 > 0.80 with top-K=50 (instead of 100) → latency -20ms -
Evaluate GPT-4 in production:
An A/B test: 10% of traffic on GPT-4, 90% on GPT-3.5
If quality improves significantly → increase the GPT-4 share
Cost summary
Base monthly cost: $570/month (71% under budget)
The breakdown:
- Infrastructure (GPU + Weaviate): $503/month (88%)
- LLM (GPT-3.5): $51/month (9%)
- Other (API, storage, monitoring): $16/month (3%)
Headroom: $1430/month available for:
- Upgrades (GPT-4, more GPUs, larger servers)
- Growth (10x the queries still fits in the budget)
- Contingency
Next stage: 06-the-final-document.md — Consolidating everything into a technical design document.