Module 1: Decision Framework for LLM Access

Mini-Project: Requirements Assessment - E-commerce Chatbot

Project overview

This is the integrative project for Module 1. You'll apply the entire decision framework you learned (5 dimensions, options landscape, decision matrix, trade-offs) to a complete real case.

You won't write code in this project. The goal is for you to make an informed decision about which provider to use BEFORE implementing. This is the most valuable skill: deciding well the first time.

The project simulates a real situation: you are the tech lead of a startup and you must choose an LLM provider for the support chatbot. You have specific requirements, stakeholders with different opinions, and a limited budget.

By the end of this project:

✅ You'll have applied the complete framework (not just theory)
✅ You'll have a documented scorecard with a justified decision
✅ You'll know how to defend your choice with data
✅ You'll have a fallback plan for contingencies

🎯 Project goal

Goal: Evaluate the requirements of an e-commerce project and choose the optimal LLM provider using the Module 1 decision framework.

Deliverables:

Evaluation scorecard (5 dimensions completed)
Justified final decision (provider + why)
Accepted trade-offs (what you gain, what you lose)
Fallback plan (what you do if it fails)

📋 Project Context

Company: ShopFast (Mid-size e-commerce)

Description:

Electronics e-commerce in LATAM
50,000 registered users
5,000 transactions/month
Team: 15 people (3 backend devs, 2 frontend, 1 junior DevOps)
Founded 2 years ago, Series A ($2M funding)

Current situation:

Support by email: ~300 tickets/day
Response time: 12-24 hours
Customer satisfaction: 6.5/10
Support cost: $15k/month (2 full-time agents)

Problem:

Users want instant answers
Competitors have 24/7 chatbots
Losing conversions due to slow support

🎯 Chatbot Goal

Expected functionality:

Frequently asked questions (80% of tickets)
- "What's the status of my order?"
- "Do you have the iPhone 14 in stock?"
- "What are the payment methods?"
- "Do you ship to [city]?"
Product recommendations (15% of tickets)
- "I need a laptop for design, which do you recommend?"
- "What's the difference between model X and Y?"
Escalation to a human (5% of complex tickets)
- "I want to return a defective product"
- "My order didn't arrive after 15 days"

Success metrics:

Resolve 70% of queries without a human
Reduce response time to <30 seconds
Increase customer satisfaction to 8/10
Reduce support cost to $10k/month (recover 1 agent)

📊 Technical Requirements

Expected volume:

Active users: 5,000/day
Chatbot queries: ~15,000/day (3 queries/user on average)
Peak: Black Friday (10x normal traffic = 150,000 queries/day)
Growth: 20% monthly expected

Necessary features:

Answers in Spanish (LATAM)
Integration with the inventory system (REST API)
Integration with the orders system (tracking)
Conversational context (remember previous messages)
Automatic escalation to a human agent if it can't resolve

Constraints:

Latency: <2 seconds (users won't wait longer)
Accuracy: 85%+ (support not critical, but must be useful)
Availability: 99.5% uptime (24/7 except maintenance)

💰 Budget and Resources

Available budget:

Initial setup: $5,000 (one-time)
Monthly operational: $1,500/month
First-year total: $23,000 ($5k setup + $1.5k × 12 months)

Budget justification:

Current support cost: $15k/month
Target cost with chatbot: $10k/month (savings $5k/month)
Chatbot budget: $1.5k/month
ROI: Net savings $3.5k/month = $42k/year

Available team:

Lead backend dev: María (5 years exp, Python/FastAPI expert)
Junior backend dev: Carlos (1 year exp, basic Python)
Junior DevOps: Ana (2 years exp, basic Docker/AWS)
Availability: 2 weeks full-time for setup, then 20% time maintenance

Team skills:

Python: ✅ Strong (María), ⚠️ Basic (Carlos)
FastAPI: ✅ Strong (María)
Docker: ⚠️ Basic (Ana)
AWS: ⚠️ Basic (Ana)
Advanced DevOps: ❌ They don't have it
ML/AI: ❌ Zero experience

🔐 Data and Privacy

Data type:

User queries: NOT sensitive (questions about products)
Order data: Partially sensitive (order number, status)
- Does NOT include: Cards, full addresses, personal data
- Only: Order ID, status, estimated delivery date
Inventory: NOT sensitive (public on the web)

Compliance:

GDPR: Does NOT apply (LATAM users, not EU)
LGPD: Applies (Brazil), but the processed data is public
Contracts: No special privacy clauses with clients

Internal policy:

Preference for on-premise data (not mandatory)
Legal approved the use of US cloud providers (OpenAI, Google, etc.)
Constraint: Don't send PII (emails, phone numbers, addresses)

Conclusion: Privacy is IMPORTANT but NOT critical (data is mostly public).

⚡ Stakeholder Expectations

CEO (Pedro):

"I need this FAST, we have to launch before the end of the quarter"
Timeline: 4-6 weeks maximum
Priority: Simplicity and speed of implementation

CFO (Laura):

"A budget of $1.5k/month is a hard limit, we can't go over"
ROI must be positive in the first year
Priority: Low and predictable cost

CTO (Roberto, your boss):

"I prefer a solution the team can maintain without hiring"
Concern: Operational complexity
Priority: Maintainability and team-skills match

Head of Support (Claudia):

"It must answer well; if it gives bad answers we lose clients"
Minimum accuracy: 85%
Priority: Answer quality

Your role: Balance the expectations and choose the option that satisfies the majority.

📝 Your Task: Complete Evaluation

Step 1: 5-Dimension Scorecard

Complete this scorecard:

## Scorecard: ShopFast E-commerce Chatbot

### 1. Cost
**Available budget:**
- Setup: $______
- Monthly: $______/month
- Hard limit: YES / NO

**Expected volume:**
- Queries/day: ______
- Queries/month: ______
- Peaks: ______ (Black Friday)

**Estimated cost calculation per provider:**

OpenAI API (GPT-3.5):
- 15k queries × 500 tokens × $0.002/1k × 30 days = $____/month
- Black Friday (10x): $____/month (1 day)
- Within budget?: YES / NO

Local Ollama:
- Operational cost: $0
- Hardware upfront: $______ (within the $5k setup?)
- Maintenance (DevOps time): $____/month
- Within budget?: YES / NO

[Repeat for OpenRouter, Modal...]

**Priority:** CRITICAL / IMPORTANT / SECONDARY

**Justification:**
[Write why it's critical/important/secondary for this project]

---

### 2. Quality
**Accuracy needed:** _____%
**Do I need complex reasoning?:** YES / NO
**Context window needed:** ____ tokens
**Language:** Spanish (LATAM)

**Evaluation per model:**

GPT-3.5 (OpenAI):
- Accuracy: ~90% (public benchmark)
- Spanish: ✅ Excellent
- Context: 16k tokens
- Meets the 85% requirement?: YES / NO

Mistral 7B (local Ollama):
- Accuracy: ~80-85%
- Spanish: ✅ Good (trained with Spanish)
- Context: 8k tokens
- Meets the 85% requirement?: YES / NO (borderline)

[Repeat for other models...]

**Priority:** CRITICAL / IMPORTANT / SECONDARY

**Justification:**
[Write why]

---

### 3. Privacy
**Data type:** SENSITIVE / NORMAL / PUBLIC
**Mandatory compliance:** GDPR / HIPAA / LGPD / NONE
**Can it go to the cloud?:** YES / NO
**Constraints:**
- [List specific constraints]

**Evaluation:**

OpenAI API:
- Data retention: 30 days
- Training: NO (opt-out by default)
- Meets requirements?: YES / NO

Local Ollama:
- Data retention: NEVER (local)
- Training: NEVER
- Is it overkill for this project?: YES / NO

**Priority:** CRITICAL / IMPORTANT / SECONDARY

**Justification:**
[Write why]

---

### 4. Speed
**Acceptable latency:** ____ seconds
**Simultaneous requests:** ______
**Do I need streaming?:** YES / NO

**Evaluation:**

OpenAI API:
- Latency: ~1.5s (GPT-3.5)
- Meets <2s?: YES / NO
- Throughput: High (rate limits 3500 req/min)

Local Ollama:
- Latency: ~5-10s (depends on hardware)
- Meets <2s?: YES / NO
- Throughput: Low (needs multiple instances)

**Priority:** CRITICAL / IMPORTANT / SECONDARY

**Justification:**
[Write why]

---

### 5. Simplicity
**Time available:** ____ weeks
**Team skills:** JUNIOR / MID / SENIOR
**Who maintains it?:** NOBODY / SMALL TEAM / DEDICATED

**Evaluation:**

OpenAI API:
- Setup time: <1 week (María can do it)
- Maintenance: Low (managed)
- Can the team maintain it?: YES / NO

Local Ollama:
- Setup time: 2-3 weeks (needs advanced DevOps)
- Maintenance: High (monitoring, updates)
- Can the team maintain it?: YES / NO (Ana is junior)

**Priority:** CRITICAL / IMPORTANT / SECONDARY

**Justification:**
[Write why]

Step 2: Identify the Critical Dimensions

Of the 5 dimensions, choose the 2-3 that are NON-NEGOTIABLE:

## CRITICAL dimensions (2-3 max):

1. _____________ is CRITICAL because _______________________
2. _____________ is CRITICAL because _______________________
3. _____________ is CRITICAL because _______________________

## IMPORTANT dimensions (the rest):

- _____________ is important but negotiable
- _____________ is important but negotiable

## SECONDARY dimensions:

- _____________ is nice-to-have

Step 3: Apply the Decision Matrix

Look up in the decision matrix (capsule 04) the combination that matches:

## Matrix Lookup

**My CRITICAL dimensions:** _______ + _______

**Match in the matrix:**
- Case X: _______ + _______ → Provider: _______

**Is there an exact match?:** YES / NO

**If there's no exact match, candidates:**
1. Provider: _______ (meets dimension 1 + 2)
2. Provider: _______ (meets dimension 1 + 3)
3. Provider: _______ (meets dimension 2 + 3)

Step 4: Evaluate Trade-offs

For your recommended option, list the trade-offs:

## Recommended Option: _____________

### ✅ What you GAIN:
- Dimension 1: _______________________
- Dimension 2: _______________________
- Dimension 3: _______________________

### ⚠️ What you LOSE (trade-offs):
- Dimension A: _______________________ (acceptable because _______)
- Dimension B: _______________________ (acceptable because _______)

### 🤔 Are the trade-offs acceptable?
[Write the analysis: why the trade-offs are OK given the context]

Step 5: Justified Final Decision

## FINAL DECISION

**Chosen provider:** _____________

**Justification in 3 paragraphs:**

[Paragraph 1: Why this option meets the critical dimensions]

[Paragraph 2: How the trade-offs are acceptable given the context]

[Paragraph 3: Why other options do NOT work]

**Stakeholders:**
- CEO (speed): ✅ / ❌ [How it meets/doesn't meet their expectation]
- CFO (cost): ✅ / ❌ [Within/over budget]
- CTO (maintainability): ✅ / ❌ [The team can/can't maintain it]
- Head Support (quality): ✅ / ❌ [Accuracy sufficient/insufficient]

**Balance:** __/4 stakeholders satisfied

Step 6: Fallback Plan

## Fallback Plan

**Primary option:** _____________

**Triggers to activate the fallback:**
1. If _______________ (e.g.: OpenAI raises the price 50%)
2. If _______________ (e.g.: Latency > 5s for 10 min)
3. If _______________ (e.g.: Budget is reduced to $500/month)

**Fallback option:** _____________

**How you'd migrate:**
- Estimated time: ____ weeks
- Migration cost: $______
- Reusable code: ____%
- Risk: LOW / MEDIUM / HIGH

✅ Self-Evaluation Rubric

Evaluate your work:

1. Complete scorecard (30 points)

(10 pts) All 5 dimensions have quantitative values (not "high/low", but numbers)
(10 pts) I calculated the estimated cost for AT LEAST 2 providers
(10 pts) I justified why each dimension is critical/important/secondary

2. Justified decision (40 points)

(15 pts) I identified 2-3 critical dimensions with a clear reason
(15 pts) The final decision has 3 paragraphs of justification (not just "I chose X")
(10 pts) I evaluated the satisfaction of 4 stakeholders (CEO, CFO, CTO, Support)

3. Explicit trade-offs (20 points)

(10 pts) I listed WHAT I gain with my choice
(10 pts) I listed WHAT I lose (trade-offs) and WHY they're acceptable

4. Fallback plan (10 points)

(5 pts) I have a clear option B
(5 pts) I have specific triggers (not "if something goes wrong")

Total: ___/100 points

Interpretation:

90-100: ✅ Excellent, you fully understood the framework
70-89: ⚠️ Good, but the justification lacks depth
50-69: ⚠️ Acceptable, revisit capsules 02-04
<50: ❌ Insufficient, re-read the whole module

📊 Reference Solution

See the recommended solution (only after completing your evaluation)

Reference Solution: ShopFast Chatbot

Completed Scorecard:

1. Cost:

Budget: $5k setup + $1.5k/month (HARD LIMIT)
Volume: 15k queries/day × 30 = 450k queries/month
Average tokens: 500 tokens/query

Calculations:

OpenAI GPT-3.5:

450k queries × 500 tokens = 225M tokens/month
225M tokens × $0.002/1k = $450/month
Black Friday (10x = 150k queries): $150 that day
✅ WITHIN BUDGET ($450 << $1500)

Local Ollama Mistral 7B:

Operational: $0
Hardware: $2000 (GPU) ✅ Within the $5k setup
Maintenance (Ana 20% time): ~$800/month
✅ WITHIN BUDGET ($800 < $1500)

Priority: IMPORTANT (the CFO demands it, but there's margin)

2. Quality:

Accuracy needed: 85%+
Spanish: Mandatory

GPT-3.5: ~90% accuracy, excellent Spanish ✅ Mistral 7B: ~80-85% accuracy, good Spanish ⚠️ (borderline)

Priority: IMPORTANT (Head of Support demands 85%+)

3. Privacy:

Data: Mostly public (FAQs, products)
Order IDs: Not sensitive (only status, no PII)
Legal: Approved US cloud

Priority: SECONDARY (on-premise nice-to-have, NOT mandatory)

4. Speed:

Latency: <2s (e-commerce users won't wait)
Simultaneous queries: ~200 peak hour

OpenAI: ~1.5s ✅ Local Ollama: ~5-10s ❌ (with modest hardware)

Priority: CRITICAL (e-commerce UX requires instant answers)

5. Simplicity:

Timeline: 4-6 weeks (CEO deadline)
Team: María (senior), Carlos (junior), Ana (junior DevOps)

OpenAI: Setup 1 week ✅ Ollama: Setup 2-3 weeks ⚠️ (Ana is junior, needs help)

Priority: CRITICAL (tight deadline, team with a skills gap)

CRITICAL dimensions:

Speed (<2s latency, non-negotiable for e-commerce UX)
Simplicity (6-week deadline, team junior in AI)

IMPORTANT dimensions:

Cost ($1.5k/month limit, but $450-800 is OK)
Quality (85%+ needed)

SECONDARY dimensions:

Privacy (nice-to-have, NOT critical)

Matrix Lookup:

Critical dimensions: Speed + Simplicity

Match: Case 2 (Quality + Speed) → OpenAI API

(There's no exact match for "Speed + Simplicity", but Case 2 is the closest)

Final Decision: OpenAI API (GPT-3.5-turbo)

Justification:

Paragraph 1 - Meets the critical dimensions:

The OpenAI API meets the 2 critical dimensions without compromise. Speed: ~1.5s latency (within the <2s requirement) with global infrastructure that guarantees low latency in LATAM. Simplicity: María can implement it in 1 week (OpenAI SDK setup + FastAPI wrapper + inventory integration), meeting the 6-week deadline with margin. The junior team (Carlos, Ana) can maintain it because it's a managed service (zero DevOps, zero GPU tuning, zero advanced Linux).

Paragraph 2 - Acceptable trade-offs:

The trade-offs are minimal and acceptable. Cost: $450/month is well below the $1.5k/month limit (70% margin). Quality: GPT-3.5 has ~90% accuracy, exceeding the 85% minimum required by Head of Support. Privacy: Data goes to US cloud, but Legal explicitly approved it and the processed data is mostly public (FAQs, inventory, order status without PII). Black Friday (10x traffic = 150k queries) costs ~$150 that day, negligible against the current support cost ($15k/month).

Paragraph 3 - Why other options do NOT work:

Local Ollama was considered but rejected for 3 reasons: (1) Latency of 5-10s with modest hardware does NOT meet the critical <2s requirement, (2) Setup of 2-3 weeks eats half the timeline and Ana (junior DevOps) would need mentoring we don't have, (3) Mistral 7B has 80-85% accuracy (borderline of the minimum). OpenRouter was considered but OpenAI directly is simpler (one less API, 100ms lower latency). Modal was discarded because it adds unnecessary complexity (cold starts) when OpenAI managed already scales automatically.

Stakeholder Satisfaction:

✅ CEO (Pedro): Meets the deadline (1 week setup vs 6 available)
✅ CFO (Laura): $450/month << $1.5k budget ($1050/month of margin)
✅ CTO (Roberto): The team can maintain it without hiring
✅ Head Support (Claudia): 90% accuracy > 85% minimum

Balance: 4/4 stakeholders satisfied ✅

Fallback Plan:

Primary option: OpenAI API (GPT-3.5)

Triggers:

If OpenAI raises the price 3x: ($450/month → $1350/month approaches the limit)
- Action: Migrate to OpenRouter with economical models ($0.001/1k = $225/month)
- Timeline: 1 week (compatible API, drop-in replacement)
If latency degrades >3s for 24hrs:
- Immediate action: Switch to OpenRouter with GPT-3.5 (same model, different infrastructure)
- Long-term action: Evaluate Modal serverless (warm instances ~1s)
If privacy requirements change (a new enterprise client with an on-premise clause):
- Action: Migrate to local Ollama (plan B already evaluated)
- Timeline: 4-6 weeks
- Cost: $2k hardware + hire a temp senior DevOps ($5k/month for 2 months)

Preparation:

Abstract the OpenAI calls behind an LLMProvider interface (makes swapping easier)
Monitor latency and cost daily (automatic alerts)
Have a pre-configured OpenRouter account (activate in <1 hour if needed)

📝 Final Reflection

After completing your evaluation and comparing it with the reference solution:

Reflection questions:

Was your final decision different?
- If YES: Why did you prioritize different dimensions?
- If NO: Did you reach the same conclusion by the same reasoning?
Did you identify the same critical dimensions?
- If NO: Why did you consider others more important?
Did your justification include specific numbers?
- Good example: "$450/month is 70% below the $1.5k limit"
- Bad example: "The cost is acceptable"
Did you evaluate stakeholder satisfaction?
- Many "technically correct" decisions fail because they ignore stakeholders
Do you have a viable fallback plan?
- Or did you assume your first option will always work?

🎯 Success Criteria

You've successfully completed this project when:

✅ You applied the framework systematically:

You didn't choose by "feeling" or popularity
You used a structured scorecard
You identified the critical dimensions with a reason

✅ The decision is justified with data:

Specific numbers ($/month, latency ms, accuracy %)
Not just "OpenAI is good"
You compared AT LEAST 2 options

✅ The trade-offs are explicit:

You know WHAT you gain and WHAT you lose
You justify WHY the trade-offs are acceptable

✅ A fallback plan exists:

You have a clear option B
Specific triggers (not vague)
You know how to migrate if necessary

If you meet all 4 criteria: ✅ You PASSED Module 1

You're ready for modules 2-8 (technical implementation with specific providers).

🔗 Additional resources

Decision Framework Template (Google Doc) - Reusable template for future projects
ShopFast Case Study (Full) - Extended analysis with more scenarios
LLM Decision Calculator - Interactive tool
ROI Calculator for AI Projects - Calculate return on investment

➡️ Next step

Congratulations! You completed Module 1: Decision Framework

You now know how to:

✅ Evaluate projects against 5 dimensions
✅ Apply the decision matrix
✅ Justify decisions with data
✅ Identify trade-offs
✅ Create fallback plans

Next module: Module 2: OpenAI API

In Module 2, you'll learn to IMPLEMENT your decision. If you chose the OpenAI API in your project, you'll start with setup, API keys, first requests, and you'll build the working chatbot.

If you chose another provider (Ollama, Modal, etc.), you can jump straight to that module. Or continue sequentially to learn all the options.

Estimated time: 60-90 minutes
Next: Module 2: OpenAI API

← PreviousCommon Mistakes When Choosing an LLM Provider Next →Module 2: OpenAI API - Introduction