Module 6: The AI API Ecosystem

7. Exercise: Choosing the Appropriate Provider and Stack

Next →

Description

Goal: Apply everything you learned in the module by choosing the appropriate stack (provider, model, local/cloud) based on requirements.

Format: 6 real use cases → you decide which stack to use and justify your decision.

Case 1: A Startup MVP (a Support Chatbot)

Context

  • Company: A pre-seed startup ($50K raised).
  • Product: A support chatbot for ecommerce.
  • Expected volume: 1K requests/month (at the start).
  • AI budget: $100/month.
  • Requirements: Launch fast (2 weeks), fluent Spanish, quality responses.

Your Decision

Which provider/model would you use?

  • OpenAI GPT-4
  • OpenAI GPT-3.5-turbo
  • Anthropic Claude 3 Haiku
  • Self-hosted Llama 3 (70B)
  • OpenRouter (multi-model)

Local or Cloud?

  • Local (Ollama)
  • Cloud (API)

Justification: (Think about your answer before looking at the solution)

Solution

Recommended stack:

  • Provider/Model: Claude 3 Haiku or GPT-3.5-turbo.
  • Local/Cloud: Cloud (API).

Why:

  1. Low volume (1K requests/month): A cloud API is cheaper than self-hosting ($5-20/month vs $500/month for a GPU).
  2. A limited budget ($100/month): GPT-4 ($30-90/month with 1K requests) is expensive → GPT-3.5 or Haiku ($2-10/month) is appropriate.
  3. Launching fast: An API integrates in 1 day (vs self-hosting: 1-2 weeks).
  4. Fluent Spanish: GPT-3.5 and Claude Haiku handle Spanish well (GPT-4 isn't required).

Accepted trade-off: Quality slightly below GPT-4, but 20-30× lower cost.

Case 2: A Company with Sensitive Data (Healthcare)

Context

  • Company: A health startup (HIPAA compliance).
  • Product: An AI assistant for doctors (it summarizes medical notes).
  • Expected volume: 50K requests/month.
  • AI budget: $2K/month.
  • Requirements: HIPAA compliance (data can NOT leave your own infrastructure).

Your Decision

Which provider/model would you use?

  • OpenAI GPT-4
  • Anthropic Claude 3 Opus
  • Self-hosted Llama 3 (70B)
  • OpenRouter (multi-model)

Local or Cloud?

  • Local/Self-hosted
  • Cloud (API)

Justification: (Think about your answer before looking at the solution)

Solution

Recommended stack:

  • Model: Self-hosted Llama 3 (70B).
  • Local/Cloud: Self-hosted (on your own servers or in a private cloud with encryption).

Why:

  1. HIPAA compliance: Patient data can NOT be sent to external APIs (OpenAI, Anthropic) → this rules out cloud APIs.
  2. Volume (50K requests/month): Self-hosted is cheaper ($1.5K-2K/month for a fixed GPU vs $5K-10K/month with APIs).
  3. Total control: Self-hosting allows complete security audits (required by HIPAA).

Setup:

  • A private cloud (AWS, GCP with encryption, a private VPC).
  • Llama 3 (70B) on A100/H100 GPUs.
  • vLLM or TGI (Text Generation Inference) for serving.

Accepted trade-off: It requires a DevOps team to maintain the infrastructure.

Case 3: A High-Volume Application (Millions of Requests)

Context

  • Company: A Series B startup ($20M raised).
  • Product: An AI-assisted writing platform (autocomplete, rewriting).
  • Volume: 10M requests/month (growing 20% monthly).
  • AI budget: $50K/month.
  • Requirements: Low latency (<500ms), Spanish and English.

Your Decision

Which provider/model would you use?

  • OpenAI GPT-4 (for everything)
  • GPT-4 (complex) + GPT-3.5 (simple)
  • Self-hosted Llama 3 (70B)
  • Hybrid: Self-hosted (simple) + API (complex)

Local or Cloud?

  • Cloud (API)
  • Self-hosted
  • Hybrid

Justification: (Think about your answer before looking at the solution)

Solution

Recommended stack:

  • Model: A hybrid strategy:
    • Self-hosted Llama 3 (70B): Simple tasks (~80% of requests: autocomplete, spell checking).
    • The GPT-4 API: Complex tasks (~20%: advanced rewriting, tone adjustment).

Why:

  1. Volume (10M requests/month): A pure cloud API would be prohibitive ($100K-500K/month with GPT-4).
  2. Self-hosted for simple (80%): Llama 3 (70B) is enough for autocomplete → a fixed cost ($5K-10K/month for GPUs).
  3. An API for complex (20%): GPT-4 is better → a variable cost ($10K-30K/month).
  4. Total: $15K-40K/month (vs $100K+ with a pure API).

Setup:

  • Self-hosted: Llama 3 (70B) on multiple A100 GPUs (vLLM for serving).
  • Cloud: OpenAI GPT-4 for complex cases.

Accepted trade-off: The complexity of maintaining infrastructure (it requires a DevOps team).

Case 4: A Freelance Developer (a Prototype)

Context

  • Person: A freelance developer.
  • Product: A prototype of a content generation app (blog posts, emails).
  • Expected volume: 50 requests/month (testing).
  • AI budget: $0-20/month.
  • Requirements: Fast iteration, no long-term commitment.

Your Decision

Which provider/model would you use?

  • OpenAI GPT-4
  • OpenAI GPT-3.5-turbo
  • Local (Ollama + Llama 3 8B)
  • OpenRouter

Local or Cloud?

  • Local
  • Cloud (API)

Justification: (Think about your answer before looking at the solution)

Solution

Recommended stack:

  • Model: Local (Ollama + Llama 3 8B).
  • Local/Cloud: Local.

Why:

  1. A $0-20/month budget: Local is free (only the hardware you already have).
  2. Low volume (50 requests/month): It doesn't require powerful GPUs → Llama 3 8B runs on a laptop.
  3. Fast iteration: Local allows testing without worrying about API costs.
  4. A prototype: It doesn't require GPT-4 quality, Llama 3 8B is enough.

Setup:

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Download Llama 3 (8B)
ollama pull llama3:8b

# Run it
ollama run llama3:8b

When it scales (real users): Migrate to a cloud API (GPT-3.5 or Haiku).

Accepted trade-off: Quality below GPT-4 (but enough for a prototype).

Case 5: A Multimodal App (Text + Image)

Context

  • Company: An education startup.
  • Product: An educational assistant that analyzes diagrams/charts (text + image).
  • Expected volume: 10K requests/month.
  • AI budget: $500/month.
  • Requirements: Multimodal processing (text + image), Spanish.

Your Decision

Which provider/model would you use?

  • OpenAI GPT-4 Vision
  • Anthropic Claude 3 Opus (multimodal)
  • Google Gemini 1.5 Pro (multimodal)
  • Self-hosted Llama 3 (text-only)

Local or Cloud?

  • Local
  • Cloud (API)

Justification: (Think about your answer before looking at the solution)

Solution

Recommended stack:

  • Model: OpenAI GPT-4 Vision or Claude 3 Opus (multimodal).
  • Local/Cloud: Cloud (API).

Why:

  1. The multimodal requirement: Llama 3 is text-only → ruled out.
  2. Volume (10K requests/month): A cloud API is appropriate ($200-500/month with GPT-4 Vision).
  3. GPT-4 Vision vs Claude 3 Opus:
    • GPT-4 Vision: A better ecosystem (more documentation, examples).
    • Claude 3 Opus: Slightly better at analyzing complex documents (charts, diagrams).
    • Recommendation: Try both (an A/B test) → choose the best one for your case.

Setup:

import openai

response = openai.ChatCompletion.create(
    model="gpt-4-vision-preview",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What does this diagram show?"},
            {"type": "image_url", "image_url": {"url": "https://..."}}
        ]
    }]
)

Accepted trade-off: A cost ~2-3× higher than text-only (but it's a critical requirement).

Case 6: A Multi-Model Strategy (Cost Optimization)

Context

  • Company: A Series A startup ($5M raised).
  • Product: A customer service chatbot.
  • Expected volume: 500K requests/month.
  • AI budget: $10K/month.
  • Requirements:
    • 80% of questions are simple FAQs.
    • 20% require complex reasoning.

Your Decision

Which stack would you use?

  • GPT-4 for everything
  • GPT-3.5 for everything
  • GPT-3.5 (simple) + GPT-4 (complex)
  • OpenRouter (multi-model with fallback)

Justification: (Think about your answer before looking at the solution)

Solution

Recommended stack:

  • Model: A multi-model strategy:
    • Claude 3 Haiku or GPT-3.5-turbo: Simple FAQs (80% = 400K requests).
    • GPT-4: Complex cases (20% = 100K requests).
  • Implementation: OpenRouter (for easy switching and fallback).

Why:

  1. Cost optimization:
    • Haiku alone (400K simple): $100/month.
    • GPT-4 (100K complex): $3K-6K/month.
    • Total: $3.1K-6.1K/month (vs $15K-30K/month with pure GPT-4).
  2. OpenRouter: It allows switching between models with 1 line of code + automatic fallback.

Setup:

import openai

client = openai.OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key="sk-or-..."
)

def answer_query(query, is_complex):
    model = "openai/gpt-4" if is_complex else "anthropic/claude-3-haiku"
    
    response = client.chat.completions.create(
        model=model,
        fallbacks=["anthropic/claude-3-opus"],  # Backup if it fails
        messages=[{"role": "user", "content": query}]
    )
    return response.choices[0].message.content

Accepted trade-off: The complexity of determining what's "complex" (it requires a classifier).

Final Reflection

Key questions when choosing a stack:

  1. What volume do you expect? (low → an API, high → self-hosted).
  2. What budget do you have? (limited → GPT-3.5/Haiku, high → GPT-4).
  3. Is privacy critical? (yes → self-hosted, no → an API).
  4. Do you need multimodal? (yes → GPT-4 Vision/Claude/Gemini, no → text-only).
  5. What latency do you require? (<200ms → self-hosted or edge, >500ms → an API is fine).

There's no "perfect stack": It depends on your requirements, constraints, trade-offs.

Module Summary

What you learned:

  1. The main providers: OpenAI, Anthropic, Google, Meta, Mistral → differences, pricing, capabilities.
  2. Open-source vs Proprietary: The trade-offs (control vs convenience, fixed vs variable cost).
  3. Pricing: How you're charged (per token), optimization (the appropriate model, caching, concise prompts).
  4. Local vs Cloud: When to use Ollama/LM Studio (privacy, development, offline) vs APIs (production, scale).
  5. Aggregators: OpenRouter, Together.ai, Replicate → unified access, automatic fallback.
  6. Application: Choosing a stack based on requirements (volume, budget, privacy, latency).

Next step in the guide: Module 7 (AI Engineering) → the AI Engineer role, how it differs from ML Engineering and Data Science.

← Previous6. Aggregators and Routers: OpenRouter, Together.ai and Unified AccessNext →1. Module Introduction: A Role That Didn't Exist 3 Years Ago