Module 5: LLMs (Large Language Models) - GPT, Claude and More

8. Integrative Exercise: Exploring and Comparing LLMs

Next →

Description

This is the final lesson of Module 5, where you'll apply everything you learned about LLMs. The goal is for you to experiment with parameters, compare models, and calculate costs.

What you'll practice:

  1. Counting tokens in texts (English vs Spanish).
  2. Experimenting with temperature (0 vs 0.5 vs 1).
  3. Comparing LLMs (GPT-4 vs Claude 3 vs Llama 3) on a task.
  4. Calculating estimated costs for an application.
  5. Choosing the appropriate LLM based on requirements.

Exercise 1: Counting Tokens

Task: Use the OpenAI Tokenizer to count tokens.

Texts:

  1. "Hello, how are you today?" (English)
  2. "Hola, ¿cómo estás hoy?" (Spanish)
  3. "你好，今天怎么样？" (Chinese)

Question: How many tokens does each one have? Which language uses the most tokens?

See solution

Approximate results:

  1. English: ~7 tokens
  2. Spanish: ~9 tokens (~29% more)
  3. Chinese: ~15-20 tokens (~114-186% more)

Moral: Non-English languages use more tokens → more expensive.

Exercise 2: Experimenting with Temperature

Task: Use ChatGPT (or Claude) with different temperatures.

Prompt: "Write a creative tagline for a coffee shop"

Parameters:

  • Run 1: temp=0
  • Run 2: temp=0.5
  • Run 3: temp=1.0

Question: What differences do you observe? Which would you use for your support chatbot?

See expected solution

Observations:

  • temp=0: The same tagline every time (e.g. "The best coffee in town"). Generic, predictable.
  • temp=0.5: Varied but reasonable (e.g. "Where every cup tells a story").
  • temp=1.0: Very creative, different every time (e.g. "Awakening your senses, one sip at a time").

For a support chatbot: temp=0.3-0.5 (consistent but not robotic responses).

Exercise 3: Comparing LLMs on a Task

Task: Compare GPT-4 vs Claude 3 vs Llama 3 on reasoning.

Prompt:

Solve step by step:
If 5 machines can make 5 widgets in 5 minutes, how long does it take 100 machines to make 100 widgets?

Question: Which model answers correctly? Are there differences in the explanation?

See expected solution

Correct answer: 5 minutes (not 100 minutes).

Reasoning:

  • 1 machine makes 1 widget in 5 minutes.
  • 100 machines make 100 widgets in parallel → also 5 minutes.

Models that usually get it right:

  • GPT-4: The correct answer with clear reasoning.
  • Claude 3 Opus: The correct answer, a detailed explanation.
  • Llama 3 (70B/400B): The correct answer (but 8B can fail).

Moral: Large models (GPT-4, Claude Opus, Llama 400B) have better reasoning than small models.

Exercise 4: Calculating Costs

Scenario: A customer support chatbot.

Data:

  • Average: 10K requests/day.
  • Average prompt: 50 tokens.
  • Average response: 200 tokens.

Question: What's the monthly cost with GPT-4 vs GPT-3.5 vs Claude 3 Haiku?

See solution

GPT-4:

Input: 10K requests × 50 tokens = 500K tokens/day
Output: 10K requests × 200 tokens = 2M tokens/day

Daily cost:
- Input: 500K × $0.03/1K = $15
- Output: 2M × $0.06/1K = $120
- Total: $135/day

Monthly cost: $135 × 30 = $4,050

GPT-3.5-turbo:

Daily cost:
- Input: 500K × $0.001/1K = $0.50
- Output: 2M × $0.002/1K = $4
- Total: $4.50/day

Monthly cost: $4.50 × 30 = $135

Claude 3 Haiku:

Daily cost:
- Input: 500K × $0.00025/1K = $0.125
- Output: 2M × $0.00125/1K = $2.50
- Total: $2.625/day

Monthly cost: $2.625 × 30 = $78.75

Comparison:

  • GPT-4: $4,050/month (more expensive, smarter).
  • GPT-3.5: $135/month (30× cheaper than GPT-4).
  • Claude Haiku: $79/month (51× cheaper than GPT-4, almost 2× cheaper than GPT-3.5).

Moral: For a simple chatbot with high volume, Claude Haiku or GPT-3.5 are better options (cost/benefit).

Exercise 5: Choosing an LLM Based on Requirements

For each scenario, choose the most appropriate LLM and justify it:

Scenario 1: A startup with a $500/month AI budget, a simple FAQ chatbot.

See solution

Recommendation: Claude 3 Haiku or GPT-3.5-turbo.

Justification:

  • A limited budget → you need a cheap model.
  • A simple FAQ → it doesn't require GPT-4's complex reasoning.
  • Claude Haiku is ~50× cheaper than GPT-4.

Calculation: With $500/month and Claude Haiku, you can do ~190K requests with 200 tokens of response each.

Scenario 2: A legal firm that analyzes 100-300 page contracts.

See solution

Recommendation: Claude 3 Opus or Gemini 1.5.

Justification:

  • Long contracts (100-300 pages → 75K-225K words → 100K-300K tokens).
  • GPT-4 (128K tokens) can't process complete contracts of 200+ pages.
  • Claude 3 (200K tokens) or Gemini 1.5 (1M tokens) can.
  • Accuracy is critical in legal → use Opus (more capable), not Haiku.

Scenario 3: A health startup with patient data (HIPAA compliance).

See solution

Recommendation: Llama 3 (local, self-hosted).

Justification:

  • Sensitive data (HIPAA) → you can't send it to external APIs (OpenAI, Anthropic).
  • Llama 3 is open-source → you can run it locally (total control, privacy).
  • It requires infrastructure (GPUs), but that's necessary for compliance.

Scenario 4: A multimodal application that analyzes educational videos.

See solution

Recommendation: Gemini 1.5.

Justification:

  • Advanced multimodal (video + audio + text).
  • GPT-4 only supports text + images (no video).
  • Gemini 1.5 can process video directly.

Exercise 6: Detecting When RAG is Necessary

For each case, say whether you need RAG:

  1. A FAQ chatbot (50 frequently asked questions, each answer ~100 words).
  2. A chatbot that answers questions about a knowledge base of 1,000 articles (total: 500K words).
  3. Analysis of a 50-page contract.
See solutions

Case 1: A simple FAQ

  • Is RAG necessary? NO (optional).
  • Why: 50 questions × 100 words = 5K words (~6.6K tokens) → it fits in any LLM's context window.
  • Option: Pass all the FAQs in the prompt (they fit comfortably).

Case 2: A knowledge base (1,000 articles, 500K words)

  • Is RAG necessary? YES.
  • Why: 500K words → ~666K tokens → it exceeds the context window of GPT-4 (128K), Claude 3 (200K).
  • Solution: RAG → search for the relevant articles based on the query, pass only the top-5 articles (~10K tokens).

Case 3: A contract (50 pages)

  • Is RAG necessary? It depends.
  • Analysis: 50 pages → ~12.5K words → ~16.6K tokens.
  • GPT-4 (128K tokens): You do NOT need RAG (it fits comfortably).
  • Option: Pass the complete contract (if it's <100 pages, you generally don't need RAG).

Summary of the exercise

What you practiced:

  1. Counting tokens: Understanding the differences between languages (English uses fewer tokens than Spanish/Chinese).
  2. Experimenting with temperature: Observing the differences (temp=0 deterministic, temp=1 creative).
  3. Comparing LLMs: GPT-4 has better reasoning, Claude 3 has a larger context window, Llama 3 is free.
  4. Calculating costs: Understanding the impact of volume and the model selected (GPT-4 is 30-50× more expensive than GPT-3.5/Haiku).
  5. Choosing a model: Based on requirements (intelligence, context window, cost, privacy, multimodal).

Evidence of success:

  • You can calculate estimated costs for an application.
  • You can choose the appropriate LLM based on the use case (without consulting the lessons).
  • You understand when you need RAG (documents that exceed the context window).

Next module: With this exercise you finish Module 5 (LLMs). In Module 6 (The AI API ecosystem) you'll see providers, detailed pricing, local vs cloud, aggregators (OpenRouter), and how to choose based on your use case.

Additional resources

  1. OpenAI Tokenizer — Count tokens online.
  2. OpenAI Pricing — Up-to-date prices for GPT-4, GPT-3.5.
  3. Anthropic Pricing — Claude 3's prices.
  4. Artificial Analysis — A comparison of LLMs (quality, speed, price).
  5. LLM Pricing Calculator — A cost calculator.
← Previous7. The Major LLMs: GPT-4, Claude 3, Llama 3, Gemini and MoreNext →1. Module Introduction: APIs as Access to AI