Module 6: The AI API Ecosystem

4. Pricing and Token Economics: How You're Charged and How to Optimize Costs

Description

One of the most important decisions when using LLMs: how much does it cost? And more importantly: how do you optimize?

In this lesson you'll understand:

How pricing works (per token).
The cost differences between models.
How to optimize costs (caching, efficient prompts, the appropriate model).

How Pricing Works: Per Token

Basic rule: You pay for input tokens + output tokens.

Input tokens: The tokens you send (the prompt).
Output tokens: The tokens the model generates (the response).

Example (GPT-4):

Input: $0.03 / 1,000 tokens.
Output: $0.06 / 1,000 tokens (2× more expensive than input).

An example calculation

Prompt:

User: "Explain in 100 words what a transformer is."

Input: ~20 tokens.

Response (GPT-4):

"A transformer is a neural network architecture that uses attention to process sequences... [100 words]"

Output: ~130 tokens.

Total cost (GPT-4):

Input: 20 tokens × $0.03 / 1,000 = $0.0006
Output: 130 tokens × $0.06 / 1,000 = $0.0078
Total: $0.0084 (~$0.01 per request)

Pricing Comparison (2024)

Model	Input ($/1K tokens)	Output ($/1K tokens)	Example cost (10K tokens in, 10K tokens out)
GPT-4	$0.03	$0.06	$0.90
GPT-4 Turbo	$0.01	$0.03	$0.40
GPT-3.5-turbo	$0.0005	$0.0015	$0.02
Claude 3 Opus	$0.015	$0.075	$0.90
Claude 3 Sonnet	$0.003	$0.015	$0.18
Claude 3 Haiku	$0.00025	$0.00125	$0.0125
Gemini 1.5 Pro	$0.0035	$0.0105	$0.14
Llama 3 (self-hosted)	Free (a fixed hosting cost)	Free (a fixed hosting cost)	$0 (but you pay for the GPU: $500-2K/month)

Observations:

GPT-4 is 30-60× more expensive than GPT-3.5.
Claude Haiku is 120× cheaper than GPT-4.
Self-hosted (Llama 3) is free per request, but you pay a fixed hosting cost.

Why Output Is More Expensive Than Input

Reason: Generating text requires more compute than processing it.

Input: A forward pass (1× for the whole input).
Output: A forward pass for each token generated (if you generate 100 tokens → 100 forward passes).

Implication: Generating long responses is more expensive than processing a long input.

Factors That Affect Cost

1. Language

Tokenization varies by language:

English: ~1.3 tokens per word.
Spanish: ~1.5 tokens per word.
Chinese, Japanese: ~2-3 tokens per word.

Implication: The same prompt in Spanish costs ~15% more than in English.

2. Response Length

max_tokens: It controls the maximum length of the response.

Example:

max_tokens=100 → a maximum of 100 tokens → a known maximum cost.
max_tokens=4000 → the model can generate up to 4000 tokens → an unpredictable cost.

Optimization: Use a low max_tokens when you don't need long responses.

3. Usage Frequency

High volume → self-hosted is better:

Example:

1M requests/month, 500 input tokens, 500 output tokens per request.

GPT-4:

Input: 1M × 500 × $0.03 / 1,000 = $15,000
Output: 1M × 500 × $0.06 / 1,000 = $30,000
Total: $45,000/month

Llama 3 (self-hosted on an A100):

GPU: $1,500/month (cloud)
Total: $1,500/month

Saving: 30× (self-hosted vs API).

Cost Optimization Strategies

1. Use the Appropriate Model

Don't use GPT-4 for everything:

Simple tasks (FAQ, categorization): GPT-3.5 or Claude Haiku (30-120× cheaper).
Complex tasks (reasoning, code): GPT-4.

Example: A chatbot with 90% simple questions, 10% complex ones.

Simple → GPT-3.5 ($0.02/request).
Complex → GPT-4 ($0.90/request).
Saving: 80% vs using GPT-4 for everything.

2. Prompt Caching

Problem: If you send the same context in every request (e.g. system instructions), you pay for that input repeatedly.

Solution (Anthropic Prompt Caching):

First request: You pay for the complete input.
Subsequent requests: The context is cached → you only pay for the new input (a 90% discount).

Example:

System instructions: 5,000 tokens (the same in every request).
Without cache: You pay 5,000 tokens per request.
With cache: You pay 5,000 tokens the first time, 0 tokens afterward.

3. Prompt Optimization

A concise prompt:

User: "Give me the 3 main features of TypeScript."

Input: 10 tokens.

A verbose prompt:

User: "Hi, I'm learning TypeScript. I'd like you to tell me what the 3 main features of TypeScript are that differentiate it from JavaScript. Please be brief."

Input: 35 tokens.

Saving: 70% (with the concise prompt).

4. Limit the Output

A low max_tokens:

response = openai.ChatCompletion.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Explain what AI is in 50 words."}],
    max_tokens=100  # Limits the response
)

Without max_tokens: The model can generate 1,000 tokens → $0.06.

With max_tokens=100: The model generates a maximum of 100 tokens → $0.006.

Saving: 10×.

5. Batch Requests

Problem: 1,000 individual requests → 1,000 API calls → overhead.

Solution: Process multiple inputs in a single prompt.

Example:

Individual: "Categorize: 'I love this product'" → 1 request.
Batch: "Categorize: 1. 'I love this product' 2. 'It's horrible' 3. 'It's okay'" → 1 request, 3 results.

Saving: ~70% (less overhead).

Common Cost Mistakes

1. Using GPT-4 for everything

Mistake: Using GPT-4 even for simple tasks (categorization, FAQ).

Solution: GPT-3.5 or Claude Haiku for simple ones, GPT-4 only for complex ones.

2. Not limiting max_tokens

Mistake: Not specifying max_tokens → the model generates very long responses.

Solution: Always specify an appropriate max_tokens.

3. Not caching repeated context

Mistake: Sending the same context (system instructions) in every request.

Solution: Use prompt caching (Anthropic) or a system message (OpenAI), which is cached automatically.

Why this matters for an AI Engineer

1. Budget

A startup with a $1K/month budget:

GPT-4: ~1,000 requests (with 1K input/output tokens).
GPT-3.5: ~50,000 requests.

Decision: GPT-3.5 for the MVP, then GPT-4 for premium features.

2. Scaling

Growth from 1K → 1M requests/month:

APIs: The cost grows linearly → it can become prohibitive.
Self-hosted: A fixed cost (the GPU) → more predictable.

Summary

Pricing:

You pay per token (input + output, output is 2× more expensive).
GPT-4: 30-60× more expensive than GPT-3.5.
Claude Haiku: 120× cheaper than GPT-4.

Optimization:

The appropriate model: GPT-3.5/Haiku for simple, GPT-4 for complex.
Prompt caching: Cache repeated context (Anthropic).
A concise prompt: Fewer tokens = less cost.
max_tokens: Limit unnecessarily long responses.
Batch requests: Process multiple inputs in one prompt.

Self-hosted (Llama 3):

Free per request, but a fixed cost (a GPU: $500-2K/month).
Better for high volume (>1M requests/month).

Next step: Lesson 05: Local vs Cloud — When to run models locally (LM Studio, Ollama) vs APIs.

← Previous3. Open-Source vs Proprietary: Trade-Offs and When to Use Each One Next →5. Local vs Cloud: LM Studio, Ollama and When to Use Each One