3. Cost Estimation Checks

Overview

A prompt that works perfectly in development can be a financial disaster in production. The difference between a 200-token system prompt and a 2,000-token one gets multiplied by every request. If your app receives 10,000 requests a day, those 1,800 extra tokens cost money — and nobody finds out until the invoice arrives at the end of the month.

Cost estimation in CI solves this: before a PR gets merged, an automated check calculates how many tokens the current prompt uses, estimates the cost per request, and compares it with the previous prompt. If the cost rises more than a defined threshold, CI fails and blocks the merge.

Why cost matters in CI

The scenario

Prompt v1 (current in production):
  System prompt: 150 tokens
  Total per request: ~400 tokens → $0.00024/req (gpt-4o-mini)

Prompt v2 (proposed in a PR):
  System prompt: 1,500 tokens (they added context, examples, rules)
  Total per request: ~1,950 tokens → $0.00117/req (gpt-4o-mini)

Increase: 4.9x → almost 5 times more expensive

At 10,000 requests/day:

Without a check in CI, that PR gets merged without anyone looking at the cost impact.

Reference prices (per 1M tokens)

These prices change — the script we'll build uses an editable config (JSON) so that updating prices means changing a file, not refactoring code.

Step 1: Count tokens with tiktoken

tiktoken is OpenAI's official library for counting tokens. It works locally with no API calls — perfect for CI because it doesn't consume credits.

pip install tiktoken

# scripts/token_counter.py
"""Utilities for counting tokens with tiktoken."""

import tiktoken


def count_tokens(text: str, model: str = "gpt-4o-mini") -> int:
    encoding = tiktoken.encoding_for_model(model)
    return len(encoding.encode(text))


def count_message_tokens(
    messages: list[dict],
    model: str = "gpt-4o-mini",
) -> int:
    encoding = tiktoken.encoding_for_model(model)
    tokens = 0

    for message in messages:
        tokens += 4  # per-message overhead (role, content markers)
        for key, value in message.items():
            tokens += len(encoding.encode(value))

    tokens += 2  # conversation start/end overhead
    return tokens


if __name__ == "__main__":
    system_prompt = """You are a technical assistant specialized in Python.
    Answer clearly and concisely. Include code when relevant."""

    user_message = "How do I make a GET request in Python?"

    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_message},
    ]

    print(f"System prompt tokens: {count_tokens(system_prompt)}")
    print(f"User message tokens: {count_tokens(user_message)}")
    print(f"Total message tokens: {count_message_tokens(messages)}")

OpenAI charges for the tokens of the whole conversation, not just the text. Every message has overhead (~4 tokens per message + 2 for the conversation). For cost estimates, always include the overhead.

Step 2: Estimate costs per model

The pricing configuration

{
  "pricing": {
    "gpt-4o-mini": {
      "input_per_1m": 0.15,
      "output_per_1m": 0.60
    },
    "gpt-4o": {
      "input_per_1m": 2.50,
      "output_per_1m": 10.00
    },
    "gpt-4-turbo": {
      "input_per_1m": 10.00,
      "output_per_1m": 30.00
    }
  },
  "defaults": {
    "model": "gpt-4o-mini",
    "estimated_output_tokens": 200,
    "cost_increase_threshold_pct": 20
  }
}

Save it as config/pricing.json. When the provider changes prices, you update the JSON — not the code.

The estimation script

# scripts/estimate_costs.py
"""
Cost estimation script for CI.
Calculates the estimated cost of prompts and compares against a baseline.
"""

import json
import os
import sys
from pathlib import Path

import tiktoken


def load_pricing(path: str = "config/pricing.json") -> dict:
    with open(path) as f:
        return json.load(f)


def count_tokens(text: str, model: str = "gpt-4o-mini") -> int:
    encoding = tiktoken.encoding_for_model(model)
    return len(encoding.encode(text))


def count_message_tokens(
    messages: list[dict], model: str = "gpt-4o-mini",
) -> int:
    encoding = tiktoken.encoding_for_model(model)
    tokens = 0
    for message in messages:
        tokens += 4
        for key, value in message.items():
            tokens += len(encoding.encode(value))
    tokens += 2
    return tokens


def estimate_request_cost(
    system_prompt: str,
    sample_user_message: str,
    model: str,
    pricing: dict,
    estimated_output_tokens: int = 200,
) -> dict:
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": sample_user_message},
    ]

    input_tokens = count_message_tokens(messages, model)
    output_tokens = estimated_output_tokens

    model_pricing = pricing["pricing"][model]
    input_cost = (input_tokens / 1_000_000) * model_pricing["input_per_1m"]
    output_cost = (output_tokens / 1_000_000) * model_pricing["output_per_1m"]
    total_cost = input_cost + output_cost

    return {
        "model": model,
        "input_tokens": input_tokens,
        "output_tokens": output_tokens,
        "total_tokens": input_tokens + output_tokens,
        "input_cost": input_cost,
        "output_cost": output_cost,
        "total_cost_per_request": total_cost,
        "cost_per_1k_requests": total_cost * 1_000,
        "cost_per_day_10k": total_cost * 10_000,
        "cost_per_month_10k": total_cost * 10_000 * 30,
    }


def run_cost_estimation(
    prompts_path: str = "config/prompts.json",
    pricing_path: str = "config/pricing.json",
) -> dict:
    pricing = load_pricing(pricing_path)
    with open(prompts_path) as f:
        prompts = json.load(f)

    model = pricing["defaults"]["model"]
    estimated_output = pricing["defaults"]["estimated_output_tokens"]
    results = []

    print("Cost Estimation Report")
    print("=" * 60)

    for prompt_config in prompts["prompts"]:
        name = prompt_config["name"]
        system_prompt = prompt_config["system_prompt"]
        sample_input = prompt_config.get(
            "sample_user_message", "Sample user question"
        )

        estimate = estimate_request_cost(
            system_prompt=system_prompt,
            sample_user_message=sample_input,
            model=model,
            pricing=pricing,
            estimated_output_tokens=estimated_output,
        )

        print(f"\n📋 Prompt: {name}")
        print(f"  Input tokens: {estimate['input_tokens']}")
        print(f"  Cost/request: ${estimate['total_cost_per_request']:.6f}")
        print(f"  Cost/day (10K req): ${estimate['cost_per_day_10k']:.2f}")
        print(f"  Cost/month (10K req): ${estimate['cost_per_month_10k']:.2f}")

        results.append({"name": name, **estimate})

    report = {
        "model": model,
        "estimated_output_tokens": estimated_output,
        "prompts": results,
        "total_cost_per_request": sum(r["total_cost_per_request"] for r in results),
    }

    print(f"\nTotal cost/request: ${report['total_cost_per_request']:.6f}")
    return report


def compare_with_baseline(
    current: dict, baseline_path: str = "cost_baseline.json",
) -> bool:
    threshold_pct = float(os.environ.get("COST_INCREASE_THRESHOLD_PCT", "20"))

    if not Path(baseline_path).exists():
        print(f"\nNo cost baseline found — skipping comparison")
        return True

    with open(baseline_path) as f:
        baseline = json.load(f)

    current_cost = current["total_cost_per_request"]
    baseline_cost = baseline["total_cost_per_request"]

    if baseline_cost == 0:
        return True

    pct_change = ((current_cost - baseline_cost) / baseline_cost) * 100

    print(f"\nCost comparison:")
    print(f"  Baseline: ${baseline_cost:.6f} | Current: ${current_cost:.6f}")
    print(f"  Change: {pct_change:+.1f}% | Threshold: {threshold_pct}%")

    if pct_change > threshold_pct:
        print(f"  ❌ COST INCREASE DETECTED: +{pct_change:.1f}% exceeds threshold")
        return False

    print(f"  ✅ Cost within threshold")
    return True


def save_report(report: dict, path: str = "cost_report.json") -> None:
    with open(path, "w") as f:
        json.dump(report, f, indent=2)
    print(f"\nReport saved to {path}")


if __name__ == "__main__":
    report = run_cost_estimation()
    save_report(report)

    passed = compare_with_baseline(report)

    if not passed:
        print("\n❌ Cost estimation check FAILED")
        sys.exit(1)

    print("\n✅ Cost estimation check PASSED")
    sys.exit(0)

The prompts file

{
  "prompts": [
    {
      "name": "main_assistant",
      "system_prompt": "You are a technical assistant specialized in Python and software development. Answer clearly and concisely. Include code examples when relevant.",
      "sample_user_message": "How do I make a GET request in Python?"
    }
  ]
}

Save it as config/prompts.json. Every prompt your app uses gets registered here.

Step 3: A cost regression scenario

A developer adds extensive context to the prompt (from 15 tokens to 160 tokens):

Cost comparison:
  Baseline cost/request: $0.000039
  Current cost/request:  $0.000156
  Change:                +300.0%
  Threshold:             20%
  ❌ COST INCREASE DETECTED: +300.0% exceeds 20% threshold

CI fails. The developer sees exactly why: a 10x increase in the system prompt that translates into +300% total cost.

Step 4: Integrate it into GitHub Actions

# .github/workflows/cost-check.yml
name: Cost Estimation
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
    # Only run when cost-relevant files change — saves CI minutes on unrelated PRs
    paths:
      - "config/prompts.json"
      - "config/pricing.json"
      - "scripts/estimate_costs.py"
      - "src/**"

jobs:
  cost-estimation:
    runs-on: ubuntu-latest
    timeout-minutes: 5

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Setup Python 3.12
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"

      - name: Install dependencies
        # tiktoken counts tokens locally — no API key needed, zero cost per run
        run: pip install tiktoken

      - name: Download cost baseline (if exists)
        uses: actions/download-artifact@v4
        with:
          name: cost-baseline
          path: .
        continue-on-error: true

      - name: Run cost estimation
        run: python scripts/estimate_costs.py
        env:
          COST_INCREASE_THRESHOLD_PCT: "20"

      - name: Upload cost report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: cost-report
          path: cost_report.json
          retention-days: 30

      - name: Update cost baseline on main
        if: github.ref == 'refs/heads/main' && success()
        uses: actions/upload-artifact@v4
        with:
          name: cost-baseline
          path: cost_report.json
          # Baseline retention spans billing cycles — enables month-over-month cost tracking
          retention-days: 90

The paths filter: The cost check only runs when relevant files change. There's no point estimating costs if only a README changed.

It needs no API key: Unlike prompt regression testing, cost estimation uses tiktoken locally. Zero API cost.

      - name: Post cost report as PR comment
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const report = JSON.parse(fs.readFileSync('cost_report.json', 'utf8'));

            let body = '## 💰 Cost Estimation Report\n\n';
            body += '| Prompt | Input Tokens | Cost/Request | Cost/Month (10K) |\n';
            body += '|--------|-------------|-------------|------------------|\n';

            for (const p of report.prompts) {
              body += `| ${p.name} | ${p.input_tokens} | $${p.total_cost_per_request.toFixed(6)} | $${p.cost_per_month_10k.toFixed(2)} |\n`;
            }

            body += `\n**Total cost/request:** $${report.total_cost_per_request.toFixed(6)}`;

            const { data: comments } = await github.rest.issues.listComments({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
            });

            const existing = comments.find(c => c.body.includes('Cost Estimation Report'));
            if (existing) {
              await github.rest.issues.updateComment({
                owner: context.repo.owner,
                repo: context.repo.repo,
                comment_id: existing.id,
                body: body,
              });
            } else {
              await github.rest.issues.createComment({
                owner: context.repo.owner,
                repo: context.repo.repo,
                issue_number: context.issue.number,
                body: body,
              });
            }

Troubleshooting

"tiktoken doesn't recognize the model"

Cause: The model's name doesn't match what tiktoken expects.

Solution: Use a fallback to the encoding:

try:
    encoding = tiktoken.encoding_for_model("gpt-4o-mini")
except KeyError:
    encoding = tiktoken.get_encoding("cl100k_base")

cl100k_base is the encoding most GPT-4 and GPT-3.5-turbo models use.

"The estimated cost doesn't match the real invoice"

Cause: The estimate counts input tokens but can't predict the output tokens exactly.

Solution: Use a conservative estimated_output_tokens based on real data. If you have no data, 200 tokens is a reasonable default for conversation, 400-500 for code generation.

"The 20% threshold is too sensitive"

Cause: Legitimate changes to the prompt can increase tokens by 25-30%.

Solution: Adjust it via an environment variable (COST_INCREASE_THRESHOLD_PCT: "50"), or use an absolute threshold:

MAX_COST_PER_REQUEST = 0.005
if current_cost > MAX_COST_PER_REQUEST:
    print(f"❌ Cost exceeds absolute limit")
    return False

Exercises

Exercise 1: Count the tokens of different prompts

Use tiktoken to compare the token count of these three system prompts and determine which is the most economical:

prompt_a = "Answer technical questions."
prompt_b = "You are a technical assistant specialized in Python. Answer clearly and concisely."
prompt_c = """You are a senior technical assistant with 15 years of experience in Python, 
JavaScript, and cloud computing. Your goal is to provide detailed answers, 
with code examples, best practices, and references to official documentation. 
Always structure your answer with: 1) Conceptual explanation, 2) Example code, 
3) Edge cases, 4) Additional resources."""

import tiktoken


def count_tokens(text: str, model: str = "gpt-4o-mini") -> int:
    encoding = tiktoken.encoding_for_model(model)
    return len(encoding.encode(text))


prompts = {"A (minimal)": prompt_a, "B (medium)": prompt_b, "C (extensive)": prompt_c}

for name, prompt in prompts.items():
    tokens = count_tokens(prompt)
    cost_input = (tokens / 1_000_000) * 0.15
    print(f"Prompt {name}: {tokens} tokens, ${cost_input * 10_000:.4f}/10K requests")

Prompt A (minimal): 7 tokens, $0.0105/10K requests
Prompt B (medium): 19 tokens, $0.0285/10K requests
Prompt C (extensive): 87 tokens, $0.1305/10K requests

Exercise 2: A pricing config for multiple models

Create config/pricing.json with prices for gpt-4o-mini, gpt-4o, and claude-3-haiku. Then write a function that, given a model name, returns the estimated cost per request.

import json
import tiktoken


def load_pricing(path: str = "config/pricing.json") -> dict:
    with open(path) as f:
        return json.load(f)


def estimate_cost(
    text: str, model: str, pricing: dict, estimated_output_tokens: int = 200,
) -> dict:
    try:
        encoding = tiktoken.encoding_for_model(model)
    except KeyError:
        encoding = tiktoken.get_encoding("cl100k_base")

    input_tokens = len(encoding.encode(text)) + 6
    model_pricing = pricing["pricing"][model]
    input_cost = (input_tokens / 1_000_000) * model_pricing["input_per_1m"]
    output_cost = (estimated_output_tokens / 1_000_000) * model_pricing["output_per_1m"]

    return {
        "model": model,
        "input_tokens": input_tokens,
        "total_cost": input_cost + output_cost,
    }


if __name__ == "__main__":
    pricing = load_pricing()
    prompt = "You are a technical assistant. Answer concisely."
    for model in ["gpt-4o-mini", "gpt-4o", "claude-3-haiku"]:
        result = estimate_cost(prompt, model, pricing)
        print(f"{model}: {result['input_tokens']} tokens, ${result['total_cost']:.6f}/req")

Exercise 3: An absolute threshold in addition to a percentage one

Modify compare_with_baseline so it has two modes: percentage (vs baseline) and absolute (maximum per request). If the cost exceeds an absolute maximum, it fails regardless of the percentage.

import json
import os
from pathlib import Path


def compare_with_baseline(
    current: dict, baseline_path: str = "cost_baseline.json",
) -> bool:
    threshold_pct = float(os.environ.get("COST_INCREASE_THRESHOLD_PCT", "20"))
    max_absolute = float(os.environ.get("MAX_COST_PER_REQUEST", "0.01"))
    current_cost = current["total_cost_per_request"]

    print(f"\nAbsolute cost check: ${current_cost:.6f} (max: ${max_absolute:.6f})")
    if current_cost > max_absolute:
        print(f"  ❌ EXCEEDS absolute limit")
        return False
    print(f"  ✅ Within absolute limit")

    if not Path(baseline_path).exists():
        print(f"No baseline found — skipping percentage comparison")
        return True

    with open(baseline_path) as f:
        baseline = json.load(f)

    baseline_cost = baseline["total_cost_per_request"]
    if baseline_cost == 0:
        return True

    pct_change = ((current_cost - baseline_cost) / baseline_cost) * 100
    print(f"\nPercentage check: {pct_change:+.1f}% (threshold: {threshold_pct}%)")

    if pct_change > threshold_pct:
        print(f"  ❌ EXCEEDS percentage threshold")
        return False
    print(f"  ✅ Within percentage threshold")
    return True

Exercise 4: A workflow with cost estimation and a path filter

Create a workflow that only runs cost estimation when prompt or pricing files change, and that includes posting the results as a PR comment.

# .github/workflows/cost-check.yml
name: Cost Estimation
on:
  pull_request:
    branches: [main]
    paths:
      - "config/prompts.json"
      - "config/pricing.json"
      - "scripts/estimate_costs.py"

jobs:
  cost-check:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    permissions:
      pull-requests: write

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
          cache: "pip"
      - run: pip install tiktoken

      - name: Download cost baseline
        uses: actions/download-artifact@v4
        with:
          name: cost-baseline
          path: .
        continue-on-error: true

      - name: Run cost estimation
        run: python scripts/estimate_costs.py
        env:
          COST_INCREASE_THRESHOLD_PCT: "20"
          MAX_COST_PER_REQUEST: "0.01"

      - name: Post cost report to PR
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const report = JSON.parse(fs.readFileSync('cost_report.json', 'utf8'));
            let body = '## Cost Estimation Report\n\n';
            body += '| Prompt | Tokens | Cost/Req | Cost/Month (10K) |\n';
            body += '|--------|--------|----------|------------------|\n';
            for (const p of report.prompts) {
              body += `| ${p.name} | ${p.input_tokens} | $${p.total_cost_per_request.toFixed(6)} | $${p.cost_per_month_10k.toFixed(2)} |\n`;
            }
            await github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: body,
            });

      - name: Upload cost report
        uses: actions/upload-artifact@v4
        if: always()
        with:
          name: cost-report
          path: cost_report.json

Summary

• ✅ Cost estimation in CI is a financial guardrail that detects cost increases before the merge
• ✅ tiktoken counts tokens locally with no API calls — free and instant
• ✅ The pricing config is a separate JSON so you can update prices without changing code
• ✅ The script estimates cost per request, per 1K requests, per day, and per month
• ✅ Two types of threshold: percentage (vs the previous baseline) and absolute (maximum per request)
• ✅ Path filters in the workflow avoid running the check when the prompts didn't change
• ✅ Baselines are updated on main — on PRs they're only compared

Additional resources

1. tiktoken — OpenAI — Official library for counting tokens
2. OpenAI Pricing — Up-to-date prices per model
3. Anthropic Pricing — Claude's prices
4. OpenAI Tokenizer Tool — Visual tool for exploring tokens
5. GitHub Actions — actions/github-script — For posting comments on PRs