6. Comparison: Casual Prompt vs Engineered Prompt

Capsule overview

In this capsule you'll see side-by-side the results of naive prompts versus prompts designed with this module's principles. You'll measure the difference with concrete data: format consistency, parseability by code, tokens consumed, and estimated cost. Without this evidence, it's easy to underestimate the value of prompt engineering — it looks like extra effort for something that "already works".

The goal isn't for you to abandon simple prompts forever, but for you to understand in which contexts the difference is critical and in which it doesn't matter. You'll learn to pick the appropriate level of engineering for each situation and to measure the improvement objectively, not by intuition.

Why it matters: In Module 7 you'll learn to evaluate prompts systematically. This capsule is the bridge between "I know I improved it" and "I have data showing how much I improved it". The comparison framework you'll see here is the simplified version of what you'll use with formal metrics in production.

Fundamentals: why casual prompts are inconsistent

A casual prompt fails because it specifies none of these elements:

1. No system prompt: The model uses its default behavior, which varies by model and version
2. No output format: The model picks how to respond (it could be one word or three paragraphs)
3. No controlled temperature: The default varies across providers (typically 0.7-1.0), adding variability
4. No constraints: The model can add explanations, warnings, unrequested suggestions
5. No edge case handling: Atypical inputs produce unpredictable outputs

Analogy: A casual prompt is like a function with no type hints, no input validation, and no defined return value. It works sometimes, and fails silently other times.

Experiment 1: sentiment classification

Experiment setup

from openai import OpenAI
import json

client = OpenAI()

test_inputs = [
    "The service was terrible, I won't be back.",
    "The product arrived in perfect condition, very happy.",
    "So-so, it could be better but it's not bad either.",
    "AMAZING! Best purchase I made this year!!",
    "It doesn't work like I expected. Disappointing."
]

Casual prompt — run and analysis

def classify_casual(text: str) -> str:
    """Prompt with no structure, no system, default temperature."""
    r = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "user", "content": f"Is this text positive or negative? \"{text}\""}
        ]
        # No temperature (the model's default)
        # No max_tokens
        # No system prompt
    )
    return r.choices[0].message.content.strip()

print("=== Casual Prompt Results ===")
casual_results = []
for text in test_inputs:
    result = classify_casual(text)
    casual_results.append(result)
    print(f"Input: {text[:50]}")
    print(f"Output: {result}\n")

Typical output (varies between runs):

Input: The service was terrible, I won't be back.
Output: This text is clearly negative. The author expresses dissatisfaction with the service and states they won't return.

Input: The product arrived in perfect condition, very happy.
Output: Positive

Input: So-so, it could be better but it's not bad either.
Output: The text has a neutral or mixed tone. It's neither clearly positive nor negative.

Input: AMAZING! Best purchase I made this year!!
Output: It's positive.

Input: It doesn't work like I expected. Disappointing.
Output: Negative. The text expresses disappointment.

Problem: Five different formats in five responses. Impossible to parse with code.

Engineered prompt — run and analysis

SYSTEM_CLASSIFIER = """
You are a text sentiment classifier.

TASK: Classify the sentiment of the text into exactly one of these categories:
- POSITIVE
- NEGATIVE
- NEUTRAL

RULES:
- Respond ONLY with the category (one word in uppercase)
- No explanations, no punctuation, no extra text
- For mixed texts, pick the DOMINANT sentiment
"""

def classify_engineered(text: str) -> str:
    """Structured prompt, system prompt, temperature=0."""
    r = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": SYSTEM_CLASSIFIER},
            {"role": "user", "content": text}
        ],
        temperature=0,       # Deterministic
        max_tokens=10        # You only need one word
    )
    return r.choices[0].message.content.strip()

print("=== Engineered Prompt Results ===")
eng_results = []
for text in test_inputs:
    result = classify_engineered(text)
    eng_results.append(result)
    print(f"Input: {text[:50]}")
    print(f"Output: {result}\n")

Consistent output:

Input: The service was terrible, I won't be back.
Output: NEGATIVE

Input: The product arrived in perfect condition, very happy.
Output: POSITIVE

Input: So-so, it could be better but it's not bad either.
Output: NEUTRAL

Input: AMAZING! Best purchase I made this year!!
Output: POSITIVE

Input: It doesn't work like I expected. Disappointing.
Output: NEGATIVE

Five responses, five identical formats. Directly parseable: result in ["POSITIVE", "NEGATIVE", "NEUTRAL"].

Quantitative metrics: a direct comparison

def measure_format_consistency(results: list[str], 
                                valid_formats: set[str]) -> float:
    """What % of responses have a valid, parseable format."""
    valid = sum(1 for r in results if r in valid_formats)
    return valid / len(results)

def measure_avg_tokens(results: list[str]) -> float:
    """Average tokens per response (approximate)."""
    return sum(len(r.split()) for r in results) / len(results)

valid_formats = {"POSITIVE", "NEGATIVE", "NEUTRAL"}

casual_consistency = measure_format_consistency(casual_results, valid_formats)
eng_consistency = measure_format_consistency(eng_results, valid_formats)

tokens_casual = measure_avg_tokens(casual_results)
tokens_eng = measure_avg_tokens(eng_results)

print("=== Metrics Comparison ===")
print(f"Casual consistency:      {casual_consistency:.0%}")
print(f"Engineered consistency:  {eng_consistency:.0%}")
print(f"Tokens/response casual:  {tokens_casual:.1f}")
print(f"Tokens/response eng:     {tokens_eng:.1f}")

Expected output:

=== Metrics Comparison ===
Casual consistency:      20%
Engineered consistency:  100%
Tokens/response casual:  18.4
Tokens/response eng:     1.0

Full metrics table

Experiment 2: structured data extraction

The difference is even more pronounced when the output has to be structured.

Casual prompt for extraction

def extract_casual(text: str) -> str:
    r = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": f"Extract the name and email from this text: \"{text}\""
        }]
    )
    return r.choices[0].message.content.strip()

texts = [
    "Contact Maria Garcia at mgarcia@company.com about the meeting",
    "The owner is Juan (juan.perez@gmail.com) and his backup Carlos (carlos@mail.com)",
    "There is no contact information in this message"
]

print("=== Casual Extraction ===")
for t in texts:
    print(f"Input: {t[:60]}")
    print(f"Output: {extract_casual(t)}\n")

Typical output (problematic):

Input: Contact Maria Garcia at mgarcia@company.com about the meeting
Output: Name: Maria Garcia
        Email: mgarcia@company.com

Input: The owner is Juan (juan.perez@gmail.com) and his backup Carlos (carlos@mail.com)
Output: Two people with their emails are mentioned in the text:
        1. Juan - juan.perez@gmail.com
        2. Carlos - carlos@mail.com

Input: There is no contact information in this message
Output: No name or email was found in the provided text.

Problem: Three completely different formats. How do you parse this with code?

Engineered prompt for extraction

SYSTEM_EXTRACTOR = """
Extract names and emails from text. 

RESPONSE FORMAT (strict JSON, nothing else):
{"names": ["name1", "name2"], "emails": ["email1", "email2"]}

RULES:
- If there are multiple people, include all of them
- If there are no names: {"names": [], "emails": [...]}
- If there are no emails: {"names": [...], "emails": []}
- If there's nothing: {"names": [], "emails": []}
- Only valid JSON. No extra text.
"""

def extract_engineered(text: str) -> dict:
    r = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": SYSTEM_EXTRACTOR},
            {"role": "user", "content": text}
        ],
        temperature=0,
        response_format={"type": "json_object"}
    )
    return json.loads(r.choices[0].message.content)

print("=== Engineered Extraction ===")
for t in texts:
    result = extract_engineered(t)
    print(f"Input: {t[:60]}")
    print(f"Output: {result}")
    # Check that it's parseable
    assert isinstance(result["names"], list)
    assert isinstance(result["emails"], list)
    print("  ✅ Parseable and valid\n")

Consistent output:

Input: Contact Maria Garcia at mgarcia@company.com about the meeting
Output: {'names': ['Maria Garcia'], 'emails': ['mgarcia@company.com']}
  ✅ Parseable and valid

Input: The owner is Juan (juan.perez@gmail.com) and his backup Carlos (carlos@mail.com)
Output: {'names': ['Juan', 'Carlos'], 'emails': ['juan.perez@gmail.com', 'carlos@mail.com']}
  ✅ Parseable and valid

Input: There is no contact information in this message
Output: {'names': [], 'emails': []}
  ✅ Parseable and valid

When a casual prompt is enough

You don't always need full engineering. The decision depends on the context of use:

A casual prompt is enough when:

1. Personal exploration: You're testing ideas, generating drafts, exploring possibilities. The output goes to your eyes, not to a system.

2. Manual one-off: You run the prompt a single time, you review the output by hand, and you won't repeat it.

3. Rapid prototyping: You're validating whether a concept works before investing time in design. A 30-minute validation cycle.

4. Task with a human in the loop: The output goes through human review before it's used. The person can fix inconsistencies.

5. Trivially simple task: "Translate 'hello' into Spanish." There's no possible ambiguity; the model's natural format is already the right one.

You need prompt engineering when:

1. Production: The output goes to a system, to real users, or to code that processes it. Failures have consequences.

2. Parseable output: Your code expects JSON, a specific category, a number, a concrete format. A badly formatted output breaks the system.

3. Consistency required: The same input must produce the same output (or a predictable range). Multi-user, multi-region, multi-provider.

4. Evaluation and improvement: You need to compare prompt versions with objective metrics. Without a consistent format, you can't measure.

5. Scale: The prompt runs 1,000+ times a day. Every extra token is real cost. Every inconsistent format is a parsing failure.

6. Multi-provider: Your prompt has to work on OpenAI, Anthropic or any other model. Without explicit structure, behavior varies.

# Practical decision rule
def needs_engineering(context: dict) -> tuple[bool, str]:
    """
    context: dict with flags for the use case.
    Returns (needs, reason).
    """
    if context.get("production"):
        return True, "Production: consistency is critical"
    if context.get("parseable_output"):
        return True, "Parseable output: strict format required"
    if context.get("calls_per_day", 0) >= 100:
        return True, f"Scale: {context['calls_per_day']} calls/day → optimize tokens"
    if context.get("multi_provider"):
        return True, "Multi-provider: needs explicit portable instructions"
    if context.get("needs_evaluation"):
        return True, "Evaluation: without a consistent format you can't measure improvements"
    return False, "Exploration/one-off: a casual prompt is enough"

# Examples
cases = [
    {"description": "Support chatbot in production", "production": True, "parseable_output": True},
    {"description": "One-off script to translate 10 titles", "calls_per_day": 1},
    {"description": "Data extractor running 500 times/day", "calls_per_day": 500, "parseable_output": True},
    {"description": "Quick prototype for an internal demo", "calls_per_day": 5}
]

for case in cases:
    needs, reason = needs_engineering(case)
    prompt_type = "ENGINEERED" if needs else "CASUAL"
    print(f"{case['description']}: [{prompt_type}] — {reason}")

Output:

Support chatbot in production: [ENGINEERED] — Production: consistency is critical
One-off script to translate 10 titles: [CASUAL] — Exploration/one-off: a casual prompt is enough
Data extractor running 500 times/day: [ENGINEERED] — Scale: 500 calls/day → optimize tokens
Quick prototype for an internal demo: [CASUAL] — Exploration/one-off: a casual prompt is enough

A framework for comparing prompts

When you want to measure whether a new version of a prompt is better than the previous one, you need a basic benchmark:

from typing import Callable
import time

def benchmark_prompts(
    fn_casual: Callable[[str], str],
    fn_engineered: Callable[[str], str],
    test_inputs: list[str],
    ground_truth: list[str],
    valid_formats: set[str]
) -> dict:
    """
    Compares two prompt versions with quantitative metrics.
    
    Returns:
        dict with metrics for each version
    """
    results = {"casual": {}, "engineered": {}}
    
    for name, fn in [("casual", fn_casual), ("engineered", fn_engineered)]:
        outputs = []
        times = []
        
        for inp in test_inputs:
            start = time.time()
            output = fn(inp)
            elapsed = time.time() - start
            outputs.append(output)
            times.append(elapsed)
        
        # Metrics
        valid_format = sum(1 for o in outputs if o in valid_formats)
        avg_tokens = sum(len(o.split()) for o in outputs) / len(outputs)
        
        # Accuracy vs ground truth (if applicable)
        accuracy = None
        if ground_truth:
            correct = sum(1 for o, g in zip(outputs, ground_truth) if o == g)
            accuracy = correct / len(ground_truth)
        
        results[name] = {
            "format_consistency": valid_format / len(test_inputs),
            "avg_tokens": avg_tokens,
            "avg_latency_s": sum(times) / len(times),
            "accuracy": accuracy,
            "outputs": outputs
        }
    
    return results

# Usage
ground_truth = ["NEGATIVE", "POSITIVE", "NEUTRAL", "POSITIVE", "NEGATIVE"]

metrics = benchmark_prompts(
    fn_casual=classify_casual,
    fn_engineered=classify_engineered,
    test_inputs=test_inputs,
    ground_truth=ground_truth,
    valid_formats={"POSITIVE", "NEGATIVE", "NEUTRAL"}
)

print("=== Benchmark Results ===\n")
for version, m in metrics.items():
    print(f"[{version.upper()}]")
    print(f"  Format consistency:       {m['format_consistency']:.0%}")
    print(f"  Average tokens:           {m['avg_tokens']:.1f}")
    print(f"  Average latency:          {m['avg_latency_s']:.2f}s")
    if m['accuracy'] is not None:
        print(f"  Accuracy vs ground truth: {m['accuracy']:.0%}")
    print()

Expected output:

=== Benchmark Results ===

[CASUAL]
  Format consistency:       20%
  Average tokens:           18.4
  Average latency:          0.38s
  Accuracy vs ground truth: 60%

[ENGINEERED]
  Format consistency:       100%
  Average tokens:           1.0
  Average latency:          0.31s
  Accuracy vs ground truth: 100%

Impact at scale

The differences look small on a single prompt. At scale, they change your operating cost:

def calculate_scale_cost(
    avg_tokens: float,
    calls_per_day: int,
    price_per_1k_tokens: float = 0.00015  # gpt-4o-mini output
) -> dict:
    """Calculates the estimated monthly cost."""
    daily_tokens = avg_tokens * calls_per_day
    daily_cost = (daily_tokens / 1000) * price_per_1k_tokens
    monthly_cost = daily_cost * 30
    
    return {
        "daily_tokens": daily_tokens,
        "daily_cost_usd": daily_cost,
        "monthly_cost_usd": monthly_cost
    }

calls_per_day = 10_000  # A mid-sized real system

casual_cost = calculate_scale_cost(18.4, calls_per_day)
eng_cost = calculate_scale_cost(1.0, calls_per_day)

print(f"=== Token impact at {calls_per_day:,} calls/day ===\n")
print(f"Casual prompt:     ${casual_cost['monthly_cost_usd']:.2f}/month")
print(f"Engineered prompt: ${eng_cost['monthly_cost_usd']:.2f}/month")
savings = casual_cost['monthly_cost_usd'] - eng_cost['monthly_cost_usd']
print(f"Monthly savings:   ${savings:.2f} (-{savings/casual_cost['monthly_cost_usd']:.0%})")

Output:

=== Token impact at 10,000 calls/day ===

Casual prompt:     $0.83/month
Engineered prompt: $0.05/month
Monthly savings:   $0.78 (-95%)

Note: with gpt-4o-mini the costs are low, but with gpt-4o or with premium models the numbers multiply 10-20x. The principle of minimizing unnecessary tokens applies in every case.

Connection to the project

In the Prompt Analyzer (capsule 08) you'll implement a function that detects whether a prompt has "casual" or "engineered" characteristics based on:

• Presence of a system prompt (yes/no)
• Explicit components detected (instruction, format, constraints)
• Temperature specified (yes/no)
• Edge case handling documented

The resulting score is one of the "improvement suggestions" the analyzer will generate: "This prompt is casual (2/5 engineered characteristics). Add a system prompt and an explicit format to increase consistency."

Troubleshooting

Problem 1: The engineered prompt sometimes fails too

Cause: Edge cases not covered in the design. No prompt is 100% perfect on the first try.

Fix: Document the inputs that fail, identify the pattern, and add to the prompt:

• In Insight: a specific rule for that kind of input
• In Experiment: an example of the edge case with its correct output

# Add the edge case to the Experiment
IMPROVED_EXPERIMENT = """
...previous examples...

Edge cases:
- Empty input or only whitespace → NEUTRAL
- Input with only positive emojis (😊👍) → POSITIVE
- Input in another language → classify it anyway (sentiment is universal)
"""

Problem 2: I don't know which metrics to use for comparison

Cause: No definition of "success" before comparing.

Fix: Define it before you run:

• For classification: accuracy vs ground truth (labels verified by humans)
• For extraction: precision and recall of fields (does it extract all of them? only the correct ones?)
• For format: % of responses that parse without error
• For cost: average tokens per successful call

Problem 3: My "ground truth" is hard to build

Cause: You don't have labeled examples to measure accuracy.

Fix: Start with a small but representative set (20-50 examples):

1. Pick cases covering the happy path, edge cases, and ambiguous cases
2. Label them by hand (or with a high-quality model as "judge")
3. Use those same cases on every prompt version

Problem 4: The comparison doesn't show a noticeable difference

Cause: The test inputs are too simple (they don't exercise edge cases).

Fix: Add inputs that force the model to choose:

• Mixed texts (positive and negative in the same sentence)
• Inputs with irony or sarcasm
• Very short inputs ("OK", "no")
• Inputs with typos or informality

Exercises

Exercise 1: Design your comparison (Easy)

Pick a task you like (e.g.: extract URLs from a text, classify email type, detect language) and write (a) a casual prompt and (b) an engineered prompt. Run both 5 times with the same input and compare format consistency.

from openai import OpenAI
import json

client = OpenAI()

# Task: Extract URLs from a text

# (a) Casual prompt
def casual_urls(text: str) -> str:
    r = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"Extract the URLs from this text: {text}"}]
    )
    return r.choices[0].message.content

# (b) Engineered prompt
SYSTEM_URLS = """
Extract all the URLs from a text.
Respond ONLY with JSON: {"urls": ["url1", "url2"]}
If there are no URLs: {"urls": []}
No extra text.
"""

def engineered_urls(text: str) -> dict:
    r = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": SYSTEM_URLS},
            {"role": "user", "content": text}
        ],
        temperature=0,
        response_format={"type": "json_object"}
    )
    return json.loads(r.choices[0].message.content)

# Test
text = "Visit https://openai.com and also https://anthropic.com for more info"

print("Casual (5 runs):")
for _ in range(5):
    print(f"  {casual_urls(text)[:80]}")

print("\nEngineered (5 runs):")
for _ in range(5):
    result = engineered_urls(text)
    print(f"  {result}")
    # Check parseability
    assert isinstance(result["urls"], list)

Exercise 2: Pick the right level (Easy)

For each scenario, decide whether a casual prompt is enough or you need engineering, and justify it:

a) An internal tool for a dev to extract data from logs by hand
b) A support chatbot that classifies intent before routing to a team
c) A one-off script to translate 10 marketing titles
d) A public API that processes user text and returns a JSON analysis

Exercise 3: Build the basic benchmark (Medium)

Extend the benchmark_prompts from the example to add an "input tokens" metric (not just output). Why is it relevant to also measure the tokens of the prompt itself?

def benchmark_with_input_tokens(
    fn_casual: Callable,
    fn_engineered: Callable,
    system_casual: str,
    system_eng: str,
    test_inputs: list[str]
) -> dict:
    """
    Includes input tokens (system + user) in the comparison.
    Input tokens are billed too and they affect the total cost.
    """
    import tiktoken
    enc = tiktoken.encoding_for_model("gpt-4o-mini")
    
    results = {}
    for name, fn, system in [
        ("casual", fn_casual, system_casual),
        ("engineered", fn_engineered, system_eng)
    ]:
        outputs = [fn(inp) for inp in test_inputs]
        
        # Input tokens = system + each user message
        total_input_tokens = 0
        for inp in test_inputs:
            total_input_tokens += len(enc.encode(system + inp))
        
        total_output_tokens = sum(len(enc.encode(o)) for o in outputs)
        avg_total_tokens = (total_input_tokens + total_output_tokens) / len(test_inputs)
        
        results[name] = {
            "avg_input_tokens": total_input_tokens / len(test_inputs),
            "avg_output_tokens": total_output_tokens / len(test_inputs),
            "avg_total_tokens": avg_total_tokens
        }
    
    return results

Exercise 4: Measure temperature's impact on consistency (Medium)

Run the same casual prompt with temperature 0, 0.5, and 1.0 (5 times each). How does temperature affect format consistency?

from openai import OpenAI
client = OpenAI()

text = "The product is good but the price is very high"
simple_prompt = "Is this text positive or negative?"

print("=== Temperature's impact on consistency ===\n")
for temp in [0.0, 0.5, 1.0]:
    outputs = []
    for _ in range(5):
        r = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"{simple_prompt}\n\n{text}"}],
            temperature=temp
        )
        outputs.append(r.choices[0].message.content.strip()[:60])
    
    unique_formats = len(set(outputs))
    print(f"temperature={temp}:")
    for o in outputs:
        print(f"  '{o}'")
    print(f"  Unique formats: {unique_formats}/5 (fewer = more consistent)\n")

Summary

In this capsule you learned:

• Casual prompt: No system prompt, no explicit format, no controlled temperature → variable results, hard to parse, expensive at scale
• Engineered prompt: System prompt + explicit format + temperature=0 + controlled max_tokens → consistent results, directly parseable, cost-optimized
• Comparison metrics: Format consistency (%), average tokens, accuracy vs ground truth, latency, direct parseability
• When casual is enough: Personal exploration, one-off with human review, rapid prototyping
• When you need engineering: Production, output parsed by code, scale, systematic evaluation, multi-provider
• Impact at scale: 18 average tokens vs 1 token = a 95% cost difference at 10k calls/day
• Benchmark framework: benchmark_prompts() measures consistency, tokens, latency and accuracy — the foundation of Module 7 (Evaluation)

Next capsule: Behavioral differences between providers — why the prompt that works on OpenAI may need adjustments on Anthropic, and how to design portable prompts.

Additional resources

1. OpenAI Evals — The official framework for evaluating prompts with datasets and metrics, the basis for what you'll do in Module 7
2. OpenAI Pricing — Current per-token pricing to calculate the real impact at scale
3. Prompt Engineering Best Practices — OpenAI — The official guide with strategies for improving consistency and reducing tokens
4. tiktoken — OpenAI's library for counting tokens exactly before you run (avoids cost surprises)
5. Anthropic Prompt Caching — A technique for reducing costs on prompts with long repeated system prompts (a preview of Module 8)
6. BLEU, ROUGE and evaluation metrics — Hugging Face — More formal metrics you'll use in Module 7 to evaluate the quality of free-text outputs