7. Zero-Shot vs Few-Shot: Decision Framework

Overview

When should you use zero-shot and when few-shot? This capsule lays out a decision framework based on task type, consistency requirements, cost, and latency. It includes a comparison table with metrics, benchmarks on common tasks, a decision tree, and an implementable decision function.

Why it matters: The wrong call has real costs: unnecessary few-shot increases tokens and latency by 2-3x; zero-shot when few-shot is needed produces inconsistent outputs that break your downstream pipeline. This framework lets you make the call in seconds, with objective criteria.

The Complete Comparison Table

The Decision Tree

Step 1: Is the task standard or niche-domain?
│
├── STANDARD (translation, summarization, sentiment, basic NER)
│   │
│   ├── Do you need a very specific format (custom schema, internal fields)?
│   │   ├── NO → Zero-shot ✅
│   │   └── YES → Few-shot with 2-3 format examples
│   │
│   └── Have you already tried zero-shot and it gets >88% accuracy on your dataset?
│       ├── YES → Stay with zero-shot ✅
│       └── NO → Try few-shot with 3 examples
│
└── NICHE (internal categories, jargon, custom format, specialized classification)
    │
    ├── Do you have labeled examples available?
    │   ├── YES → Few-shot (3-5 examples) ✅
    │   └── NO → Generate synthetics with an LLM → validate 10 by hand → use few-shot
    │
    └── Is cost/latency critical?
        ├── YES → Try zero-shot first; if accuracy < 80%, few-shot
        └── NO → Few-shot directly

Step 2: If you go with few-shot, how many examples?
│
├── 2-3 examples: When the pattern is simple and format is the main thing
├── 4-5 examples: When there's variety in the cases or multiple categories
└── 6-10 examples: When the categories are very similar to each other (high ambiguity)

Benchmark: 10 Common Tasks

These values are averages documented in the literature and validated in practice:

Note: The values vary with the model, the prompt's quality, and the domain. Use them as a reference for what to expect, not as absolute truth.

Implementation: The Decision System

from openai import OpenAI
from enum import Enum
from typing import Literal
import time

client = OpenAI()

class TaskType(str, Enum):
    STANDARD = "standard"          # translation, summarization, sentiment
    NICHE_DOMAIN = "niche"         # custom categories, internal jargon
    CRITICAL_FORMAT = "format"     # a very specific schema

def recommend_technique(
    task_type: TaskType,
    has_examples: bool,
    cost_critical: bool = False,
    required_accuracy: float = 0.80
) -> tuple[Literal["zero-shot", "few-shot"], int, str]:
    """
    Recommends a technique, a number of examples, and a rationale.
    
    Returns:
        (technique, n_examples, rationale)
    """
    if task_type == TaskType.STANDARD:
        if cost_critical:
            return "zero-shot", 0, "Standard task + critical cost → zero-shot first, always"
        return "zero-shot", 0, "Standard task → zero-shot is enough in 85%+ of cases"
    
    if task_type == TaskType.NICHE_DOMAIN:
        if not has_examples:
            return "zero-shot", 0, "No examples: try zero-shot, if accuracy < 80% generate synthetics"
        n = 5 if required_accuracy > 0.85 else 3
        return "few-shot", n, f"Niche domain with examples: few-shot ({n} ex) for {required_accuracy:.0%} accuracy"
    
    if task_type == TaskType.CRITICAL_FORMAT:
        return "few-shot", 3, "Critical format: 2-3 examples anchor the exact schema"
    
    # Default: few-shot if there are examples, zero-shot if not
    if has_examples:
        return "few-shot", 3, "Default: few-shot available → use it"
    return "zero-shot", 0, "No examples available → zero-shot"

# A function that runs and measures both techniques
def measure_techniques(
    test_texts: list[tuple[str, str]],  # (text, true_label)
    categories: list[str],
    few_shot_examples: list[tuple[str, str]],
    k: int = 3
) -> dict:
    """
    Runs zero-shot and few-shot on a test set and compares the metrics.
    """
    stats = {
        "zero_shot": {"correct": 0, "total_tokens": 0, "total_latency": 0},
        "few_shot": {"correct": 0, "total_tokens": 0, "total_latency": 0}
    }
    
    cats_str = ", ".join(categories)
    
    # Build the base few-shot prompt
    examples_str = "\n".join([f"Example: '{t}' → {c}" for t, c in few_shot_examples[:k]])
    
    for text, true_label in test_texts:
        
        # === ZERO-SHOT ===
        t0 = time.time()
        r_zs = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {
                    "role": "system",
                    "content": f"Classify into: {cats_str}. Respond with the category only."
                },
                {"role": "user", "content": text}
            ],
            temperature=0,
            max_tokens=20
        )
        latency_zs = (time.time() - t0) * 1000
        
        pred_zs = r_zs.choices[0].message.content.strip().upper()
        if true_label.upper() in pred_zs or pred_zs in true_label.upper():
            stats["zero_shot"]["correct"] += 1
        stats["zero_shot"]["total_tokens"] += r_zs.usage.total_tokens
        stats["zero_shot"]["total_latency"] += latency_zs
        
        # === FEW-SHOT ===
        t0 = time.time()
        prompt_fs = f"""
Classify into: {cats_str}.

{examples_str}

Text: '{text}'
Category:"""
        
        r_fs = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt_fs}],
            temperature=0,
            max_tokens=20
        )
        latency_fs = (time.time() - t0) * 1000
        
        pred_fs = r_fs.choices[0].message.content.strip().upper()
        if true_label.upper() in pred_fs or pred_fs in true_label.upper():
            stats["few_shot"]["correct"] += 1
        stats["few_shot"]["total_tokens"] += r_fs.usage.total_tokens
        stats["few_shot"]["total_latency"] += latency_fs
    
    n = len(test_texts)
    return {
        "n_test": n,
        "zero_shot": {
            "accuracy": stats["zero_shot"]["correct"] / n,
            "avg_tokens": stats["zero_shot"]["total_tokens"] / n,
            "avg_latency_ms": stats["zero_shot"]["total_latency"] / n
        },
        "few_shot": {
            "accuracy": stats["few_shot"]["correct"] / n,
            "avg_tokens": stats["few_shot"]["total_tokens"] / n,
            "avg_latency_ms": stats["few_shot"]["total_latency"] / n
        }
    }

Cost vs Accuracy Trade-Off: The Real Math

Approximate cost per request (GPT-4o-mini, March 2026)

# Simplified cost, for analysis
COST_PER_1M_TOKENS = 0.15  # USD, gpt-4o-mini input

def calculate_monthly_cost(
    requests_per_day: int,
    tokens_per_request_zs: int,    # Zero-shot
    tokens_per_request_fs: int,    # Few-shot
    accuracy_zs: float,
    accuracy_fs: float,
    cost_per_error: float = 0.0    # Cost in USD of handling an incorrect output
) -> dict:
    """
    Computes and compares the total cost (API + errors) of each technique.
    """
    days_month = 30
    total_requests = requests_per_day * days_month
    
    # API cost
    api_cost_zs = total_requests * tokens_per_request_zs * COST_PER_1M_TOKENS / 1_000_000
    api_cost_fs = total_requests * tokens_per_request_fs * COST_PER_1M_TOKENS / 1_000_000
    
    # Cost of errors
    errors_zs = total_requests * (1 - accuracy_zs)
    errors_fs = total_requests * (1 - accuracy_fs)
    error_cost_zs = errors_zs * cost_per_error
    error_cost_fs = errors_fs * cost_per_error
    
    return {
        "zero_shot": {
            "api_cost_usd": round(api_cost_zs, 2),
            "error_cost_usd": round(error_cost_zs, 2),
            "total_cost_usd": round(api_cost_zs + error_cost_zs, 2)
        },
        "few_shot": {
            "api_cost_usd": round(api_cost_fs, 2),
            "error_cost_usd": round(error_cost_fs, 2),
            "total_cost_usd": round(api_cost_fs + error_cost_fs, 2)
        },
        "recommendation": "few-shot" if (api_cost_fs + error_cost_fs) < (api_cost_zs + error_cost_zs) else "zero-shot"
    }

# Example: a support chatbot with 1000 requests/day
result = calculate_monthly_cost(
    requests_per_day=1000,
    tokens_per_request_zs=200,
    tokens_per_request_fs=500,
    accuracy_zs=0.75,     # 75% in a niche domain
    accuracy_fs=0.90,     # 90% with few-shot
    cost_per_error=0.50   # $0.50 per error (wrong routing to an agent)
)

print("Monthly cost (30 days, 1000 req/day):")
print(f"  Zero-shot: ${result['zero_shot']['total_cost_usd']} (API: ${result['zero_shot']['api_cost_usd']} + Errors: ${result['zero_shot']['error_cost_usd']})")
print(f"  Few-shot:  ${result['few_shot']['total_cost_usd']} (API: ${result['few_shot']['api_cost_usd']} + Errors: ${result['few_shot']['error_cost_usd']})")
print(f"  Recommendation: {result['recommendation']}")

# Expected output:
# Monthly cost (30 days, 1000 req/day):
#   Zero-shot: $3375.90 (API: $0.90 + Errors: $3375.00)
#   Few-shot:  $4502.25 (API: $2.25 + Errors: $4500.00)
# Hold on — let's check the numbers...
# Zero-shot errors: 1000*30*0.25 = 7500 errors * $0.50 = $3750
# Few-shot errors: 1000*30*0.10 = 3000 errors * $0.50 = $1500
# Few-shot total: $2.25 + $1500 = $1502.25 → MUCH LOWER
# Recommendation: few-shot ✅

The key lesson: The cost of a downstream error usually dwarfs few-shot's extra token cost. Always include the cost of errors in your analysis.

When Few-Shot Is Worth It: A Quick Guide

Comparison with Code: The Complete Benchmark

from openai import OpenAI
import time
from collections import Counter

client = OpenAI()

# Evaluation dataset: classifying tickets into 4 internal categories
CATEGORIES = ["ACCESS", "INVOICE", "INTEGRATION", "OTHER"]

TEST_SET = [
    ("I haven't been able to get into the system since yesterday", "ACCESS"),
    ("My password expired and I can't reset it", "ACCESS"),
    ("The March invoice has a duplicate charge", "INVOICE"),
    ("Can I see my payment history?", "INVOICE"),
    ("I need to connect with Salesforce", "INTEGRATION"),
    ("Do you have an API to integrate with our ERP?", "INTEGRATION"),
    ("Do you have weekend support?", "OTHER"),
    ("What are your business hours?", "OTHER"),
]

TRAINING_EXAMPLES = [
    ("Login error with the correct username", "ACCESS"),
    ("Double charge on my latest statement", "INVOICE"),
    ("The webhook isn't reaching our server", "INTEGRATION"),
]

def run_benchmark() -> dict:
    correct_zs = 0
    correct_fs = 0
    tokens_zs = []
    tokens_fs = []
    
    examples_text = "\n".join([f"'{t}' → {c}" for t, c in TRAINING_EXAMPLES])
    cats_str = ", ".join(CATEGORIES)
    
    for text, true_label in TEST_SET:
        # Zero-shot
        r_zs = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": f"Classify into: {cats_str}. The category only."},
                {"role": "user", "content": text}
            ],
            temperature=0, max_tokens=15
        )
        pred_zs = r_zs.choices[0].message.content.strip().upper()
        if true_label in pred_zs:
            correct_zs += 1
        tokens_zs.append(r_zs.usage.total_tokens)
        
        # Few-shot
        r_fs = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"""
Classify into: {cats_str}.

Examples:
{examples_text}

Text: '{text}'
Category:"""}],
            temperature=0, max_tokens=15
        )
        pred_fs = r_fs.choices[0].message.content.strip().upper()
        if true_label in pred_fs:
            correct_fs += 1
        tokens_fs.append(r_fs.usage.total_tokens)
    
    n = len(TEST_SET)
    return {
        "zero_shot": {
            "accuracy": f"{correct_zs/n:.0%}",
            "avg_tokens": sum(tokens_zs) // n
        },
        "few_shot": {
            "accuracy": f"{correct_fs/n:.0%}",
            "avg_tokens": sum(tokens_fs) // n
        }
    }

results = run_benchmark()
print("BENCHMARK RESULTS:")
for technique, data in results.items():
    print(f"  {technique}: accuracy={data['accuracy']}, tokens/req={data['avg_tokens']}")

Connection to the Project

In the Few-Shot Classification System (capsule 08) you'll implement this decision framework:

• The recommend_technique() function as part of the engine
• The benchmark as the system's evaluation tool
• The cost calculation as part of the comparison report

Troubleshooting

Problem 1: Zero-shot gives inconsistent formats on standard tasks

Cause: A task that's "standard" for the model, but with an output that isn't standard for your case.

Fix: Zero-shot + an explicit output format with an example. If it still fails, then yes — few-shot.

Problem 2: Few-shot doesn't improve accuracy over zero-shot

Possible causes:

• The examples are redundant with each other (all the same pattern)
• The task is already well solved by zero-shot
• The examples aren't representative of the cases that fail

Diagnosis:

# Measure accuracy on the specific cases where zero-shot fails
failures_zs = [(t, e) for t, e in TEST_SET if classify_zs(t) != e]
print(f"Zero-shot fails on {len(failures_zs)}/{len(TEST_SET)} cases")
# Does few-shot solve them?
for text, label in failures_zs:
    pred_fs = classify_fs(text)
    print(f"  '{text[:40]}' → expected: {label}, few-shot: {pred_fs}")

Problem 3: No labeled examples for a new domain

Step-by-step fix:

1. Generate 10 synthetic examples with an LLM: "Give me 10 example tickets for the ACCESS category"
2. Validate by hand that they're correct (5 minutes)
3. Use them as the initial few-shot
4. Add real examples as they come in from the system

Problem 4: Few-shot's cost is prohibitive at high volume

Fix: Dynamic few-shot — only include the K most relevant for each specific input:

# Instead of always using the same 5 examples,
# pick the 3 most similar to the current input
relevant_examples = bank.k_nearest(current_text, k=3)
# Cuts average tokens and can improve accuracy

Exercises

Exercise 1: Apply the decision tree

For each task, walk the tree and justify: (a) Translate tweets into Spanish, (b) Classify tickets into 12 internal categories, (c) Extract dates in YYYY-MM-DD format, (d) Generate support responses in a formal tone.

Exercise 2: Compute the cost break-even

If zero-shot gets 75% accuracy and few-shot 90% on intent classification, and each misclassification costs $1 in extra manual support, how many requests/day justify few-shot's extra cost (assuming +400 tokens/request and a cost of $0.15 per million)?

# Few-shot's additional cost (API only) per request
extra_tokens = 400
cost_per_extra_token = 0.15 / 1_000_000
extra_api_cost_per_req = extra_tokens * cost_per_extra_token  # $0.00006

# Error reduction per request (15% fewer errors)
error_reduction = 0.90 - 0.75  # 15%
savings_per_req = error_reduction * 1.0  # $1 per error saved = $0.15 per request

# Break-even: extra_api_cost == error_savings → always positive!
# $0.00006 of extra cost vs $0.15 of savings per req
# Few-shot pays for itself from the FIRST request in this scenario

print(f"Extra API cost per request: ${extra_api_cost_per_req:.6f}")
print(f"Error savings per request: ${savings_per_req:.4f}")
print(f"Few-shot ROI: {savings_per_req/extra_api_cost_per_req:.0f}x")
# ROI: 2500x — few-shot is the obvious call

Exercise 3: Your own benchmark

Implement the run_benchmark() benchmark with your own dataset of 10 examples, on a task you care about. Report accuracy and tokens for both techniques.

# A customizable benchmark template
def my_benchmark(
    my_test_set: list[tuple[str, str]],
    my_categories: list[str],
    my_examples: list[tuple[str, str]],
    k: int = 3
) -> None:
    """
    Runs the benchmark and shows formatted results.
    """
    result = measure_techniques(my_test_set, my_categories, my_examples, k)
    
    print(f"\n=== BENCHMARK ({result['n_test']} examples) ===")
    print(f"{'Technique':<12} {'Accuracy':<12} {'Tokens/req':<12}")
    print("-" * 36)
    for technique in ["zero_shot", "few_shot"]:
        data = result[technique]
        print(f"{technique:<12} {data['accuracy']:.0%}{'':<9} {data['avg_tokens']:.0f}")
    
    # Difference
    zs = result["zero_shot"]["accuracy"]
    fs = result["few_shot"]["accuracy"]
    diff = (fs - zs) * 100
    print(f"\nFew-shot improvement: +{diff:.1f}%")
    if diff > 5:
        print("→ Few-shot JUSTIFIED")
    else:
        print("→ Zero-shot may be enough")

Exercise 4: Document your decision

For a real or hypothetical project, document: task type, chosen technique, a rationale with expected metrics, and a validation plan.

## Technique Decision: [System name]

### Task
Classify [description] into categories: [list]

### Analysis
- Task type: standard / niche / critical format
- Examples available: Yes (N examples) / No
- Required accuracy: X%
- Cost critical: Yes / No
- Estimated volume: N requests/day

### Decision
Chosen technique: zero-shot / few-shot (N examples)

### Rationale
- [Main reason, based on the decision tree]
- Expected accuracy: X-Y%
- Estimated cost: $X/month

### Validation plan
- Test dataset: N labeled examples
- Metrics to measure: accuracy, tokens/req, latency
- Success criterion: accuracy > X%
- If it fails: [contingency plan]

Summary

• Zero-shot: Standard tasks (translation, summarization, sentiment), flexible format, critical cost
• Few-shot: Niche domain, custom categories, specific format, accuracy > 85% required
• Number of examples: 2-3 for a simple format, 4-5 for multiple categories, 6-10 for high ambiguity
• The real cost: Include the cost of downstream errors. Few-shot with a 15% improvement can have a 2500x ROI
• Benchmark first: Measure on your dataset before deciding. The tables are references, not truths
• The decision tree: Standard + no format → zero-shot. Niche/format/custom categories → few-shot

Special Cases: When the Rules Don't Apply

1. Models with reasoning capabilities (o1, o3)

Models with internal reasoning (OpenAI o1/o3, Gemini Flash Thinking) tend to work better with zero-shot, even on niche tasks:

# o1 and o3 do internal reasoning — less dependent on examples
def classify_with_o1(text: str, categories: list[str]) -> str:
    """
    For reasoning models, a more elaborate zero-shot can beat few-shot.
    These models don't support the 'temperature' parameter — it's ignored.
    """
    response = client.chat.completions.create(
        model="o1-mini",
        messages=[
            {
                "role": "user",
                "content": f"""
Classify the following text into one of these categories: {', '.join(categories)}.

Reason briefly about the text's content and its best category.
End with: Category: [CATEGORY_NAME]

Text: {text}
"""
            }
        ],
        max_completion_tokens=200
    )
    
    content = response.choices[0].message.content
    # Pull the category out of the structured response
    import re
    match = re.search(r"Category:\s*(\w+)", content)
    if match:
        cat = match.group(1).upper()
        for c in categories:
            if c.upper() == cat:
                return c
    return content.strip()

# Rule of thumb: if you use o1/o3, always start with zero-shot.
# If accuracy < threshold, add reasoning instructions, not examples.

2. Extraction tasks vs classification tasks

The extraction/classification distinction shifts the balance:

The rule for extraction: If the entity is unknown to the model, few-shot is required. If it's only the format, one example is enough.

The Automated Decision Function

This function takes measurable parameters and returns a recommendation:

from dataclasses import dataclass
from typing import Optional

@dataclass
class DecisionContext:
    task_type: str               # "classification", "extraction", "generation", "translation"
    domain: str                  # "standard", "niche", "proprietary"
    required_accuracy: float     # e.g. 0.85
    has_examples: bool
    n_categories: Optional[int]  # Classification only
    critical_format: bool        # Do you need an exact schema?
    requests_per_day: int
    cost_per_error_usd: float    # The downstream cost of an error
    model_name: str              # "gpt-4o-mini", "gpt-4o", "o1-mini", etc.

@dataclass
class Recommendation:
    technique: str                   # "zero-shot" | "few-shot"
    n_examples: int
    rationale: list[str]
    risks: list[str]
    next_steps: list[str]

def decision_framework(ctx: DecisionContext) -> Recommendation:
    """
    An automated decision framework for zero-shot vs few-shot.
    """
    technique = "zero-shot"
    n_examples = 0
    rationale = []
    risks = []
    steps = []
    
    # 1. Reasoning models — zero-shot by default
    if ctx.model_name in ("o1", "o1-mini", "o3", "o3-mini"):
        rationale.append("Model with internal reasoning: zero-shot first")
        return Recommendation(
            technique="zero-shot", n_examples=0,
            rationale=rationale,
            risks=["variable accuracy in very niche domains"],
            next_steps=["Measure accuracy", "If < threshold, refine the reasoning instructions (not the examples)"]
        )
    
    # 2. Standard task + standard domain
    if ctx.task_type in ("translation", "summarization") and ctx.domain == "standard":
        rationale.append(f"Standard '{ctx.task_type}' task: zero-shot is enough")
        if ctx.critical_format:
            technique = "few-shot"
            n_examples = 2
            rationale.append("But the format is critical: 2 examples to anchor the schema")
        return Recommendation(
            technique=technique, n_examples=n_examples,
            rationale=rationale,
            risks=["The format may be inconsistent"] if not ctx.critical_format else [],
            next_steps=["Try it with 5 representative inputs", "Measure format consistency"]
        )
    
    # 3. Classification
    if ctx.task_type == "classification":
        n_cats = ctx.n_categories or 3
        
        if ctx.domain == "proprietary":
            # Proprietary domain: few-shot, always
            n_examples = min(max(3, n_cats), 10)  # 1 per category, min 3
            rationale.append(f"Proprietary domain with {n_cats} categories: few-shot with {n_examples} examples")
            
            if not ctx.has_examples:
                risks.append("No examples: you need to generate them (synthetic or by hand)")
                steps.append("Generate 2-3 synthetic examples per category with an LLM")
                steps.append("Validate by hand for 20 minutes")
            
            technique = "few-shot"
        
        elif ctx.domain == "niche" and n_cats > 5:
            technique = "few-shot"
            n_examples = 5
            rationale.append(f"{n_cats} niche categories: few-shot required")
        
        elif ctx.required_accuracy > 0.90:
            technique = "few-shot"
            n_examples = 5
            rationale.append(f"Required accuracy {ctx.required_accuracy:.0%}: few-shot for the extra margin")
        
        else:
            rationale.append("Standard classification: zero-shot as the baseline")
            steps.append("Measure zero-shot accuracy first. If < 80%, come back with few-shot")
    
    # 4. Extraction
    elif ctx.task_type == "extraction":
        if ctx.domain == "proprietary" or ctx.critical_format:
            technique = "few-shot"
            n_examples = 3
            rationale.append("Extraction with custom entities or format: few-shot is essential")
        else:
            technique = "zero-shot"
            rationale.append("Standard entity extraction: zero-shot + an explicit format")
            steps.append("State the schema explicitly in the prompt (even without giving examples)")
    
    # 5. ROI calculation
    if technique == "few-shot" and ctx.requests_per_day > 0 and ctx.cost_per_error_usd > 0:
        estimated_extra_tokens = n_examples * 30  # ~30 tokens per example
        extra_api_cost_per_month = (
            ctx.requests_per_day * 30 * estimated_extra_tokens * 0.15 / 1_000_000
        )
        estimated_improvement = 0.10  # A conservative 10% improvement
        error_savings_per_month = (
            ctx.requests_per_day * 30 * estimated_improvement * ctx.cost_per_error_usd
        )
        
        if error_savings_per_month > extra_api_cost_per_month * 5:
            rationale.append(
                f"Positive ROI: estimated savings ${error_savings_per_month:.0f}/month "
                f"vs extra API cost ${extra_api_cost_per_month:.2f}/month"
            )
    
    return Recommendation(
        technique=technique, n_examples=n_examples,
        rationale=rationale,
        risks=risks,
        next_steps=steps or ["Test on a dataset of 20+ examples before production"]
    )

# Example usage
ctx_tickets = DecisionContext(
    task_type="classification",
    domain="proprietary",
    required_accuracy=0.88,
    has_examples=True,
    n_categories=12,
    critical_format=False,
    requests_per_day=500,
    cost_per_error_usd=2.00,
    model_name="gpt-4o-mini"
)

rec = decision_framework(ctx_tickets)
print(f"\nTechnique: {rec.technique} ({rec.n_examples} examples)")
print("Rationale:")
for j in rec.rationale:
    print(f"  • {j}")
if rec.next_steps:
    print("Next steps:")
    for p in rec.next_steps:
        print(f"  → {p}")

Anti-Few-Shot Patterns (When NOT to Use It)

Even though few-shot helps in many cases, there are patterns where it actively hurts:

Anti-pattern 1: Stale examples

# ❌ If the examples reflect the old behavior, few-shot cements the error
BAD_EXAMPLES = [
    ("Code: ERR-4001", "ACCESS"),      # ERR-4001 used to be access
    ("Code: ERR-5003", "BILLING"),     # ERR-5003 has since been redefined
]

# ✅ Keep the example bank current, or use zero-shot if the examples can go stale

Anti-pattern 2: Unrepresentative examples

# ❌ 3 identical examples teach nothing new
REDUNDANT_EXAMPLES = [
    ("I can't get into the system", "ACCESS"),
    ("I can't log in", "ACCESS"),
    ("I'm having trouble accessing", "ACCESS"),
]
# The model already knows how to classify access. These tokens are wasted.

# ✅ Diversity across categories + diversity within each category
DIVERSE_EXAMPLES = [
    ("I can't get into the system", "ACCESS"),      # Access
    ("My invoice has an extra charge", "INVOICE"),  # Invoice
    ("I want to cancel the subscription", "ACCOUNT"), # Account
]

Anti-pattern 3: Too many examples with low diversity

# ❌ 10 examples where 8 are ACCESS
UNBALANCED_EXAMPLES = [
    ("I can't get in", "ACCESS"),
    ("Login error", "ACCESS"),
    ("Incorrect password", "ACCESS"),
    ("It won't let me access", "ACCESS"),
    ("Error logging in", "ACCESS"),
    ("Access denied", "ACCESS"),
    ("Session expired", "ACCESS"),
    ("Locked out after failed attempts", "ACCESS"),
    ("Duplicate invoice", "INVOICE"),
    ("Incorrect charge", "INVOICE"),
]
# The model learns to predict ACCESS for almost everything

# ✅ Balance the examples across categories

Further resources

1. OpenAI Prompt Engineering Guide — Recommended strategies for zero-shot and few-shot
2. Scaling Laws for Few-Shot Learning (Zhao et al., 2021) — An empirical analysis of when few-shot works
3. Calibrate Before Use: Improving Few-Shot Performance (Zhao et al., 2021) — The impact of example order in few-shot
4. Prompt Engineering Guide (DAIR.AI) - Few-Shot — A technical summary with comparative benchmarks
5. LangSmith Evaluation — A tool for benchmarking prompts systematically