3. Few-Shot: Choosing Examples

Capsule overview

The quality of few-shot prompting depends critically on which examples you pick, how many you use, and what order you present them in. Badly chosen examples can make results worse than zero-shot. Redundant examples waste tokens without improving accuracy. Biased examples teach the model incorrect patterns that are hard to diagnose later.

In this capsule you'll learn the 3-5 example sweet spot (and why it works), how to select for diversity and representativeness, the impact of ordering, and when to use negative examples to cover edge cases. All of it with runnable Python code and metrics to verify your decisions.

Why it matters: In the final project's Few-Shot Classification System, the quality of the examples in the bank directly determines the system's accuracy. If you understand the selection principles, you can build a bank that works well from day one instead of iterating blindly.

How Many Examples to Use: The Sweet Spot

The performance curve

Empirical research shows that most classification tasks follow this curve:

Accuracy
  │
  │          ●────────────────── plateau
  │        ●
  │      ●
  │    ●
  │  ●
  │●
  └─────────────────────────
  0    2    4    6    8   10
           # of examples

• 0 examples (zero-shot): Baseline accuracy. Varies with how familiar the task is.
• 1-2 examples: Noticeable improvement. The model "gets" the format.
• 3-5 examples: The sweet spot for most tasks. Maximum improvement per token.
• 6-10 examples: Diminishing returns. Cost grows, accuracy flattens.
• 10+ examples: Little additional benefit. Consider fine-tuning if you need more.

Experiment: measuring the impact of the number of examples

from openai import OpenAI
import time

client = OpenAI()

# Full bank of 8 representative examples
FULL_BANK = [
    ("Error 500 when loading the page", "TECHNICAL"),
    ("I can't log in", "TECHNICAL"),
    ("My invoice has an incorrect charge", "BILLING"),
    ("Can I switch my monthly plan to annual?", "BILLING"),
    ("I want to delete my account permanently", "ACCOUNT"),
    ("How do I update my contact email?", "ACCOUNT"),
    ("When is the next maintenance window?", "INFORMATION"),
    ("The app closes when I open attachments", "TECHNICAL"),
]

# Ground truth to evaluate accuracy
TEST_CASES = [
    ("The platform doesn't load in Firefox", "TECHNICAL"),
    ("Duplicate charge on my card", "BILLING"),
    ("Delete all my account data", "ACCOUNT"),
    ("Error exporting reports", "TECHNICAL"),
    ("Do you have weekend support?", "INFORMATION"),
]

def classify_with_n_examples(query: str, n: int) -> tuple[str, float]:
    """
    Classifies using n examples from the bank.
    Returns: (classification, latency_ms)
    """
    examples = FULL_BANK[:n]
    
    examples_text = "\n".join([
        f"Input: {inp}\nOutput: {out}" for inp, out in examples
    ])
    
    prompt = f"""Classify support queries. Categories: TECHNICAL, BILLING, ACCOUNT, INFORMATION.

Examples:
{examples_text}

Classify (the category only):
Input: {query}
Output:"""
    
    start = time.time()
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        max_tokens=20
    )
    latency = (time.time() - start) * 1000
    
    return response.choices[0].message.content.strip(), latency

# Benchmark across different N
print(f"{'N':>3} | {'Accuracy':>8} | {'Avg latency':>12} | {'Input tokens':>12}")
print("-" * 45)

for n in [1, 2, 3, 5, 8]:
    correct = 0
    latencies = []
    
    for query, ground_truth in TEST_CASES:
        result, lat = classify_with_n_examples(query, n)
        if result == ground_truth:
            correct += 1
        latencies.append(lat)
    
    accuracy = correct / len(TEST_CASES)
    # Approximate tokens: ~15 tokens per example + overhead
    est_tokens = n * 15 + 50
    
    print(f"{n:>3} | {accuracy:>8.0%} | {sum(latencies)/len(latencies):>12.0f}ms | ~{est_tokens:>10} tkns")

Typical output:

  N | Accuracy |  Avg latency | Input tokens
---------------------------------------------
  1 |      60% |         320ms | ~        65 tkns
  2 |      80% |         340ms | ~        80 tkns
  3 |      80% |         350ms | ~        95 tkns
  5 |     100% |         380ms | ~       125 tkns
  8 |     100% |         410ms | ~       170 tkns

Conclusion: From 3 to 5 examples you already hit the ceiling. From 5 to 8, same accuracy, more tokens.

Selecting for Diversity

Why diversity matters more than quantity

If you have 5 examples but they're all variations of the same input, the model learns a narrow pattern. Five diverse examples teach the model the full range of the category.

# ❌ Redundant examples (all variations of the same type)
REDUNDANT_EXAMPLES = [
    ("Error 404 on the page", "TECHNICAL"),
    ("Error 500 when loading", "TECHNICAL"),
    ("Error opening the app", "TECHNICAL"),
    ("Error screen on startup", "TECHNICAL"),
    ("The system throws an error", "TECHNICAL"),
]

# ✅ Diverse examples (they cover different kinds of TECHNICAL and other categories)
DIVERSE_EXAMPLES = [
    ("Error 500 when loading the page", "TECHNICAL"),   # Server error
    ("The app freezes when uploading files", "TECHNICAL"), # Performance
    ("My invoice has a duplicate charge", "BILLING"),   # Billing
    ("I want to cancel my account", "ACCOUNT"),          # Account
    ("How do I export my data?", "INFORMATION"),         # Info request
]

Diversity metric: To verify your examples are diverse, measure the average pairwise similarity:

def measure_diversity(examples: list[tuple[str, str]]) -> float:
    """
    Computes diversity as 1 - average pairwise similarity.
    Closer to 1.0 = more diverse.
    """
    def jaccard(a: str, b: str) -> float:
        wa = set(a.lower().split())
        wb = set(b.lower().split())
        if not wa or not wb:
            return 0
        return len(wa & wb) / len(wa | wb)
    
    if len(examples) < 2:
        return 1.0
    
    similarities = []
    for i in range(len(examples)):
        for j in range(i + 1, len(examples)):
            sim = jaccard(examples[i][0], examples[j][0])
            similarities.append(sim)
    
    avg_similarity = sum(similarities) / len(similarities)
    return 1 - avg_similarity

# Test
div_redundant = measure_diversity(REDUNDANT_EXAMPLES)
div_diverse = measure_diversity(DIVERSE_EXAMPLES)

print(f"Diversity of redundant examples: {div_redundant:.2f}")  # ~0.60
print(f"Diversity of diverse examples:   {div_diverse:.2f}")    # ~0.90

Rule of thumb: Aim for diversity > 0.80. If it's below that, replace the examples that are most similar to each other.

Selecting for Representativeness

When representativeness matters more than diversity

If your domain has a heavily skewed distribution (e.g. 70% of tickets are TECHNICAL, 15% BILLING, 15% OTHER), your examples should reflect that — or at least not ignore it completely.

# The domain's real distribution
REAL_DISTRIBUTION = {
    "TECHNICAL": 0.65,
    "BILLING": 0.20,
    "ACCOUNT": 0.10,
    "INFORMATION": 0.05,
}

# ❌ Unbalanced examples (they ignore the real distribution)
# 5 examples with 1 per category = over-represents rare categories
FORCED_BALANCED_EXAMPLES = [
    ("Error loading", "TECHNICAL"),         # 1/5 = 20%
    ("Invoice is wrong", "BILLING"),        # 1/5 = 20%
    ("Cancel account", "ACCOUNT"),           # 1/5 = 20%
    ("Support hours?", "INFORMATION"),       # 1/5 = 20%
    ("App crashes", "TECHNICAL"),           # 1/5 = 20%
]

# ✅ Examples that reflect the distribution (with 5 examples, 3-2-1-0 or 3-1-1-0)
REPRESENTATIVE_EXAMPLES = [
    ("Error 500 when saving", "TECHNICAL"),           # TECHNICAL dominates
    ("The app closes on its own", "TECHNICAL"),         # TECHNICAL dominates
    ("App unresponsive after the update", "TECHNICAL"), # TECHNICAL dominates
    ("My invoice has a double charge", "BILLING"),    # Second most common
    ("I want to change my email", "ACCOUNT"),           # Third most common
    # INFORMATION: so infrequent that it can rely on zero-shot
]

Rule of thumb: If all categories have a similar distribution → prioritize diversity. If one category dominates (>60%) → include at least 2 examples of it and make sure there's 1 of each of the others.

The Impact of Ordering

Why order matters

LLMs have a known bias: they tend to imitate the most recent examples (recency bias). If the last example before the real input is category A, the model is more likely to classify the input as A, especially in ambiguous cases.

def classify_with_order(query: str, examples: list[tuple[str, str]]) -> str:
    """Classifies with the examples in the given order."""
    examples_text = "\n".join([f"Input: {i}\nOutput: {o}" for i, o in examples])
    prompt = f"Classify. Categories: TECHNICAL, BILLING, ACCOUNT.\n\n{examples_text}\n\nInput: {query}\nOutput:"
    
    r = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        max_tokens=15
    )
    return r.choices[0].message.content.strip()

# Test: same input, different orders
ambiguous_query = "I have a problem with my account and I can't pay"
# This query could be TECHNICAL, BILLING or ACCOUNT

base_examples = [
    ("Error loading the page", "TECHNICAL"),
    ("My invoice has an error", "BILLING"),
    ("I want to change my personal details", "ACCOUNT"),
]

# Order 1: ends with ACCOUNT
order_1 = base_examples.copy()

# Order 2: ends with BILLING
order_2 = [base_examples[0], base_examples[2], base_examples[1]]

# Order 3: ends with TECHNICAL
order_3 = [base_examples[1], base_examples[2], base_examples[0]]

print(f"Query: '{ambiguous_query}'\n")
print(f"Order 1 (last=ACCOUNT):   {classify_with_order(ambiguous_query, order_1)}")
print(f"Order 2 (last=BILLING):   {classify_with_order(ambiguous_query, order_2)}")
print(f"Order 3 (last=TECHNICAL): {classify_with_order(ambiguous_query, order_3)}")

Typical result: The classification can change with the order, especially on ambiguous inputs.

Ordering rules

1. Clear cases first: Start with unambiguous examples that anchor the pattern
2. Edge cases at the end: If you have ambiguous examples, put them last — but before the real input
3. Rotate if there's bias: If you notice the last example skewing the result, rotate the bank between calls
4. Consistency across categories: If you have 2+ examples of one category, don't place them consecutively

# Recommended order for 5 examples
OPTIMAL_ORDER = [
    ("Error 500 when loading", "TECHNICAL"),      # 1. Clear, first type
    ("Wrong invoice this month", "BILLING"),      # 2. Clear, second type
    ("Cancel my subscription today", "ACCOUNT"),   # 3. Clear, third type
    ("The app crashes on iOS 17", "TECHNICAL"),    # 4. Second TECHNICAL (most common)
    ("How does module X work?", "INFORMATION"),    # 5. Last: least common
    # The real input comes here
]

# ❌ Problematic order
BAD_ORDER = [
    ("Something isn't working", "OTHER"),      # Starts with a vague case
    ("Error 500 when loading", "TECHNICAL"),
    ("Not sure, I think it's an error", "TECHNICAL"),  # Weak example
    ("Wrong invoice", "BILLING"),
    ("I can't get in or pay", "TECHNICAL"),    # Ambiguous at the end → skews
    # The real input comes here
]

Negative Examples

When to use them

Negative examples explicitly show what NOT to do. They're especially useful for:

• Edge cases the model consistently misclassifies
• Categories that look similar but aren't
• Preventing over-classification into the most common category

from openai import OpenAI

client = OpenAI()

def classify_with_negatives(query: str) -> str:
    """
    Uses negative examples to clarify the boundary between similar categories.
    """
    prompt = """
Classify the query into: TECHNICAL, BILLING, ACCOUNT, INFORMATION.

## Positive examples (what to do)
Input: "The reports module doesn't load" → TECHNICAL
Input: "My October invoice has a double charge" → BILLING
Input: "I want to close my account" → ACCOUNT
Input: "Do you have support in Spanish?" → INFORMATION

## Negative examples (common mistakes to avoid)
Input: "I can't get in and I can't see my invoice either"
  DO NOT CLASSIFY as: TECHNICAL (even though it mentions access)
  CLASSIFY as: BILLING (the main problem is the invoice)

Input: "How do I change the plan to see if it comes out cheaper?"
  DO NOT CLASSIFY as: INFORMATION (even though it asks a question)
  CLASSIFY as: BILLING (the intent is about prices/plans)

## Classify:
"""
    r = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": prompt},
            {"role": "user", "content": query}
        ],
        temperature=0,
        max_tokens=15
    )
    return r.choices[0].message.content.strip()

# Test with cases that usually get confused
ambiguous_cases = [
    "I can't see my latest invoice and the system says I owe money",
    "How much would it cost me to move to the Pro plan?",
    "Error paying with my saved card"
]

for case in ambiguous_cases:
    result = classify_with_negatives(case)
    print(f"'{case[:50]}' → {result}")

Rule: Add a negative example when you see the model consistently misclassifying the same type of input. One well-placed negative example can eliminate a systematic error.

ExampleBank: A Reusable Class

from typing import Optional
import json

class ExampleBank:
    """
    Example bank with dynamic selection and persistence.
    """
    
    def __init__(self, categories: list[str], max_per_prompt: int = 5):
        self.categories = categories
        self.max_per_prompt = max_per_prompt
        self.examples: list[dict] = []
    
    def add(self, input_text: str, output: str, notes: str = "") -> None:
        """Adds an example to the bank, with validation."""
        if output not in self.categories:
            raise ValueError(f"Category '{output}' is not valid. Valid ones: {self.categories}")
        if not input_text.strip():
            raise ValueError("Input cannot be empty")
        
        self.examples.append({
            "input": input_text.strip(),
            "output": output,
            "notes": notes,
            "negative": notes.startswith("NO:")  # Marks negative examples
        })
    
    def select(self, 
               new_input: str,
               k: Optional[int] = None,
               balance_categories: bool = True) -> list[dict]:
        """
        Selects the best examples for the new input.
        
        Args:
            new_input: The input that is going to be classified
            k: Number of examples. Default: self.max_per_prompt
            balance_categories: If True, includes at least 1 example per category if there's room
        """
        k = k or self.max_per_prompt
        if not self.examples:
            return []
        
        # Compute similarity
        def jaccard(a: str, b: str) -> float:
            wa = set(a.lower().split())
            wb = set(b.lower().split())
            if not wa or not wb:
                return 0
            return len(wa & wb) / len(wa | wb)
        
        # Score = similarity (more similar = more relevant)
        scored = [(jaccard(ex["input"], new_input), ex) for ex in self.examples]
        scored.sort(key=lambda x: x[0], reverse=True)
        
        if not balance_categories:
            return [ex for _, ex in scored[:k]]
        
        # Guarantee at least 1 example per category
        selected = []
        covered_cats = set()
        
        # First pass: 1 example per category (the most similar one)
        for _, ex in scored:
            if ex["output"] not in covered_cats and len(covered_cats) < len(self.categories):
                selected.append(ex)
                covered_cats.add(ex["output"])
        
        # Second pass: fill up to k with the most similar ones
        for _, ex in scored:
            if len(selected) >= k:
                break
            if ex not in selected:
                selected.append(ex)
        
        return selected[:k]
    
    def build_prompt(self, task: str, new_input: str) -> str:
        """Builds the complete few-shot prompt."""
        examples = self.select(new_input)
        
        lines = [task, ""]
        for i, ex in enumerate(examples, 1):
            if ex.get("notes"):
                lines.append(f"# {ex['notes']}")
            lines.append(f"Input: {ex['input']}")
            lines.append(f"Output: {ex['output']}")
            lines.append("")
        
        lines.append(f"Input: {new_input}")
        lines.append("Output:")
        
        return "\n".join(lines)
    
    def save(self, path: str) -> None:
        with open(path, "w", encoding="utf-8") as f:
            json.dump({"categories": self.categories, "examples": self.examples}, f, 
                     ensure_ascii=False, indent=2)
    
    @classmethod
    def load(cls, path: str) -> "ExampleBank":
        with open(path, encoding="utf-8") as f:
            data = json.load(f)
        bank = cls(data["categories"])
        bank.examples = data["examples"]
        return bank

# Complete usage
bank = ExampleBank(
    categories=["TECHNICAL", "BILLING", "ACCOUNT", "INFORMATION"],
    max_per_prompt=4
)

bank.add("Error 500 when loading the dashboard", "TECHNICAL")
bank.add("My January invoice has a double charge", "BILLING")
bank.add("I want to cancel my subscription", "ACCOUNT")
bank.add("Do you have an API available for developers?", "INFORMATION")
bank.add("The app crashes when I open notifications", "TECHNICAL")

# Build the prompt for a new input
new_query = "The reports module hasn't responded since yesterday"
prompt = bank.build_prompt(
    "Classify the support query. The category only.",
    new_query
)
print(prompt)

Troubleshooting

Problem 1: More examples make the result worse

Cause: Redundant or contradictory examples dilute the signal. The model averages contradictory patterns.

Fix: Cut back to 3-4 well-chosen examples. Measure diversity with measure_diversity() — it should be > 0.80. Drop examples with diversity < 0.3 against another one in the bank.

Problem 2: The model ignores the examples and uses its own format

Cause: There's too much text between the examples and the final input, or the examples' format isn't explicit enough.

Fix:

# ✅ Clean, consistent format
# Input: [text]
# Output: [category]

# Add at the end of the prompt:
"Follow the format exactly: a single word in uppercase."

Problem 3: Bias toward one category

Cause: Too many examples of one category, or the last example is always from the same one.

Fix: Balance the example distribution. If one category has more examples, use the ExampleBank's balance_categories=True parameter.

Problem 4: Edge cases fail consistently

Cause: There are no examples covering those cases.

Fix: Add negative examples for the input types that fail. One well-designed negative example can eliminate a systematic error.

Problem 5: The example bank grows too big

Cause: Examples keep getting added without cleaning out the redundant ones.

Fix: Every time the bank passes 20-30 examples, run a dedup and remove the ones with diversity < 0.3 against another example in the same category.

Exercises

Exercise 1: Pick the best 4 out of 10 (Easy)

Given this bank of 10 examples for TECHNICAL/BILLING/ACCOUNT classification, pick the best 4 considering diversity and representativeness:

1. "Error 404" → TECHNICAL
2. "Error 500" → TECHNICAL
3. "Error loading" → TECHNICAL
4. "App crashes" → TECHNICAL
5. "My invoice has an error" → BILLING
6. "Incorrect charge" → BILLING
7. "Cancel account" → ACCOUNT
8. "Close down the service" → ACCOUNT
9. "Login doesn't work" → TECHNICAL
10. "Change payment plan" → BILLING

Exercise 2: Order the examples (Easy)

Order these 4 examples to maximize clarity and minimize recency bias:

A. "Something isn't working right" → TECHNICAL (ambiguous)
B. "Error processing my payment" → BILLING (clear)
C. "I want to export my data before cancelling" → ACCOUNT (mixed)
D. "The app won't open on my new phone" → TECHNICAL (clear)

Exercise 3: Design a negative example (Medium)

The model consistently classifies "Can I return the product if I already used it?" as RETURN instead of POLICY. Design a negative example that fixes this.

# Negative example to clarify POLICY vs RETURN
Input: "Can I return a product I've already used for a week?"
  It is NOT: RETURN (they aren't starting an actual return)
  It IS: POLICY (they're asking about the rules, not executing an action)

Compare with the positives:
Input: "I want to return the product I bought yesterday" → RETURN (a real action)
Input: "What's the return policy?" → POLICY (a question about the rules)
Input: "Can I return it if I already used it?" → POLICY (it's a policy question)

Exercise 4: Implement ExampleBank with quality validation (Hard)

Extend the ExampleBank class so that when you add an example, it validates that it isn't too similar to an existing one (minimum diversity of 30%).

class ValidatedExampleBank(ExampleBank):
    """ExampleBank with quality validation when adding examples."""
    
    def __init__(self, categories: list[str], max_per_prompt: int = 5, 
                 min_diversity: float = 0.3):
        super().__init__(categories, max_per_prompt)
        self.min_diversity = min_diversity
    
    def _jaccard(self, a: str, b: str) -> float:
        wa = set(a.lower().split())
        wb = set(b.lower().split())
        if not wa or not wb:
            return 0
        return len(wa & wb) / len(wa | wb)
    
    def add(self, input_text: str, output: str, notes: str = "", 
            force: bool = False) -> None:
        """
        Adds an example with diversity validation.
        
        Args:
            force: If True, adds it even if it violates the minimum diversity
        """
        if not force:
            for ex in self.examples:
                sim = self._jaccard(input_text, ex["input"])
                if sim > (1 - self.min_diversity):  # Too similar to an existing one
                    raise ValueError(
                        f"Example too similar (sim={sim:.2f}) to: '{ex['input']}'. "
                        f"Use force=True if you want to add it anyway."
                    )
        
        super().add(input_text, output, notes)

# Test
bank = ValidatedExampleBank(["TECHNICAL", "BILLING"], min_diversity=0.3)
bank.add("Error loading the page", "TECHNICAL")

try:
    bank.add("Error loading the system", "TECHNICAL")  # Too similar
except ValueError as e:
    print(f"Rejected: {e}")

bank.add("The app crashes on iOS", "TECHNICAL")  # Different — accepted
print(f"Bank: {len(bank.examples)} examples")

Summary

In this capsule you learned:

• Sweet spot: 3-5 examples for most tasks. More than 5 = diminishing returns
• Diversity: Examples covering different kinds of inputs > examples repeating the same pattern. Measure it with diversity > 0.80
• Representativeness: If one category dominates the domain, include more examples of it (don't force 1-per-category)
• Order: Clear cases first, edge cases at the end. Rotate if you detect recency bias on ambiguous inputs
• Negatives: To eliminate systematic errors in similar categories or specific edge cases
• ExampleBank: A reusable class with dynamic selection, balancing, and persistence — the basis of the final project

Next capsule: Advanced example engineering — how to generate synthetic examples with an LLM, dynamic few-shot with embeddings, and per-domain banks.

Further resources

1. Language Models are Few-Shot Learners (Brown et al., 2020) — The original GPT-3 paper, with analysis of how the number of examples and their selection matter
2. What Makes Good In-Context Examples for GPT-3? — An empirical analysis of which characteristics make few-shot examples better
3. Rethinking the Role of Demonstrations (Min et al., 2022) — Research on the impact of ordering and label distribution in few-shot
4. OpenAI Few-Shot Best Practices — The official guide, with practical selection recommendations
5. Anthropic: Using Examples Effectively — Claude's perspective on when and how to use few-shot examples