5. Verification Patterns in CoT

Overview

CoT can produce reasoning that sounds perfectly logical but arrives at incorrect conclusions. This capsule covers four verification patterns for catching and fixing those errors: self-verification, backward verification, cross-checking against constraints, and confidence scoring. We also implement a two-pass verification system where a second LLM acts as an independent auditor.

Estimated time: 75-90 minutes

The Problem: Correct CoT ≠ Correct Answer

Consider this real CoT failure:

Problem: A room has 3 tables. Each table has 4 chairs.
         5 people walk in. How many chairs are left empty?

Model's CoT:
- Total chairs: 3 tables × 4 chairs = 12 chairs
- People walking in: 5
- Occupied chairs: 5
- Empty chairs: 12 - 5 = 7

Answer: 7 ✓ (CORRECT)

Now the same model with a slightly more complex problem:

Problem: A room has 3 tables. Each table has 4 chairs.
         If 7 of the chairs already have objects on them and 5 people walk in,
         how many chairs are available for the people?

Model's CoT:
- Total chairs: 3 × 4 = 12
- Chairs with objects: 7
- Available chairs: 12 - 7 = 5
- People: 5
- All the people can sit down.
- Chairs available for people: 5

Answer: 5 (INCORRECT - after the 5 people sit down,
           0 chairs would be available, but if the question is
           "how many are there before they walk in", the answer is 5.
           The ambiguity went undetected.)

Verification patterns catch these failures.

Pattern 1: Self-Verification

The model reviews its own work at the end of the reasoning.

Basic Implementation

from openai import OpenAI

client = OpenAI()

SELF_VERIFY_COT = """Solve the problem and then verify your answer.

PART 1 - SOLUTION:
Think step by step. Show every operation.

PART 2 - VERIFICATION:
After you get your answer:
- Is every mathematical step correct? (check the operations)
- Does the answer make sense in context? (is it a reasonable number?)
- Did you answer exactly what was asked? (no more, no less)
- If you find any error, fix it here.

PART 3 - FINAL ANSWER:
VERIFIED ANSWER: [your corrected answer if it applied, or the original if it was correct]

Problem: {problem}
"""


def solve_with_self_verification(problem: str) -> dict:
    """
    Solves a problem with built-in self-verification.

    Returns:
        dict with 'reasoning', 'verification', 'final_answer'
    """
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "user",
                "content": SELF_VERIFY_COT.format(problem=problem)
            }
        ],
        temperature=0,
        max_tokens=900
    )
    output = response.choices[0].message.content

    # Parse the sections
    import re
    parts = {
        "reasoning": "",
        "verification": "",
        "final_answer": ""
    }

    # Extract the verified answer
    answer_match = re.search(
        r'VERIFIED ANSWER:\s*(.+?)(?:\n|$)',
        output, re.IGNORECASE
    )
    if answer_match:
        parts["final_answer"] = answer_match.group(1).strip()

    parts["full_output"] = output

    return parts

Explicit Self-Verification with a Checklist

SELF_VERIFY_CHECKLIST = """Solve the problem. Then complete the verification checklist.

=== SOLUTION ===
{problem}

Reasoning:

=== VERIFICATION CHECKLIST ===
After solving, answer each point with ✓ or ✗ and a short note:

[ ] 1. Did you interpret the question correctly?
[ ] 2. Did you use all the data provided?
[ ] 3. Are the mathematical operations correct?
[ ] 4. Are the units consistent (km, kg, $, etc.)?
[ ] 5. Is the magnitude of the answer reasonable?
[ ] 6. Did you answer exactly what was asked?

=== VERIFIED FINAL ANSWER ===
[If every check is ✓: confirm the answer. If any is ✗: correct it and give a new answer]
"""

Limitations of Self-Verification

# ⚠️ The model tends to confirm itself even when it's wrong
# This is called "sycophantic verification"

def measure_self_correction_rate(problems_with_induced_error: list) -> dict:
    """
    Measures how often the model corrects its own errors.

    Note: In practice, models self-confirm ~70-80% of the time
    even when the reasoning contains an error.
    """
    self_corrected = 0
    incorrectly_confirmed = 0

    for problem, correct_answer in problems_with_induced_error:
        result = solve_with_self_verification(problem)

        if correct_answer in str(result["final_answer"]):
            self_corrected += 1
        else:
            incorrectly_confirmed += 1

    total = len(problems_with_induced_error)
    return {
        "self_correction_rate": self_corrected / total,
        "error_confirmation_rate": incorrectly_confirmed / total,
        "conclusion": "Self-verification is not enough for critical problems"
    }

Pattern 2: Backward Verification

Once it has the answer, the model "substitutes it backwards" to check that it solves the original problem.

Implementation: Mathematical Backward Check

BACKWARD_VERIFY_COT = """Solve the problem step by step.

FORWARD STEP (Solution):
Compute the answer.

BACKWARD STEP (Verification):
Take your answer and apply it to the original problem:
- Substitute the value you got into the problem's conditions
- Are ALL the original conditions satisfied?
- If yes: VERIFIED ✓
- If no: Identify the error and solve again

FINAL ANSWER: [verified answer]

Problem: {problem}
"""


def solve_with_backward_check(problem: str) -> dict:
    """Solves with backward verification."""
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": BACKWARD_VERIFY_COT.format(problem=problem)}],
        temperature=0,
        max_tokens=800
    )
    output = response.choices[0].message.content

    verified = "VERIFIED ✓" in output or "VERIFIED" in output.upper()

    import re
    match = re.search(r'FINAL ANSWER:\s*(.+?)(?:\n|$)', output, re.IGNORECASE)

    return {
        "output": output,
        "verified": verified,
        "answer": match.group(1).strip() if match else None
    }


# Examples where the backward check catches errors
BACKWARD_TEST_PROBLEMS = [
    # Problem 1: Linear equation
    "Solve: 3x + 7 = 22. Value of x:",
    # Backward: is 3(5)+7=22? 15+7=22 ✓

    # Problem 2: Inverse percentage
    "A product with a 20% discount costs $80. What was the original price?",
    # Forward (common error): 80 + 20%*80 = 96 ✗
    # Backward: is 96 - 20%*96 = 80? 96-19.2=76.8 ≠ 80 → ERROR
    # Correct: 80/0.80 = $100. Backward: 100-20%*100=80 ✓

    # Problem 3: Work rate
    "If A does a job in 6 hours and B in 4 hours, how long do they take together?",
    # Correct: 1/6 + 1/4 = 5/12 → 12/5 = 2.4 hours
    # Backward: in 2.4h, A does 2.4/6=0.4 of the job, B does 2.4/4=0.6. 0.4+0.6=1 ✓
]

if __name__ == "__main__":
    for problem in BACKWARD_TEST_PROBLEMS:
        result = solve_with_backward_check(problem)
        print(f"\nProblem: {problem[:60]}...")
        print(f"Verified: {result['verified']}")
        print(f"Answer: {result['answer']}")

Backward Check for Logic

BACKWARD_LOGIC_COT = """Analyze the logical argument.

FORWARD ANALYSIS:
Identify the premises and analyze whether the conclusion follows.

BACKWARD VERIFICATION:
For the argument, look for a COUNTEREXAMPLE:
- Assume the premises are true AND the conclusion is false
- Is that scenario possible?
- If it is possible → INVALID argument (the conclusion doesn't necessarily follow)
- If it is impossible → VALID argument

VERDICT: VALID or INVALID

Argument: {argument}
"""

Pattern 3: Cross-Checking Against Constraints

For problems with multiple conditions, verify each one explicitly.

Implementation: Constraint Check

from dataclasses import dataclass

@dataclass
class ConstraintCheck:
    description: str
    satisfied: bool | None = None
    note: str = ""


CONSTRAINT_VERIFY_COT = """Solve the problem and then verify EACH constraint.

Problem: {problem}
Constraints:
{constraints}

=== SOLUTION ===
[Your solution here]

=== CONSTRAINT VERIFICATION ===
For each constraint, evaluate whether your solution satisfies it:
{checks_template}

=== RESULT ===
- Total constraints: {n_constraints}
- Satisfied: ?/
- Valid solution? YES/NO

If any constraint fails, adjust the solution and verify again.
"""


def solve_with_constraint_check(
    problem: str,
    constraints: list[str]
) -> dict:
    """
    Solves and verifies each constraint explicitly.

    Args:
        problem: The problem to solve
        constraints: List of constraints the solution must satisfy

    Returns:
        dict with the solution, per-constraint checks, and global validity
    """
    constraints_text = "\n".join(f"- R{i+1}: {r}" for i, r in enumerate(constraints))
    checks_template = "\n".join(
        f"R{i+1} ({r[:40]}...): ✓/✗" for i, r in enumerate(constraints)
    )

    prompt = CONSTRAINT_VERIFY_COT.format(
        problem=problem,
        constraints=constraints_text,
        checks_template=checks_template,
        n_constraints=len(constraints)
    )

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        max_tokens=1000
    )
    output = response.choices[0].message.content

    # Count the checks
    import re
    checks_passed = len(re.findall(r'✓', output))
    checks_failed = len(re.findall(r'✗', output))

    return {
        "full_output": output,
        "constraints_met": checks_passed,
        "constraints_failed": checks_failed,
        "valid_solution": checks_failed == 0 and checks_passed > 0
    }


# Example: Database design with constraints
db_problem = "Design the table schema for an e-commerce order system"

db_constraints = [
    "It must store user, products, quantities and prices",
    "An order can have multiple products",
    "It must record the date and time of the order",
    "It must allow tracking the status (pending, shipped, delivered)",
    "It must store the shipping address",
    "The relations must be in 3NF (third normal form)",
]

if __name__ == "__main__":
    result = solve_with_constraint_check(db_problem, db_constraints)
    print(result["full_output"])
    print(f"\n✓ Satisfied: {result['constraints_met']}")
    print(f"✗ Failed: {result['constraints_failed']}")
    print(f"Valid solution: {result['valid_solution']}")

Pattern 4: Confidence Scoring

The model assigns a confidence score to its answer and explains its level of certainty.

Implementation: Structured Confidence Score

CONFIDENCE_COT = """Solve the problem. At the end, assign a confidence score and explain why.

=== SOLUTION ===
Think step by step.

=== CONFIDENCE ===
Rate your answer on these dimensions:

1. PROCESS CERTAINTY (0-1): Are you sure about the steps you followed?
2. VERIFIABILITY (0-1): Were you able to verify the answer? How?
3. PROBLEM AMBIGUITY (0-1): Was the problem clear? (1=very clear, 0=very ambiguous)
4. DOMAIN KNOWLEDGE (0-1): Are you certain about the knowledge you applied?

FINAL SCORE: Average of the 4 dimensions = [X.XX]

Interpretation:
- 0.9-1.0: Very high confidence, answer almost certainly correct
- 0.7-0.9: High confidence, probably correct but verify
- 0.5-0.7: Medium confidence, there may be errors
- < 0.5: Low confidence, verify some other way

FINAL ANSWER: [answer]
CONFIDENCE: [score] - [interpretation]

Problem: {problem}
"""


def solve_with_confidence(problem: str) -> dict:
    """Solves with a detailed confidence score."""
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": CONFIDENCE_COT.format(problem=problem)}],
        temperature=0,
        max_tokens=800
    )
    output = response.choices[0].message.content

    import re

    # Extract the confidence score
    score_match = re.search(r'FINAL SCORE:.*?=\s*([\d.]+)', output, re.IGNORECASE)
    confidence_match = re.search(r'CONFIDENCE:\s*([\d.]+)', output, re.IGNORECASE)
    answer_match = re.search(r'FINAL ANSWER:\s*(.+?)(?:\n|$)', output, re.IGNORECASE)

    score = None
    if score_match:
        try:
            score = float(score_match.group(1))
        except ValueError:
            pass
    elif confidence_match:
        try:
            score = float(confidence_match.group(1))
        except ValueError:
            pass

    return {
        "output": output,
        "answer": answer_match.group(1).strip() if answer_match else None,
        "confidence_score": score,
        "confidence_level": (
            "very_high" if score and score >= 0.9 else
            "high" if score and score >= 0.7 else
            "medium" if score and score >= 0.5 else
            "low" if score else "unknown"
        )
    }


# Problems with different difficulty levels
CONFIDENCE_TEST_PROBLEMS = [
    "What is 15 × 20?",  # We expect very high confidence
    "If the economy grows 3% this year, how much will it grow next year?",  # We expect low confidence
    "Is p=NP?",  # We expect low confidence (open problem)
    "In a right triangle with legs 3 and 4, how long is the hypotenuse?",  # High confidence
]

if __name__ == "__main__":
    for problem in CONFIDENCE_TEST_PROBLEMS:
        result = solve_with_confidence(problem)
        print(f"\nProblem: {problem[:60]}")
        print(f"Answer: {result['answer']}")
        print(f"Confidence: {result['confidence_score']} ({result['confidence_level']})")

Pattern 5: Two-Pass Verification (The Most Robust)

A second LLM acts as an independent verifier. This is the most robust pattern because it removes the self-confirmation bias.

Implementation: Two-Pass with Different Roles

def two_pass_verification(
    problem: str,
    solver_temperature: float = 0.0,
    verifier_temperature: float = 0.0
) -> dict:
    """
    Two-Pass Verification:
    - Pass 1: The "Solver" solves the problem
    - Pass 2: The "Verifier" checks the solution as an independent auditor

    The Verifier gets the original problem + the proposed solution,
    but NOT the Solver's reasoning (to avoid anchoring bias).

    Args:
        problem: The problem to solve
        solver_temperature: Temperature for the solver (0 = deterministic)
        verifier_temperature: Temperature for the verifier

    Returns:
        dict with the solution, the verification, and the final verdict
    """
    # PASS 1: Solver solves
    solver_prompt = f"""You are an expert solver. Solve the following problem step by step.
At the end write EXACTLY: "SOLUTION: [your answer]"

Problem: {problem}"""

    solver_response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": solver_prompt}],
        temperature=solver_temperature,
        max_tokens=700
    )
    full_solution = solver_response.choices[0].message.content

    # Extract only the final answer to pass to the verifier
    import re
    match = re.search(r'SOLUTION:\s*(.+?)(?:\n|$)', full_solution, re.IGNORECASE)
    proposed_answer = match.group(1).strip() if match else full_solution[-100:]

    # PASS 2: Verifier audits (gets the problem + proposed answer, NOT the reasoning)
    verifier_prompt = f"""You are an auditor of mathematical/logical solutions. Your job is to verify whether a proposed solution is correct.

INSTRUCTIONS:
- Solve the problem INDEPENDENTLY (do not trust the proposed solution)
- Compare your solution with the proposed one
- Give your verdict: CORRECT or INCORRECT
- If it is incorrect, give the correct answer

Original problem: {problem}

Proposed solution: {proposed_answer}

VERIFICATION:"""

    verifier_response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": verifier_prompt}],
        temperature=verifier_temperature,
        max_tokens=500
    )
    verification = verifier_response.choices[0].message.content

    # Determine the verdict
    verification_upper = verification.upper()
    if "CORRECT" in verification_upper and "INCORRECT" not in verification_upper:
        verdict = "APPROVED"
    elif "INCORRECT" in verification_upper:
        verdict = "REJECTED"
    else:
        verdict = "UNDETERMINED"

    # Extract the corrected solution if it was rejected
    final_solution = proposed_answer
    if verdict == "REJECTED":
        correction_match = re.search(
            r'(?:correct answer|correct)[:\s]+(.+?)(?:\n|$)',
            verification, re.IGNORECASE
        )
        if correction_match:
            final_solution = correction_match.group(1).strip()

    return {
        "problem": problem,
        "solver_solution": full_solution,
        "proposed_answer": proposed_answer,
        "auditor_verification": verification,
        "verdict": verdict,
        "final_answer": final_solution,
        "total_tokens": (
            solver_response.usage.total_tokens +
            verifier_response.usage.total_tokens
        )
    }


# Benchmark: Measure the improvement of two-pass vs. single-pass
def benchmark_verification(
    problems: list[tuple[str, str]]  # (problem, correct answer)
) -> dict:
    """Compares single-pass vs. two-pass on accuracy."""
    single_pass_ok = 0
    two_pass_ok = 0

    for problem, correct_answer in problems:
        # Single pass
        r_single = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"{problem}\n\nThink step by step."}],
            temperature=0, max_tokens=500
        )
        if correct_answer in r_single.choices[0].message.content:
            single_pass_ok += 1

        # Two pass
        r_two = two_pass_verification(problem)
        if correct_answer in r_two["final_answer"]:
            two_pass_ok += 1

    n = len(problems)
    return {
        "single_pass_accuracy": single_pass_ok / n,
        "two_pass_accuracy": two_pass_ok / n,
        "improvement": (two_pass_ok - single_pass_ok) / n
    }

Pattern 6: Ensemble Verification

Combines multiple verification strategies for maximum confidence.

def ensemble_verification(
    problem: str,
    n_self_consistency: int = 3
) -> dict:
    """
    Ensemble of verification techniques:
    1. Self-Consistency (N independent solutions)
    2. Backward verification of the majority answer
    3. Final confidence score

    Args:
        problem: The problem to solve
        n_self_consistency: Number of solutions for self-consistency

    Returns:
        dict with a high-confidence answer and verification metadata
    """
    from collections import Counter

    # Step 1: Self-Consistency - generate N solutions
    solutions = []
    for _ in range(n_self_consistency):
        r = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"{problem}\n\nThink step by step."}],
            temperature=0.7,  # Moderate temperature for diversity
            max_tokens=500
        )
        output = r.choices[0].message.content

        # Extract the final number
        import re
        match = re.search(r'(?:answer|therefore|total|=)\s*:?\s*\$?([\d,.]+)', output.lower())
        answer = match.group(1).replace(',', '') if match else None
        solutions.append(answer)

    # Vote for the majority answer
    votes = Counter([s for s in solutions if s])
    if not votes:
        return {"error": "Could not extract any answer"}

    majority_answer, n_votes = votes.most_common(1)[0]
    sc_confidence = n_votes / n_self_consistency

    # Step 2: Backward verification of the majority answer
    backward_prompt = f"""
Problem: {problem}
Proposed answer: {majority_answer}

Verify that this answer is correct by substituting it into the problem.
Are all the conditions satisfied? Reply VERIFIED ✓ or ERROR ✗.
"""
    r_backward = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": backward_prompt}],
        temperature=0,
        max_tokens=300
    )
    backward_verification = r_backward.choices[0].message.content
    backward_ok = "VERIFIED" in backward_verification.upper()

    # Compute the final confidence
    final_confidence = sc_confidence * (1.0 if backward_ok else 0.5)

    return {
        "final_answer": majority_answer,
        "vote_distribution": dict(votes),
        "self_consistency_confidence": sc_confidence,
        "backward_verified": backward_ok,
        "final_confidence": final_confidence,
        "confidence_level": (
            "VERY HIGH" if final_confidence >= 0.8 else
            "HIGH" if final_confidence >= 0.6 else
            "MEDIUM" if final_confidence >= 0.4 else
            "LOW"
        )
    }

Comparison Table: Verification Patterns

Troubleshooting

Problem 1: The model confirms itself incorrectly

# ❌ The model says "VERIFIED ✓" even when the answer is wrong
# This happens ~70% of the time in self-verification

# ✅ Fix 1: Use a second model as an auditor (two-pass)
result = two_pass_verification(problem)

# ✅ Fix 2: Explicitly ask it to find errors, not to confirm
CRITICAL_VERIFICATION = """Your job is to find errors, not to confirm that it's fine.
Assume there COULD be an error and look for it actively.

Problem: {problem}
Proposed solution: {solution}

List every possible problem with this solution.
If you find none, write "No errors detected" and explain why.
"""

Problem 2: Verification adds too much cost

# ✅ Use verification only for critical calls
def solve_with_selective_verification(
    problem: str,
    complexity_threshold: int = 3
) -> dict:
    """Only uses two-pass if the problem looks complex."""

    # Quick complexity estimate
    complexity_words = ['if...then', 'multiple', 'all', 'some',
                        'except', 'unless', 'condition', 'constraint']
    complexity = sum(1 for w in complexity_words if w in problem.lower())

    if complexity >= complexity_threshold:
        return two_pass_verification(problem)
    else:
        # Simple single pass
        r = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"{problem}\n\nThink step by step."}],
            temperature=0, max_tokens=500
        )
        return {"final_answer": r.choices[0].message.content, "type": "single_pass"}

Problem 3: The model's confidence is badly calibrated

# ❌ The model reports 0.9 confidence on an incorrect answer
# LLMs tend to be overconfident

# ✅ Calibrate confidence against real data
def calibrate_confidence(
    problems_with_answers: list[tuple[str, str]],
    n_samples: int = 20
) -> dict:
    """
    Measures whether the reported confidence score is calibrated.
    A well-calibrated model should hit ~90% accuracy on the cases
    where it reports 0.9 confidence.
    """
    by_level = {"high": [], "medium": [], "low": []}

    for problem, correct_answer in problems_with_answers[:n_samples]:
        result = solve_with_confidence(problem)
        level = result["confidence_level"]

        if level in ["very_high", "high"]:
            is_correct = correct_answer in str(result["answer"] or "")
            by_level["high"].append(is_correct)
        elif level == "medium":
            is_correct = correct_answer in str(result["answer"] or "")
            by_level["medium"].append(is_correct)
        else:
            is_correct = correct_answer in str(result["answer"] or "")
            by_level["low"].append(is_correct)

    calibration = {}
    for level, results in by_level.items():
        if results:
            calibration[level] = sum(results) / len(results)

    return calibration

Exercises

Exercise 1: Implement backward verification for equations

For the problem "Solve 2x + 5 = 13", implement backward verification that confirms the value of x you found satisfies the equation.

from openai import OpenAI

client = OpenAI()

problem = "Solve the equation: 2x + 5 = 13"

# Step 1: Solve forward
r_forward = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": f"{problem}\nThink step by step. At the end write: x = [number]"}],
    temperature=0, max_tokens=300
)
forward_output = r_forward.choices[0].message.content

import re
match = re.search(r'x\s*=\s*([\d.]+)', forward_output)
proposed_x = match.group(1) if match else "?"

print(f"Forward reasoning:\n{forward_output}")
print(f"\nProposed value: x = {proposed_x}")

# Step 2: Backward verification
backward_prompt = f"""
Original problem: 2x + 5 = 13
Proposed solution: x = {proposed_x}

Verify by substituting x into the equation:
Substitute: 2({proposed_x}) + 5 = ?
Does it equal 13?

If yes: VERIFIED ✓
If no: ERROR ✗, the correct value is x = [correction]
"""

r_backward = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": backward_prompt}],
    temperature=0, max_tokens=200
)
print(f"\nBackward verification:\n{r_backward.choices[0].message.content}")
# x=4: 2(4)+5=13 ✓

Exercise 2: Confidence calibration

Run 10 math problems through solve_with_confidence. For each one, record the confidence score and whether the answer was correct. Is the model well calibrated?

from openai import OpenAI

client = OpenAI()

test_problems = [
    ("5 + 3 = ?", "8"),
    ("17 × 23 = ?", "391"),
    ("Square root of 144 = ?", "12"),
    ("30% of 250 = ?", "75"),
    ("Integral of x² = ?", "x³/3"),
    ("1/3 + 1/4 = ?", "7/12"),
    ("2^10 = ?", "1024"),
    ("What is Pi?", "3.14"),
    ("Natural logarithm of e = ?", "1"),
    ("Sin(90°) = ?", "1"),
]

results = []
for problem, correct_answer in test_problems:
    r = solve_with_confidence(problem)
    is_correct = correct_answer in str(r["answer"] or "")

    results.append({
        "problem": problem,
        "answer": r["answer"],
        "correct_answer": correct_answer,
        "is_correct": is_correct,
        "confidence": r["confidence_score"],
        "level": r["confidence_level"]
    })

    print(f"{'✓' if is_correct else '✗'} {problem[:30]:30} | Confidence: {r['confidence_score']} | {'OK' if is_correct else 'ERROR'}")

# Compute the calibration
high_confidence = [r for r in results if r["level"] in ["very_high", "high"]]
high_confidence_accuracy = sum(r["is_correct"] for r in high_confidence) / len(high_confidence) if high_confidence else 0

print(f"\nHigh confidence → Real accuracy: {high_confidence_accuracy:.0%}")
print("Model is well calibrated if high_confidence_accuracy ≈ 0.90")

Exercise 3: Two-pass vs. single-pass on trick problems

Implement and compare both approaches on problems designed to fool the model.

# Common "trick" problems for LLMs
TRICK_PROBLEMS = [
    ("A doctor has 10 years of practice. She started at age 25. How old is she now?", "35"),
    ("There are 23 students in a class. 1/3 are boys. How many are girls? (answer as a whole number)", "15"),  # 23*2/3 ≈ 15.3 → 15
    ("If there are 12 fish in an aquarium and half of them die, how many are left?", "6"),
    ("A father has 3 children. The oldest is 10. The middle one is 8. The sum of the 3 children's ages is 21. How old is the youngest?", "3"),
]

print("Comparing single-pass vs. two-pass on trick problems:\n")

for problem, correct_answer in TRICK_PROBLEMS:
    # Single pass
    r_single = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"{problem}\n\nThink step by step."}],
        temperature=0, max_tokens=300
    )
    single_ok = correct_answer in r_single.choices[0].message.content

    # Two pass
    r_two = two_pass_verification(problem)
    two_ok = correct_answer in r_two["final_answer"]

    print(f"Problem: {problem[:60]}...")
    print(f"  Single-pass: {'✓' if single_ok else '✗'}")
    print(f"  Two-pass:    {'✓' if two_ok else '✗'}")
    print()

Exercise 4: Confidence-based routing system

Implement a system that only uses two-pass verification when the confidence score is low.

def solve_with_confidence_routing(
    problem: str,
    confidence_threshold: float = 0.7
) -> dict:
    """
    First solves with single-pass and computes confidence.
    If confidence is low, escalates to two-pass.
    """
    # Step 1: Single-pass with confidence
    r_single = solve_with_confidence(problem)

    if r_single["confidence_score"] and r_single["confidence_score"] >= confidence_threshold:
        return {
            "answer": r_single["answer"],
            "confidence": r_single["confidence_score"],
            "method_used": "single_pass",
            "escalated": False
        }
    else:
        # Escalate to two-pass
        print(f"Low confidence ({r_single['confidence_score']}), escalating to two-pass...")
        r_two = two_pass_verification(problem)
        return {
            "answer": r_two["final_answer"],
            "confidence": r_single["confidence_score"],  # From the first pass
            "method_used": "two_pass",
            "escalated": True,
            "auditor_verdict": r_two["verdict"]
        }


# Test
problems = [
    "What is 5 + 3?",  # High confidence, no escalation
    "Is the following philosophical argument consistent: the universe exists, therefore God exists?",  # Low confidence, escalates
    "What is the integral of x³?",  # Medium confidence
]

for p in problems:
    r = solve_with_confidence_routing(p)
    print(f"\nProblem: {p[:50]}...")
    print(f"Method: {r['method_used']} | Escalated: {r['escalated']}")
    print(f"Answer: {r['answer']}")

Exercise 5: Detect whether the model fabricated the reasoning

Implement a function that detects "fabricated reasoning" where the model produces steps that don't check out numerically.

import re

def detect_fabricated_reasoning(cot_output: str) -> dict:
    """
    Detects whether the CoT reasoning contains incorrect mathematical operations.
    Extracts every equation from the text and verifies them.
    """
    errors = []
    verified_operations = 0

    # Look for patterns like "17 × 23 = 391" or "340 + 51 = 391"
    multiplication_pattern = r'(\d+(?:\.\d+)?)\s*[×x\*]\s*(\d+(?:\.\d+)?)\s*=\s*(\d+(?:\.\d+)?)'
    addition_pattern = r'(\d+(?:\.\d+)?)\s*\+\s*(\d+(?:\.\d+)?)\s*=\s*(\d+(?:\.\d+)?)'
    subtraction_pattern = r'(\d+(?:\.\d+)?)\s*-\s*(\d+(?:\.\d+)?)\s*=\s*(\d+(?:\.\d+)?)'
    division_pattern = r'(\d+(?:\.\d+)?)\s*[÷/]\s*(\d+(?:\.\d+)?)\s*=\s*(\d+(?:\.\d+)?)'

    def verify_pattern(pattern, operation_fn, name):
        nonlocal verified_operations
        for match in re.finditer(pattern, cot_output):
            a, b, stated_result = float(match.group(1)), float(match.group(2)), float(match.group(3))
            actual_result = operation_fn(a, b)
            verified_operations += 1

            if abs(actual_result - stated_result) > 0.01:
                errors.append(f"{name}: {a} op {b} = {stated_result} (should be {actual_result})")

    verify_pattern(multiplication_pattern, lambda a, b: a * b, "Multiplication")
    verify_pattern(addition_pattern, lambda a, b: a + b, "Addition")
    verify_pattern(subtraction_pattern, lambda a, b: a - b, "Subtraction")
    verify_pattern(division_pattern, lambda a, b: a / b if b != 0 else float('inf'), "Division")

    return {
        "verified_operations": verified_operations,
        "errors_found": errors,
        "fabricated_reasoning": len(errors) > 0,
        "error_rate": len(errors) / max(verified_operations, 1)
    }


# Test
correct_output = "17 × 20 = 340. 17 × 3 = 51. 340 + 51 = 391. Total: 391."
fabricated_output = "17 × 20 = 350. 17 × 3 = 51. 350 + 51 = 401. Total: 401."  # Error in the first step

print("Correct:", detect_fabricated_reasoning(correct_output))
print("Fabricated:", detect_fabricated_reasoning(fabricated_output))

Summary

• Self-verification: The model reviews its own work; useful but carries a self-confirmation bias
• Backward check: Substitutes the answer back into the original problem; very effective for math
• Constraint check: Verifies each constraint with ✓/✗; essential for multi-condition problems
• Confidence score: The model quantifies its certainty; calibrate it against real data
• Two-pass (auditor): A second, independent LLM verifies; more robust but 2x cost
• Ensemble: Combines Self-Consistency + Backward; maximum precision but 3-5x cost
• Rule of thumb: For critical cases, use two-pass or ensemble. For everyday use, self-verification + backward is enough.

Additional resources

1. Self-Consistency Improves Chain of Thought Reasoning (Wang et al., 2022)
2. Verify-and-Edit: A Knowledge-Enhanced CoT Framework (Zhao et al., 2023)
3. Self-RAG: Learning to Retrieve, Generate, and Critique (Asai et al., 2023)
4. Calibration of LLMs (Xiong et al., 2024)
5. OpenAI API - Best Practices for Reliability