3. Self-Consistency: Many Paths, One Answer

Overview

Self-Consistency is a technique proposed by Wang et al. in 2022 that improves the accuracy of language models on reasoning tasks through a simple statistical principle: if you generate multiple independent answers to the same problem and the majority agree, that answer is probably correct.

The intuition is analogous to how juries work in a trial, how expert committees work, or even the "wisdom of the crowd" idea: a single opinion can be biased or wrong, but the consensus of many independent paths tends to be more robust.

The Statistical Intuition

Imagine a model has a 70% chance of answering a hard question correctly with pure CoT. What happens if we ask N times?

P(at least 1 correct with N=5) = 1 - P(all wrong)
P(all wrong) = (0.30)^5 = 0.00243
P(at least 1 correct) ≈ 99.76%

BUT: with majority vote (3 out of 5), the hit distribution improves even more:
- With p=0.7, P(majority correct with N=5) ≈ 83.69%
- With p=0.7, P(majority correct with N=9) ≈ 90.12%
- With p=0.7, P(majority correct with N=15) ≈ 95.24%

The key: the variation in the reasoning (temperature > 0) makes the wrong path take different wrong routes, while the correct path tends to converge on the same answer.

Base Implementation

from openai import OpenAI
from collections import Counter
import re
from typing import Optional

client = OpenAI()

def extract_numeric_answer(text: str) -> Optional[str]:
    """
    Extracts the final numeric answer from a CoT reasoning text.
    Handles multiple formats: 'Answer: 42', '= 42', 'the answer is 42', etc.
    """
    text = text.strip()
    
    # Common final-answer patterns
    patterns = [
        r'(?:answer|result|final)[\s:=]+(-?\d+\.?\d*)',
        r'(?:therefore|thus|so|hence)[,\s]+(?:the answer is\s+)?(-?\d+\.?\d*)',
        r'=\s*(-?\d+\.?\d*)\s*$',  # At the end: = 42
        r'\*\*(-?\d+\.?\d*)\*\*',  # Bold: **42**
    ]
    
    for pattern in patterns:
        match = re.search(pattern, text, re.IGNORECASE | re.MULTILINE)
        if match:
            return match.group(1)
    
    # Fallback: last number in the text
    numbers = re.findall(r'-?\d+\.?\d*', text)
    if numbers:
        return numbers[-1]
    
    # Final fallback: last 30 chars of the text (for non-numeric answers)
    return text.strip()[-30:].strip()

def extract_categorical_answer(text: str, categories: list[str]) -> str:
    """
    Extracts a categorical answer by looking for the categories in the text.
    
    Args:
        text: The model's answer text
        categories: List of valid categories (e.g. ["POSITIVE", "NEGATIVE", "NEUTRAL"])
    """
    text_upper = text.upper()
    for cat in categories:
        if cat.upper() in text_upper:
            return cat
    return text.strip()[-30:]  # Fallback

def self_consistency(
    problem: str,
    n: int = 5,
    temperature: float = 0.7,
    max_tokens: int = 500,
    extract_fn = None
) -> dict:
    """
    Implements Self-Consistency: generate N answers and vote by majority.
    
    Args:
        problem: The problem or question to solve
        n: Number of samples (typically 3-10)
        temperature: Temperature for variation (0.5-1.0 recommended)
        max_tokens: Max tokens per answer
        extract_fn: Function to extract the final answer from the text
    
    Returns:
        dict with the winning answer, all the answers, and statistics
    """
    if extract_fn is None:
        extract_fn = extract_numeric_answer
    
    cot_prompt = f"""{problem}

Think step by step and show all your reasoning before giving the final answer.
At the end, write explicitly: "Answer: [your answer]"
"""
    
    raw_answers = []
    extracted_answers = []
    
    for i in range(n):
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": cot_prompt}],
            temperature=temperature,
            max_tokens=max_tokens
        )
        raw = response.choices[0].message.content
        extracted = extract_fn(raw)
        
        raw_answers.append(raw)
        extracted_answers.append(extracted)
    
    # Majority vote
    counts = Counter(extracted_answers)
    winner, votes = counts.most_common(1)[0]
    
    return {
        "answer": winner,
        "votes": votes,
        "total": n,
        "confidence": votes / n,
        "distribution": dict(counts),
        "raw_answers": raw_answers,
        "all_answers": extracted_answers
    }

Advanced Implementation with Normalization

def normalize_answer(text: str) -> str:
    """
    Normalizes answers so the voting is more robust.
    Handles cases like "42.0" vs "42", "42,000" vs "42000", etc.
    """
    text = text.strip()
    
    # Remove trailing punctuation
    text = text.rstrip('.,;:')
    
    # Normalize numbers: strip thousands separators
    text = text.replace(',', '')
    
    # Normalize decimals: "42.0" → "42"
    try:
        num = float(text)
        if num == int(num):
            return str(int(num))
        return str(round(num, 4))
    except ValueError:
        pass
    
    # For categorical answers: uppercase and strip
    return text.upper().strip()

def self_consistency_robust(
    problem: str,
    n: int = 5,
    temperature: float = 0.7,
    normalize: bool = True
) -> dict:
    """
    Self-Consistency with answer normalization for more robust voting.
    """
    result = self_consistency(problem, n=n, temperature=temperature)
    
    if normalize:
        # Re-normalize every answer and re-vote
        normalized_answers = [normalize_answer(a) for a in result["all_answers"]]
        normalized_counts = Counter(normalized_answers)
        normalized_winner, normalized_votes = normalized_counts.most_common(1)[0]
        
        result["normalized_answer"] = normalized_winner
        result["normalized_distribution"] = dict(normalized_counts)
        result["normalized_confidence"] = normalized_votes / n
    
    return result

# Usage example:
if __name__ == "__main__":
    problem = "In a store, an item costs $80 after a 20% discount. What was the original price?"
    
    result = self_consistency_robust(problem, n=5)
    print(f"Answer: {result['answer']}")
    print(f"Distribution: {result['distribution']}")
    print(f"Confidence: {result['confidence']:.0%}")
    print(f"Clear consensus? {result['confidence'] >= 0.6}")

Self-Consistency Variants

Variant 1: Self-Consistency for Classification

def self_consistency_classification(
    text: str,
    categories: list[str],
    n: int = 5
) -> dict:
    """
    Self-Consistency for classification tasks.
    Useful when the categorization is ambiguous.
    """
    cats_str = ", ".join(categories)
    prompt = f"""Classify the following text into one of these categories: {cats_str}

Text: {text}

First analyze the text, then classify it. Reply with exactly one of the categories.
"""
    
    def extract_category(answer: str) -> str:
        return extract_categorical_answer(answer, categories)
    
    result = self_consistency(
        prompt,
        n=n,
        temperature=0.5,  # Less temperature for classification
        extract_fn=extract_category
    )
    
    return result

# Example:
reviews = [
    "The product arrived fine but customer service was horrible.",  # Ambiguous: POSITIVE/NEGATIVE
    "Absolutely amazing, I recommend it to everyone.",  # Clear: POSITIVE
    "It could be better, but it's not that bad either."  # Ambiguous: NEUTRAL
]

for review in reviews:
    result = self_consistency_classification(
        review,
        ["POSITIVE", "NEGATIVE", "NEUTRAL"],
        n=5
    )
    print(f"Review: {review[:50]}...")
    print(f"Classification: {result['answer']} (confidence: {result['confidence']:.0%})")
    print(f"Distribution: {result['distribution']}\n")

Variant 2: Weighted Self-Consistency

def self_consistency_weighted(
    problem: str,
    n: int = 5
) -> dict:
    """
    Instead of a plain majority vote, assign weights based on the length
    and coherence of the reasoning.
    """
    answers_with_metadata = []
    
    for _ in range(n):
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"{problem}\nThink step by step. Answer: [number]"}],
            temperature=0.7,
            max_tokens=500,
            logprobs=True  # Get log-probabilities for the weights
        )
        msg = response.choices[0].message
        raw = msg.content
        
        # Extract the answer
        answer = extract_numeric_answer(raw)
        
        # Compute a weight based on the length of the reasoning
        # (longer, more detailed reasoning tends to be more reliable)
        weight = min(len(raw.split()) / 100, 2.0)  # Normalize, max weight 2x
        
        answers_with_metadata.append({
            "answer": answer,
            "weight": weight,
            "length": len(raw.split())
        })
    
    # Weighted voting
    weighted_votes = {}
    for item in answers_with_metadata:
        ans = item["answer"]
        weighted_votes[ans] = weighted_votes.get(ans, 0) + item["weight"]
    
    winner = max(weighted_votes, key=weighted_votes.get)
    
    return {
        "answer": winner,
        "weighted_votes": weighted_votes,
        "details": answers_with_metadata
    }

Variant 3: Adaptive Self-Consistency (Early Stop)

def self_consistency_adaptive(
    problem: str,
    n_min: int = 3,
    n_max: int = 10,
    confidence_threshold: float = 0.8
) -> dict:
    """
    When the answer is already very clear (high confidence),
    stop generating more samples to save tokens.
    """
    extracted_answers = []
    
    for i in range(n_max):
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"{problem}\nThink step by step. Answer: [your answer]"}],
            temperature=0.7,
            max_tokens=400
        )
        raw = response.choices[0].message.content
        extracted_answers.append(extract_numeric_answer(raw))
        
        # Only evaluate after the minimum
        if i + 1 >= n_min:
            counts = Counter(extracted_answers)
            winner, votes = counts.most_common(1)[0]
            confidence = votes / (i + 1)
            
            if confidence >= confidence_threshold:
                return {
                    "answer": winner,
                    "confidence": confidence,
                    "n_used": i + 1,
                    "early_stop": True,
                    "distribution": dict(counts)
                }
    
    # If we never hit the threshold, return the best answer
    counts = Counter(extracted_answers)
    winner, votes = counts.most_common(1)[0]
    return {
        "answer": winner,
        "confidence": votes / n_max,
        "n_used": n_max,
        "early_stop": False,
        "distribution": dict(counts)
    }

# Savings benchmark:
simple_questions = ["What is 8 * 9?", "What is 15 + 27?"]
hard_questions = ["If a circle's radius grows by 50%, by what percentage does the area grow?"]

for question in simple_questions + hard_questions:
    r = self_consistency_adaptive(question)
    print(f"Question: {question[:50]}...")
    print(f"  Answer: {r['answer']}, Confidence: {r['confidence']:.0%}, "
          f"Calls: {r['n_used']}, Early stop: {r['early_stop']}")

When to Use Self-Consistency

Tasks where it shines

Tasks where it does NOT help

• Creative tasks: Writing a poem — there's no single "correct answer"
• Format tasks: Translation — the majority vote can mix languages
• When the data is scarce: If the model doesn't know the answer, N wrong answers are still wrong
• Conversations: In multi-turn dialogue, context matters more than consistency

Trade-offs: Cost vs. Accuracy

def analyze_tradeoffs(problem: str, max_n: int = 10) -> list[dict]:
    """
    Analyzes the trade-off between N samples and confidence.
    """
    history = []
    all_answers = []
    
    for i in range(1, max_n + 1):
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"{problem}\nThink step by step. Answer: [number]"}],
            temperature=0.7,
            max_tokens=300
        )
        all_answers.append(extract_numeric_answer(response.choices[0].message.content))
        
        counts = Counter(all_answers)
        winner, votes = counts.most_common(1)[0]
        
        history.append({
            "n": i,
            "answer": winner,
            "confidence": votes / i,
            "relative_cost": i  # Simplification: every call costs the same
        })
    
    return history

# Visualize:
# import matplotlib.pyplot as plt  # If you have matplotlib
# for point in analyze_tradeoffs("If a train travels at 120 km/h, how long does it take to cover 300 km?"):
#     print(f"N={point['n']}: Answer={point['answer']}, Confidence={point['confidence']:.0%}")

Reference table

The inflection point is usually at N=5: a good balance between improvement and cost.

Self-Consistency with Anthropic

import anthropic

client_anthropic = anthropic.Anthropic()

def self_consistency_claude(
    problem: str,
    n: int = 5,
    temperature: float = 0.7
) -> dict:
    """
    Self-Consistency using Claude (Anthropic).
    """
    prompt = f"""{problem}

Analyze the problem step by step and at the end write "Answer: [your answer]"."""
    
    extracted_answers = []
    raw_answers = []
    
    for _ in range(n):
        message = client_anthropic.messages.create(
            model="claude-3-5-haiku-20241022",
            max_tokens=500,
            temperature=temperature,
            messages=[{"role": "user", "content": prompt}]
        )
        raw = message.content[0].text
        raw_answers.append(raw)
        extracted_answers.append(extract_numeric_answer(raw))
    
    counts = Counter(extracted_answers)
    winner, votes = counts.most_common(1)[0]
    
    return {
        "answer": winner,
        "confidence": votes / n,
        "distribution": dict(counts),
        "model": "claude-3-5-haiku-20241022"
    }

Real Benchmark: Self-Consistency vs Single CoT

import json
import time

# Test set with known answers
BENCHMARK = [
    {
        "problem": "A trader buys 50 kg of apples at $2/kg and sells them at $3/kg. If 10 kg rot before he sells them, what is his profit or loss?",
        "correct_answer": "20",  # (40 * 3) - (50 * 2) = 120 - 100 = 20 USD
        "problem_type": "word_problem"
    },
    {
        "problem": "What is 15% of 840?",
        "correct_answer": "126",
        "problem_type": "percentage"
    },
    {
        "problem": "If 5 machines make 5 parts in 5 minutes, how many machines do you need to make 100 parts in 100 minutes?",
        "correct_answer": "5",  # Counterintuitive answer
        "problem_type": "logic"
    },
    {
        "problem": "A boat holds 10 people. How many trips does it need to ferry 45 people across?",
        "correct_answer": "5",  # 45/10 rounded up
        "problem_type": "division"
    }
]

def evaluate_answer(pred: str, correct: str) -> bool:
    """Compares answers with normalization."""
    try:
        return abs(float(normalize_answer(pred)) - float(normalize_answer(correct))) < 0.01
    except ValueError:
        return pred.strip() == correct.strip()

def run_benchmark(n_samples: int = 5) -> dict:
    """Runs the benchmark comparing plain CoT vs Self-Consistency."""
    cot_results = []
    sc_results = []
    
    for item in BENCHMARK:
        # Single CoT
        cot_resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"{item['problem']}\nThink step by step. Answer: [number]"}],
            temperature=0
        ).choices[0].message.content
        cot_pred = extract_numeric_answer(cot_resp)
        cot_correct = evaluate_answer(cot_pred, item["correct_answer"])
        cot_results.append(cot_correct)
        
        # Self-Consistency
        sc_result = self_consistency(item["problem"], n=n_samples)
        sc_correct = evaluate_answer(sc_result["answer"], item["correct_answer"])
        sc_results.append(sc_correct)
        
        print(f"\nProblem: {item['problem'][:60]}...")
        print(f"  CoT: {cot_pred} → {'✓' if cot_correct else '✗'}")
        print(f"  SC: {sc_result['answer']} (confidence {sc_result['confidence']:.0%}) → {'✓' if sc_correct else '✗'}")
    
    accuracy_cot = sum(cot_results) / len(cot_results)
    accuracy_sc = sum(sc_results) / len(sc_results)
    
    print(f"\n=== RESULTS ===")
    print(f"Plain CoT: {accuracy_cot:.0%}")
    print(f"Self-Consistency N={n_samples}: {accuracy_sc:.0%}")
    print(f"Improvement: +{(accuracy_sc - accuracy_cot) * 100:.1f}%")
    
    return {
        "accuracy_cot": accuracy_cot,
        "accuracy_sc": accuracy_sc,
        "improvement": accuracy_sc - accuracy_cot
    }

Troubleshooting

Problem 1: Answers in different formats break the voting

Symptom: "42" and "42.0" and "42,0" count as different answers even though they're the same.

Solution:

def normalize_aggressive(text: str) -> str:
    """More aggressive normalization for voting."""
    # Strip everything except digits and the decimal point
    digits_only = re.sub(r'[^\d.-]', '', text.split()[-1] if text.split() else text)
    try:
        num = float(digits_only)
        # Round to 2 decimals to avoid precision differences
        num = round(num, 2)
        return str(int(num)) if num == int(num) else str(num)
    except ValueError:
        return text.upper().strip()

Problem 2: A tie between answers (e.g. 2 vs 2 with N=4)

Symptom: Counter.most_common(1) returns one of the tied answers arbitrarily.

Solution:

def break_tie(counts: Counter, raw_answers: list[str]) -> str:
    """
    Strategies to break ties:
    1. Pick the longest answer (more reasoning)
    2. Make a tiebreak call
    3. Return the most "conservative" one (the lower number in finance, for instance)
    """
    max_votes = counts.most_common(1)[0][1]
    tied = [a for a, v in counts.items() if v == max_votes]
    
    if len(tied) == 1:
        return tied[0]
    
    # Strategy: break the tie with an extra call
    candidates_str = "\n".join([f"- {a}" for a in tied])
    tiebreak = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": f"The following answers are tied with the same number of votes:\n{candidates_str}\n\nWhich one is mathematically more likely to be correct? Reply with the correct answer only."
        }],
        temperature=0
    )
    return tiebreak.choices[0].message.content.strip()

Problem 3: High cost for large N

Symptom: N=10 is too expensive for a high volume of queries.

Solution: Use self_consistency_adaptive (see above) or a dynamic threshold:

def self_consistency_budget(
    problem: str,
    budget_usd: float = 0.001  # $0.001 per problem
) -> dict:
    """
    Self-Consistency with a fixed budget.
    Estimates N from the available budget.
    """
    # Estimate: ~500 tokens per call, gpt-4o-mini ~$0.000195/call
    cost_per_call = 0.000195
    n_max = max(1, int(budget_usd / cost_per_call))
    n_effective = min(n_max, 10)  # Cap at 10
    
    return self_consistency(problem, n=n_effective)

Exercises

Exercise 1: Implement robust answer extraction

Improve the extract_numeric_answer function so it handles all of these formats:

"The answer is 42"
"42"
"42.0"
"= 42"
"Therefore, the answer is **42**"
"Profit: $42 USD"
"Conclusion: 42 items"
"The answer is forty-two (42)"

import re

def extract_answer_v2(text: str) -> str:
    """Robust extraction of a numeric answer."""
    text = text.strip()
    
    # Pattern 1: "Answer: 42" or "Result: 42"
    m = re.search(r'(?:answer|result|profit|loss|total|final)[:\s]+\$?(-?\d+\.?\d*)', text, re.IGNORECASE)
    if m:
        return m.group(1)
    
    # Pattern 2: "**42**" (markdown bold)
    m = re.search(r'\*\*\$?(-?\d+\.?\d*)\*\*', text)
    if m:
        return m.group(1)
    
    # Pattern 3: "= 42" at end of line
    m = re.search(r'=\s*\$?(-?\d+\.?\d*)\s*(?:USD|EUR|€|\$)?$', text, re.MULTILINE)
    if m:
        return m.group(1)
    
    # Pattern 4: number in parentheses with context "forty-two (42)"
    m = re.search(r'\((\d+)\)', text)
    if m:
        return m.group(1)
    
    # Pattern 5: "42 items", "42 USD", etc.
    m = re.search(r'\b(-?\d+\.?\d*)\s*(?:items?|USD|EUR|€|\$|kg|km|years?|days?|months?)?\s*$', text)
    if m:
        return m.group(1)
    
    # Fallback: last number in the text
    numbers = re.findall(r'-?\d+\.?\d*', text)
    return numbers[-1] if numbers else text[-20:]

# Test:
cases = [
    "The answer is 42",
    "42",
    "42.0",
    "= 42",
    "Therefore, the answer is **42**",
    "Profit: $42 USD",
    "Conclusion: 42 items",
    "The answer is forty-two (42)"
]

for case in cases:
    extracted = extract_answer_v2(case)
    print(f"'{case[:40]}' → '{extracted}'")

Exercise 2: Self-Consistency for multiple choice

Implement Self-Consistency for multiple-choice questions (A, B, C, D). The voting has to look for the letter, not the number.

def self_consistency_multiple_choice(
    question: str,
    options: dict,  # {"A": "option a", "B": "option b", ...}
    n: int = 5
) -> dict:
    """Self-Consistency for multiple choice."""
    options_str = "\n".join([f"{k}) {v}" for k, v in options.items()])
    
    prompt = f"""{question}

Options:
{options_str}

Analyze each option and reason about why it is or isn't correct. At the end write ONLY the letter of the correct answer."""
    
    def extract_letter(text: str) -> str:
        # Look for the letter at the end, or the most recent one
        letters = re.findall(r'\b([A-D])\b', text.upper())
        return letters[-1] if letters else "?"
    
    result = self_consistency(
        prompt,
        n=n,
        temperature=0.5,  # Less variation for MC
        extract_fn=extract_letter
    )
    
    answer_letter = result["answer"]
    answer_text = options.get(answer_letter, "Not found")
    
    return {
        **result,
        "full_answer": f"{answer_letter}) {answer_text}"
    }

# Test:
question = "What is the capital of Brazil?"
options = {"A": "São Paulo", "B": "Rio de Janeiro", "C": "Brasília", "D": "Buenos Aires"}
r = self_consistency_multiple_choice(question, options, n=5)
print(f"Answer: {r['full_answer']} (confidence: {r['confidence']:.0%})")

Exercise 3: Confidence analysis

Run Self-Consistency with N=10 on 5 questions of varying difficulty. Plot the relationship between confidence (majority/total) and the correctness of the answer. Is there a correlation?

questions_with_answer = [
    ("What is 7 * 8?", "56"),      # Easy, high confidence
    ("What is 23 * 17?", "391"),   # Medium
    ("A train goes at 120km/h, how long does it take to cover 300km?", "2.5"),  # Word problem
    ("If the radius grows 50%, by what % does the area grow?", "125"),  # Conceptual
    ("How many rotations does a wheel with a 2m circumference make in 1km?", "500"),  # Calculation
]

analysis = []
for question, correct in questions_with_answer:
    r = self_consistency(question, n=10)
    is_correct = evaluate_answer(r["answer"], correct)
    analysis.append({
        "question": question[:40],
        "answer": r["answer"],
        "correct": correct,
        "confidence": r["confidence"],
        "hit": is_correct
    })
    print(f"Question: {question[:40]}...")
    print(f"  Pred: {r['answer']}, Correct: {correct}")
    print(f"  Confidence: {r['confidence']:.0%}, Hit: {'✓' if is_correct else '✗'}")

# Compute the correlation (simplified):
# High confidence (>0.6) + Hit: True positive
# High confidence + No hit: False positive
# Low confidence + Hit: False negative
# Low confidence + No hit: True negative

Exercise 4: Compare temperatures

Experiment with different temperature values (0.3, 0.5, 0.7, 1.0) for N=5. Which temperature gives the best balance between reasoning diversity and convergence on the correct answer?

problem = "A trader sells an item at a 25% profit. If the cost price is $80, what is the selling price?"

temperatures = [0.3, 0.5, 0.7, 1.0]
for temp in temperatures:
    results = []
    for _ in range(5):  # 5 runs to average
        r = self_consistency(problem, n=5, temperature=temp)
        results.append(r["confidence"])
    
    avg_confidence = sum(results) / len(results)
    # Correct answer: 80 * 1.25 = 100
    print(f"Temperature={temp}: average confidence={avg_confidence:.0%}")

# Expected result:
# Temperature=0.3: high confidence (~90%), little diversity
# Temperature=0.7: balance (~80%), good diversity
# Temperature=1.0: low confidence (~65%), lots of diversity

Summary

• Self-Consistency: Generate N independent answers with temperature > 0, vote by majority
• Empirical improvement: +5-15% in accuracy for math/logic/reasoning vs single CoT
• Temperature: 0.5-0.8 is the sweet spot for the diversity/convergence balance
• Optimal N: 5 for most cases; stop early if confidence > 80%
• Normalization: Critical for the voting to work at all (42 vs 42.0 vs "42")
• When to use it: Math word problems, logical reasoning, ambiguous classification
• When NOT to use it: Creative tasks, conversations, when the model definitely doesn't know the answer

Additional resources

1. Self-Consistency Improves Chain of Thought Reasoning (Wang et al., 2022) - Original paper
2. Large Language Models are Zero-Shot Reasoners (Kojima et al., 2022) - Related to CoT
3. OpenAI API - Temperature parameter
4. Python Counter documentation
5. Prompt Engineering Guide - Self-Consistency
6. Benchmark datasets for math: GSM8K