6. Combining Techniques

Overview

The techniques we've seen —CoT, Self-Consistency, ReAct, Tree-of-Thought, Meta-prompting and Self-Refine— are not mutually exclusive. Combined strategically, they can beat the performance of any of them on its own. But combining techniques indiscriminately can also raise the cost with no real improvement.

This capsule explores when and how to combine techniques, with practical implementations and an analysis of when the combination is worth it.

Why Combine Techniques

Each technique solves a different problem:

When a task has several problems at once, combining techniques attacks each one.

Combination 1: CoT + Self-Consistency (The most common)

The most used combination in production. Each of Self-Consistency's N answers uses CoT internally, which improves both the individual quality of each answer and the final consensus.

from openai import OpenAI
from collections import Counter
import re

client = OpenAI()

def extract_numeric_answer(text: str) -> str:
    """Extracts the final numeric answer from a CoT text."""
    patterns = [
        r'(?:answer|result)[\s:=]+(-?\d+\.?\d*)',
        r'=\s*(-?\d+\.?\d*)\s*$',
        r'\*\*(-?\d+\.?\d*)\*\*'
    ]
    for pattern in patterns:
        m = re.search(pattern, text, re.IGNORECASE | re.MULTILINE)
        if m:
            return m.group(1)
    numbers = re.findall(r'-?\d+\.?\d*', text)
    return numbers[-1] if numbers else text.strip()[-20:]

def cot_self_consistency(
    problem: str,
    n: int = 5,
    temperature: float = 0.7,
    include_reasoning: bool = True
) -> dict:
    """
    CoT + Self-Consistency: N answers with CoT, majority vote.
    
    The difference from plain Self-Consistency is that the prompt
    explicitly forces more detailed reasoning before the answer.
    """
    detailed_cot_prompt = f"""{problem}

Instructions:
1. Identify the known and unknown data
2. State which formulas or concepts apply
3. Solve step by step, showing EVERY operation
4. Check whether the answer makes sense
5. At the end write: "Final answer: [value]"
"""
    
    answers = []
    reasonings = []
    
    for _ in range(n):
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": detailed_cot_prompt}],
            temperature=temperature,
            max_tokens=600
        )
        raw = response.choices[0].message.content
        answer = extract_numeric_answer(raw)
        answers.append(answer)
        reasonings.append(raw)
    
    counts = Counter(answers)
    winner, votes = counts.most_common(1)[0]
    
    # Find the reasoning that led to the winning answer
    best_reasoning = next(
        (r for r, a in zip(reasonings, answers) if a == winner),
        reasonings[0]
    )
    
    return {
        "answer": winner,
        "confidence": votes / n,
        "distribution": dict(counts),
        "best_reasoning": best_reasoning if include_reasoning else None
    }

# Benchmark vs CoT alone:
math_problems = [
    "Maria has 5 times more money than John. If between the two of them they have $720, how much does each have?",
    "A train leaves A at 80 km/h and another leaves B at 60 km/h. They are 420 km apart. How long until they meet?",
    "If VAT is 21%, what is the pre-VAT price of an item that costs $242 with VAT?"
]

for problem in math_problems:
    result = cot_self_consistency(problem, n=5)
    print(f"Problem: {problem[:60]}...")
    print(f"  Answer: {result['answer']} (confidence: {result['confidence']:.0%})")

Combination 2: ReAct + Structured Output

ReAct for reasoning with tools; the final output in JSON format parsed with Pydantic. This combination is ideal for production systems that need both external data and reliable outputs.

from pydantic import BaseModel, Field
from typing import Optional
import json

class ReActAnswer(BaseModel):
    """Schema for ReAct's structured output."""
    answer: str = Field(description="The answer to the problem")
    confidence: float = Field(ge=0, le=1, description="Confidence in the answer (0-1)")
    sources_consulted: list[str] = Field(default_factory=list)
    reasoning_steps: list[str] = Field(default_factory=list)
    needs_clarification: bool = Field(default=False)

def react_with_structured_output(problem: str) -> ReActAnswer:
    """
    ReAct that finishes with a structured JSON output validated with Pydantic.
    """
    # First run plain ReAct to get the reasoning
    from .02_react_reasoning_acting import run_react  # In production, import properly
    
    # Adapt the final prompt to ask for JSON
    system_prompt = """You are an assistant that solves problems using tools.
    
When you finish, ALWAYS reply in JSON with this format:
{
    "answer": "the answer to the problem",
    "confidence": 0.9,
    "sources_consulted": ["tool1", "tool2"],
    "reasoning_steps": ["step1", "step2"],
    "needs_clarification": false
}"""
    
    # Inline version so we don't depend on external imports
    BASIC_TOOLS = [
        {
            "type": "function",
            "function": {
                "name": "search",
                "description": "Searches for factual information",
                "parameters": {
                    "type": "object",
                    "properties": {"query": {"type": "string"}},
                    "required": ["query"]
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "calculate",
                "description": "Evaluates mathematical expressions",
                "parameters": {
                    "type": "object",
                    "properties": {"expression": {"type": "string"}},
                    "required": ["expression"]
                }
            }
        }
    ]
    
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": problem}
    ]
    
    tools_used = []
    steps = []
    
    for _ in range(8):
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=messages,
            tools=BASIC_TOOLS,
            tool_choice="auto",
            temperature=0
        )
        msg = response.choices[0].message
        
        if not msg.tool_calls:
            # Final answer - parse it as JSON
            try:
                # Try to extract the JSON from the text
                content = msg.content or "{}"
                json_match = re.search(r'\{.*\}', content, re.DOTALL)
                if json_match:
                    data = json.loads(json_match.group())
                    data["sources_consulted"] = tools_used
                    data["reasoning_steps"] = steps
                    return ReActAnswer(**data)
            except Exception:
                pass
            
            # Fallback
            return ReActAnswer(
                answer=msg.content or "No answer",
                confidence=0.5,
                sources_consulted=tools_used,
                reasoning_steps=steps
            )
        
        messages.append({
            "role": "assistant",
            "content": msg.content or "",
            "tool_calls": [tc.model_dump() for tc in msg.tool_calls]
        })
        
        for tc in msg.tool_calls:
            tool_name = tc.function.name
            args = json.loads(tc.function.arguments)
            
            # Simulate the tools
            if tool_name == "search":
                result = f"Search result: {args.get('query', '')} - [simulated data]"
            elif tool_name == "calculate":
                try:
                    result = str(eval(args.get('expression', '0'), {"__builtins__": {}}))
                except Exception:
                    result = "Calculation error"
            else:
                result = "Tool not available"
            
            tools_used.append(tool_name)
            steps.append(f"{tool_name}({args})")
            
            messages.append({
                "role": "tool",
                "tool_call_id": tc.id,
                "content": result
            })
    
    return ReActAnswer(answer="Timeout", confidence=0.0)


# Usage:
result = react_with_structured_output("What is 15% of 1500 EUR in USD using the current exchange rate?")
print(f"Answer: {result.answer}")
print(f"Confidence: {result.confidence:.0%}")
print(f"Steps: {result.reasoning_steps}")

Combination 3: ToT + Self-Consistency

For problems where there are several valid strategies AND we want to verify the result by consensus.

def tot_self_consistency(
    problem: str,
    n_trees: int = 3,
    breadth_per_tree: int = 2
) -> dict:
    """
    Runs N independent ToT trees and votes for the most common answer.
    
    Each tree starts from a different approach (temperature seed),
    and the final answer is a majority vote over the N trees.
    """
    tree_answers = []
    
    for i in range(n_trees):
        # Each tree uses a different temperature for diversity
        temperature = 0.5 + (i * 0.2)
        
        approaches_prompt = f"""Problem: {problem}

Generate {breadth_per_tree} distinct approaches to solve it.
Choose the approaches that are as DIFFERENT from each other as possible.
Approach number followed by a description of the first step."""
        
        approaches_resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": approaches_prompt}],
            temperature=temperature,
            max_tokens=300
        )
        lines = [l.lstrip('0123456789.-) ').strip() 
                 for l in approaches_resp.choices[0].message.content.split('\n') 
                 if l.strip() and l.strip()[0].isdigit()][:breadth_per_tree]
        
        # Evaluate and select the best approach
        if not lines:
            continue
        
        scores = []
        for approach in lines:
            eval_resp = client.chat.completions.create(
                model="gpt-4o-mini",
                messages=[{"role": "user", "content": f"Problem: {problem}\nApproach: {approach}\nPromising? (0-1)"}],
                temperature=0, max_tokens=10
            )
            try:
                score = float(eval_resp.choices[0].message.content.strip()[:4])
            except ValueError:
                score = 0.5
            scores.append((score, approach))
        
        best_approach = max(scores, key=lambda x: x[0])[1]
        
        # Solve from the best approach
        solution_resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"{problem}\n\nApproach: {best_approach}\n\nSolve it. Final answer: [value]"}],
            temperature=0,
            max_tokens=400
        )
        answer = extract_numeric_answer(solution_resp.choices[0].message.content)
        tree_answers.append(answer)
    
    if not tree_answers:
        return {"answer": "No result", "confidence": 0.0}
    
    # Majority vote over the trees' results
    counts = Counter(tree_answers)
    winner, votes = counts.most_common(1)[0]
    
    return {
        "answer": winner,
        "confidence": votes / len(tree_answers),
        "answers_per_tree": tree_answers,
        "n_trees": len(tree_answers)
    }

Combination 4: Meta-Prompting + Self-Refine + Evaluation

The full pipeline for automatic prompt optimization:

def full_optimization_pipeline(
    task_description: str,
    eval_dataset: list[dict],
    n_iterations: int = 3
) -> dict:
    """
    Full optimization pipeline:
    1. Meta-prompting generates the initial prompt
    2. It gets evaluated on a dataset
    3. Self-Refine improves the prompt based on the errors
    4. Repeat until convergence or N iterations
    
    Args:
        task_description: Description of the task
        eval_dataset: List of {input: str, expected: str}
        n_iterations: Improvement iterations
    
    Returns:
        dict with final_prompt, accuracy_history, improvements
    """
    # Step 1: Generate the initial prompt with meta-prompting
    meta_prompt = f"""Create an optimal prompt for: {task_description}
    
The prompt has to be specific, include a clear output format, and handle edge cases.
Return only the prompt."""
    
    current_prompt = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": meta_prompt}],
        temperature=0.3,
        max_tokens=500
    ).choices[0].message.content.strip()
    
    history = []
    
    for iteration in range(n_iterations):
        # Step 2: Evaluate the current prompt
        errors = []
        hits = 0
        
        for example in eval_dataset[:5]:  # Cap it to save calls
            resp = client.chat.completions.create(
                model="gpt-4o-mini",
                messages=[{"role": "user", "content": f"{current_prompt}\n\nInput: {example['input']}"}],
                temperature=0,
                max_tokens=200
            ).choices[0].message.content
            
            expected = example.get("expected", "")
            correct = expected.lower() in resp.lower() if expected else False
            
            if correct:
                hits += 1
            else:
                errors.append({
                    "input": example["input"][:100],
                    "expected": expected,
                    "got": resp[:100]
                })
        
        accuracy = hits / min(len(eval_dataset), 5)
        
        history.append({
            "iteration": iteration,
            "accuracy": accuracy,
            "prompt": current_prompt[:200]
        })
        
        if accuracy >= 0.9:
            break
        
        # Step 3: Self-Refine the prompt based on the errors
        if errors:
            errors_str = json.dumps(errors[:3], ensure_ascii=False, indent=2)
            refine_prompt = f"""Task: {task_description}

Current prompt:
{current_prompt}

Errors made when using this prompt:
{errors_str}

Improve the prompt to fix these errors without breaking the cases that work.
Return only the improved prompt."""
            
            improved_prompt = client.chat.completions.create(
                model="gpt-4o-mini",
                messages=[{"role": "user", "content": refine_prompt}],
                temperature=0.2,
                max_tokens=500
            ).choices[0].message.content.strip()
            
            current_prompt = improved_prompt
    
    return {
        "final_prompt": current_prompt,
        "accuracy_history": [h["accuracy"] for h in history],
        "iterations": history,
        "total_improvement": history[-1]["accuracy"] - history[0]["accuracy"] if history else 0
    }

When to Combine vs When Not To

When combining IS useful

A task with MULTIPLE simultaneous requirements:
├── Needs external data + Critical accuracy
│   └── → ReAct + Self-Consistency
├── Multiple strategies + Verification
│   └── → ToT + Self-Consistency
├── Suboptimal prompt + Improvable answer
│   └── → Meta-prompting + Self-Refine
└── External data + Structured output
    └── → ReAct + Structured Output

When combining is NOT useful (overkill)

# OVERKILL: A simple classification task
def classify_sentiment_overkill(text: str) -> str:
    # WRONG: it doesn't need ToT or Self-Consistency for a simple classification
    return tot_self_consistency(
        f"Classify the sentiment: {text}",
        n_trees=5, breadth_per_tree=3
    )["answer"]

# CORRECT: Few-shot is enough
def classify_sentiment_optimal(text: str) -> str:
    return client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"""
Classify the sentiment as POSITIVE, NEGATIVE or NEUTRAL.
Examples: "I loved it" → POSITIVE | "Horrible" → NEGATIVE | "It's fine" → NEUTRAL

Text: {text}
Reply only with: POSITIVE, NEGATIVE, or NEUTRAL"""}],
        temperature=0
    ).choices[0].message.content.strip()

General rule: the complexity ladder

Measuring the Value of Combining

def benchmark_combinations(
    problems: list[dict],  # [{"statement": str, "answer": str}]
    techniques: list[str] = None
) -> dict:
    """
    Compares several techniques and combinations on a set of problems.
    
    Args:
        problems: List of problems with a known correct answer
        techniques: List of techniques to compare
    """
    if techniques is None:
        techniques = ["cot_only", "cot_sc_n3", "cot_sc_n5"]
    
    results = {t: {"hits": 0, "total": len(problems), "calls": 0} 
               for t in techniques}
    
    def normalize(text: str) -> str:
        try:
            return str(round(float(re.sub(r'[^\d.-]', '', text.split()[-1] if text.split() else text)), 1))
        except ValueError:
            return text.strip().upper()
    
    for problem in problems:
        statement = problem["statement"]
        correct = normalize(str(problem["answer"]))
        
        for technique in techniques:
            if technique == "cot_only":
                resp = client.chat.completions.create(
                    model="gpt-4o-mini",
                    messages=[{"role": "user", "content": f"{statement}\nThink step by step. Answer: [number]"}],
                    temperature=0, max_tokens=400
                ).choices[0].message.content
                pred = normalize(extract_numeric_answer(resp))
                results["cot_only"]["calls"] += 1
            
            elif technique == "cot_sc_n3":
                r = cot_self_consistency(statement, n=3, include_reasoning=False)
                pred = normalize(r["answer"])
                results["cot_sc_n3"]["calls"] += 3
            
            elif technique == "cot_sc_n5":
                r = cot_self_consistency(statement, n=5, include_reasoning=False)
                pred = normalize(r["answer"])
                results["cot_sc_n5"]["calls"] += 5
            
            else:
                pred = "?"
            
            if pred == correct or (len(pred) > 0 and correct in pred):
                results[technique]["hits"] += 1
    
    # Compute the metrics
    for technique in results:
        r = results[technique]
        r["accuracy"] = r["hits"] / r["total"]
        r["calls_per_problem"] = r["calls"] / r["total"]
    
    return results


# Example:
benchmark_data = [
    {"statement": "What is 15% of 480?", "answer": 72},
    {"statement": "If 3x + 5 = 17, what is x?", "answer": 4},
    {"statement": "A train covers 240 km in 3 hours. What is its average speed in km/h?", "answer": 80},
]

metrics = benchmark_combinations(benchmark_data)
for technique, data in metrics.items():
    print(f"{technique}: accuracy={data['accuracy']:.0%}, calls/problem={data['calls_per_problem']:.1f}")

A Practical Combination: A Robust Q&A System

A question-and-answer system that combines several techniques depending on the type of question:

def robust_qa(question: str) -> dict:
    """
    A Q&A system that automatically selects the right combination.
    """
    # Step 1: Classify the question
    classification = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"""
Classify this question. Reply in JSON: {{"type": "...", "needs_external_data": true/false, "accuracy_critical": true/false}}

Types: math (numeric operations), logic (reasoning), factual (facts about the world), creative (no single correct answer)

Question: {question}"""}],
        temperature=0,
        response_format={"type": "json_object"}
    ).choices[0].message.content
    
    try:
        clf = json.loads(classification)
    except Exception:
        clf = {"type": "logic", "needs_external_data": False, "accuracy_critical": False}
    
    task_type = clf.get("type", "logic")
    needs_external = clf.get("needs_external_data", False)
    accuracy_critical = clf.get("accuracy_critical", False)
    
    # Step 2: Select the optimal technique
    if needs_external and accuracy_critical:
        # ReAct + Self-Consistency (more expensive but more accurate)
        # Simplified: use CoT + Self-Consistency, noting that external data is missing
        result = cot_self_consistency(question, n=5)
        technique_used = "cot_sc_n5 (ideally: react+sc)"
    
    elif needs_external:
        # ReAct
        from openai import OpenAI
        resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"{question}\nNote: if you need external data, say so explicitly."}],
            temperature=0, max_tokens=400
        ).choices[0].message.content
        result = {"answer": resp, "confidence": 0.7}
        technique_used = "react_simplified"
    
    elif task_type == "math" and accuracy_critical:
        # CoT + Self-Consistency
        result = cot_self_consistency(question, n=5)
        technique_used = "cot_sc_n5"
    
    elif task_type in ["math", "logic"]:
        # CoT alone
        resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"{question}\nThink step by step."}],
            temperature=0, max_tokens=400
        ).choices[0].message.content
        result = {"answer": resp, "confidence": 0.8}
        technique_used = "cot_only"
    
    else:
        # Zero-shot/few-shot for factual and creative
        resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": question}],
            temperature=0.3, max_tokens=400
        ).choices[0].message.content
        result = {"answer": resp, "confidence": 0.75}
        technique_used = "zero_shot"
    
    return {
        "answer": result.get("answer", ""),
        "confidence": result.get("confidence", 0.5),
        "classification": clf,
        "technique_used": technique_used
    }


# Try it:
questions = [
    "What is 23% of 1,540?",
    "What are the trends in renewable energy in 2026?",
    "If all A are B, and some B are C, does it necessarily follow that some A are C?"
]

for q in questions:
    r = robust_qa(q)
    print(f"\nQuestion: {q}")
    print(f"Technique: {r['technique_used']}")
    print(f"Answer: {r['answer'][:100]}...")

Troubleshooting

Problem 1: The combination is slower with no noticeable improvement

Symptom: Combining CoT + Self-Consistency for simple tasks improves nothing and multiplies the cost by 5x.

Diagnosis and solution:

def combine_only_if_worth_it(problem: str) -> dict:
    """First try plain CoT; only use SC if the confidence is low."""
    # Attempt 1: plain CoT
    initial_response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"{problem}\nThink step by step. Answer: [value]"}],
        temperature=0, max_tokens=400
    ).choices[0].message.content
    
    # Estimate the model's confidence (through the length of the reasoning)
    # Very short answers or ones with "I don't know" signal low confidence
    low_confidence = (
        len(initial_response.split()) < 30 or
        any(p in initial_response.lower() for p in ["i don't know", "not sure", "unclear", "ambiguous"])
    )
    
    if not low_confidence:
        return {"answer": extract_numeric_answer(initial_response), "technique": "cot_only"}
    
    # If the confidence is low, use Self-Consistency
    result = cot_self_consistency(problem, n=3)
    return {**result, "technique": "cot_sc"}

Problem 2: ReAct + Self-Consistency is too expensive

Symptom: ReAct with 4 steps * 5 samples = 20+ calls per question.

Solution: Use Self-Consistency only on ReAct's final result:

def react_with_verification(problem: str, n_verifications: int = 3) -> dict:
    """ReAct once, then verify the result with SC."""
    # ReAct a single time
    react_result = run_react(problem, verbose=False)
    react_answer = react_result["answer"]
    
    # Verify ReAct's answer with SC
    verifications = []
    for _ in range(n_verifications):
        v = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"Verify whether this answer is correct for the given problem:\nProblem: {problem}\nProposed answer: {react_answer}\n\nReply ONLY: CORRECT or INCORRECT"}],
            temperature=0.3
        ).choices[0].message.content.strip().upper()
        verifications.append("CORRECT" in v)
    
    confidence = sum(verifications) / len(verifications)
    return {
        "answer": react_answer,
        "verified": confidence > 0.6,
        "verification_confidence": confidence,
        "total_calls": react_result.get("steps_used", 4) + n_verifications
    }

Exercises

Exercise 1: Design the optimal combination

For each of these systems, design which combination of techniques you'd use and justify your choice:

a) A math tutoring system for high-school students b) A tech-support chatbot that queries a knowledge base c) A code generator that has to produce bug-free code d) A financial risk analysis system

Exercise 2: Cost benchmark

Implement a cost tracker that records how many API calls each combination makes and computes the estimated cost per problem.

class CostTracker:
    PRICES_PER_1K_TOKENS = {
        "gpt-4o-mini": {"input": 0.00015, "output": 0.00060},
        "gpt-4o": {"input": 0.0025, "output": 0.01}
    }
    
    def __init__(self):
        self.calls = []
    
    def record(self, model_name: str, input_tokens: int, output_tokens: int):
        prices = self.PRICES_PER_1K_TOKENS.get(model_name, self.PRICES_PER_1K_TOKENS["gpt-4o-mini"])
        cost = (input_tokens / 1000 * prices["input"] + 
                output_tokens / 1000 * prices["output"])
        self.calls.append({
            "model": model_name,
            "input_tokens": input_tokens,
            "output_tokens": output_tokens,
            "cost_usd": cost
        })
    
    def total_cost(self) -> float:
        return sum(c["cost_usd"] for c in self.calls)
    
    def summary(self) -> dict:
        return {
            "n_calls": len(self.calls),
            "total_cost_usd": self.total_cost(),
            "average_cost_usd": self.total_cost() / len(self.calls) if self.calls else 0
        }

tracker = CostTracker()
# To use it: every time you make a call, record the tokens
# response = client.chat.completions.create(...)
# tracker.record("gpt-4o-mini", response.usage.prompt_tokens, response.usage.completion_tokens)

Summary

• CoT + Self-Consistency: The most useful combination in production. Each SC sample uses CoT internally. Improves +5-15% with N=5.
• ReAct + Structured Output: For production systems. External data + a parseable output.
• ToT + Self-Consistency: For very hard problems with multiple strategies. Very expensive, use it sparingly.
• Meta-prompting + Self-Refine: For systematic prompt optimization.
• The golden rule: Only combine when each technique solves a distinct, real problem. Don't combine for the sake of combining.

Additional resources

1. ReAct (Yao et al., 2022)
2. Self-Consistency (Wang et al., 2022)
3. Tree of Thoughts (Yao et al., 2023)
4. Self-Refine (Madaan et al., 2023)
5. LLM-Blender: Ensembling LLMs with Pairwise Ranking - Combining the outputs of multiple LLMs
6. Prompt Engineering Guide - Combining Techniques