Module 4: Chain-of-Thought and Reasoning

8. Project: Reasoning Engine with Verifiable CoT

Next →

Overview

Build a complete reasoning engine that solves math and logic problems using CoT, verifies each step, reports a confidence score, and can compare its performance with and without CoT. This project integrates everything you learned in the module: Zero-Shot CoT, Manual CoT, Verification Patterns, and the principles of when not to use CoT.

Estimated time: 120-180 minutes

What you'll build:

  • A ReasoningEngine class with support for multiple domains
  • A multi-step verification system (backward check + constraint verification)
  • Calibrated confidence scoring
  • A CoT vs. no-CoT comparison benchmark
  • A CLI for interactive use

Project Specifications

Inputs

  • Free text: math or logic problems
  • Optional problem type: math, logic, general
  • Verification mode: none, self, backward, two_pass

Outputs

{
    "problema": "...",
    "tipo": "math",
    "razonamiento": "Step 1: ...\nStep 2: ...",
    "respuesta": "391",
    "verificacion": {
        "metodo": "backward",
        "resultado": "VERIFIED",
        "nota": "391/17=23 ✓"
    },
    "confianza": {
        "score": 0.95,
        "nivel": "VERY_HIGH",
        "factores": {
            "certeza_proceso": 1.0,
            "verificabilidad": 1.0,
            "claridad_problema": 0.9,
            "conocimiento_dominio": 0.9
        }
    },
    "tokens_usados": 342,
    "tiempo_ms": 1240
}

Project Structure

reasoning_engine/
├── engine.py          # The main ReasoningEngine class
├── verifiers.py       # Verification module
├── parsers.py         # Extraction of answers from the CoT
├── prompts.py         # Prompt templates per domain
├── benchmark.py       # Benchmark system
├── cli.py             # Command-line interface
└── problems.py        # Test problem dataset

Implementation: prompts.py

# prompts.py
"""CoT prompt templates per domain."""

MATH_COT_PROMPT = """Solve the math problem step by step.

For each step:
1. Describe the operation
2. Show the calculation
3. Write the intermediate result

At the end, write EXACTLY:
ANSWER: [number or expression]
CONFIDENCE: [number between 0 and 1]

Problem: {problema}
"""

LOGIC_COT_PROMPT = """Analyze the logical argument step by step.

1. Identify the PREMISES (numbered list)
2. Identify the CONCLUSION to be evaluated
3. Determine the logical form (modus ponens, modus tollens, syllogism, etc.)
4. Check whether the conclusion follows from the premises
5. Look for counterexamples if you suspect it's invalid

At the end, write EXACTLY:
ANSWER: VALID / INVALID
REASONING: [one line explaining why]
CONFIDENCE: [number between 0 and 1]

Argument: {problema}
"""

GENERAL_COT_PROMPT = """Solve the problem step by step.

Analyze it carefully, show your reasoning, and reach a clear conclusion.

At the end, write EXACTLY:
ANSWER: [your answer]
CONFIDENCE: [number between 0 and 1, where 1 = total certainty]

Problem: {problema}
"""

DIRECT_PROMPT = """Answer directly and concisely.

Problem: {problema}

Answer:"""

VERIFICATION_BACKWARD_MATH = """Verify that this answer is correct by substituting backward.

Original problem: {problema}
Proposed answer: {respuesta}

Verification steps:
1. Take the proposed value
2. Apply it to the original problem
3. Check that every condition holds

Write:
STATUS: VERIFIED / ERROR
DETAILS: [what you verified and how]
CORRECTED_ANSWER: [only if there was an error, the correct answer]
"""

VERIFICATION_LOGIC = """Verify the conclusion of the logical argument.

Original problem: {problema}
Proposed conclusion: {respuesta}

To verify:
1. Does the conclusion follow from the premises by some valid rule?
2. Is there any counterexample where the premises are true and the conclusion is false?

Write:
STATUS: VERIFIED / ERROR
DETAILS: [the inference rule used, or the counterexample you found]
"""

Implementation: parsers.py

# parsers.py
"""Module for extracting answers from CoT outputs."""
import re
from dataclasses import dataclass


@dataclass
class ParsedOutput:
    respuesta: str | None
    confianza: float | None
    razonamiento: str
    parse_exitoso: bool
    metodo_extraccion: str


def parse_respuesta_matematica(output: str) -> ParsedOutput:
    """
    Extracts the numeric answer and the confidence from a math CoT output.
    Tries several strategies in order of reliability.
    """
    razonamiento = output
    
    # Strategy 1: Look for the explicit ANSWER: label
    match_resp = re.search(
        r'^ANSWER:\s*([\d,.$€%\.±\-\+\/]+)',
        output,
        re.MULTILINE | re.IGNORECASE
    )
    
    if match_resp:
        respuesta = match_resp.group(1).strip().replace(',', '')
        
        # Look for the confidence
        match_conf = re.search(
            r'^CONFIDENCE:\s*(0\.\d+|1\.0|1)',
            output,
            re.MULTILINE | re.IGNORECASE
        )
        confianza = float(match_conf.group(1)) if match_conf else None
        
        return ParsedOutput(
            respuesta=respuesta,
            confianza=confianza,
            razonamiento=output,
            parse_exitoso=True,
            metodo_extraccion="explicit_label"
        )
    
    # Strategy 2: Look for conclusion phrases
    patrones_conclusion = [
        r'(?:therefore|thus|in conclusion|the answer is)[,:\s]+\$?([\d,\.]+)',
        r'(?:total|result|sum)[:\s]+\$?([\d,\.]+)',
        r'=\s*\$?([\d,\.]+)\s*$',
    ]
    
    for patron in patrones_conclusion:
        match = re.search(patron, output.lower())
        if match:
            return ParsedOutput(
                respuesta=match.group(1).replace(',', ''),
                confianza=None,
                razonamiento=output,
                parse_exitoso=True,
                metodo_extraccion="conclusion_pattern"
            )
    
    # Strategy 3: The last number in the text (fallback)
    numeros = re.findall(r'\b\d+(?:\.\d+)?\b', output)
    if numeros:
        return ParsedOutput(
            respuesta=numeros[-1],
            confianza=None,
            razonamiento=output,
            parse_exitoso=True,
            metodo_extraccion="last_number_fallback"
        )
    
    return ParsedOutput(
        respuesta=None,
        confianza=None,
        razonamiento=output,
        parse_exitoso=False,
        metodo_extraccion="none"
    )


def parse_respuesta_logica(output: str) -> ParsedOutput:
    """Extracts the VALID/INVALID verdict from a logic CoT output."""
    # Look for the explicit label
    match = re.search(
        r'^ANSWER:\s*(VALID|INVALID)',
        output,
        re.MULTILINE | re.IGNORECASE
    )
    
    if match:
        confianza_match = re.search(
            r'^CONFIDENCE:\s*(0\.\d+|1\.0)',
            output,
            re.MULTILINE | re.IGNORECASE
        )
        return ParsedOutput(
            respuesta=match.group(1).upper(),
            confianza=float(confianza_match.group(1)) if confianza_match else None,
            razonamiento=output,
            parse_exitoso=True,
            metodo_extraccion="explicit_label"
        )
    
    # Search the final part of the text
    ultima_parte = output[-400:].upper()
    if "INVALID" in ultima_parte:
        return ParsedOutput(
            respuesta="INVALID",
            confianza=None,
            razonamiento=output,
            parse_exitoso=True,
            metodo_extraccion="text_search"
        )
    elif "VALID" in ultima_parte:
        return ParsedOutput(
            respuesta="VALID",
            confianza=None,
            razonamiento=output,
            parse_exitoso=True,
            metodo_extraccion="text_search"
        )
    
    return ParsedOutput(
        respuesta=None,
        confianza=None,
        razonamiento=output,
        parse_exitoso=False,
        metodo_extraccion="none"
    )

Implementation: verifiers.py

# verifiers.py
"""Verification system for CoT answers."""
from openai import OpenAI
from dataclasses import dataclass

client = OpenAI()


@dataclass
class VerificationResult:
    metodo: str
    estado: str  # "VERIFIED", "ERROR", "UNDETERMINED"
    detalles: str
    respuesta_corregida: str | None = None
    confianza_verificacion: float | None = None


def verificar_backward_matematico(
    problema: str,
    respuesta: str
) -> VerificationResult:
    """
    Verifies a math answer by substituting backward.
    """
    from prompts import VERIFICATION_BACKWARD_MATH
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": VERIFICATION_BACKWARD_MATH.format(
                problema=problema,
                respuesta=respuesta
            )
        }],
        temperature=0,
        max_tokens=300
    )
    output = response.choices[0].message.content
    
    import re
    estado_match = re.search(r'STATUS:\s*(VERIFIED|ERROR)', output, re.IGNORECASE)
    correccion_match = re.search(r'CORRECTED_ANSWER:\s*(.+?)(?:\n|$)', output, re.IGNORECASE)
    
    estado = estado_match.group(1).upper() if estado_match else "UNDETERMINED"
    
    return VerificationResult(
        metodo="backward_matematico",
        estado=estado,
        detalles=output,
        respuesta_corregida=correccion_match.group(1).strip() if correccion_match and estado == "ERROR" else None,
        confianza_verificacion=1.0 if estado == "VERIFIED" else 0.3 if estado == "ERROR" else 0.5
    )


def verificar_logica(
    problema: str,
    respuesta: str
) -> VerificationResult:
    """
    Verifies a piece of logical reasoning by looking for counterexamples.
    """
    from prompts import VERIFICATION_LOGIC
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": VERIFICATION_LOGIC.format(
                problema=problema,
                respuesta=respuesta
            )
        }],
        temperature=0,
        max_tokens=300
    )
    output = response.choices[0].message.content
    
    import re
    estado_match = re.search(r'STATUS:\s*(VERIFIED|ERROR)', output, re.IGNORECASE)
    estado = estado_match.group(1).upper() if estado_match else "UNDETERMINED"
    
    return VerificationResult(
        metodo="verificacion_logica",
        estado=estado,
        detalles=output,
        confianza_verificacion=1.0 if estado == "VERIFIED" else 0.3
    )


def calcular_confidence_score(
    parsed_output,  # ParsedOutput
    verification_result: VerificationResult,
    tipo_problema: str
) -> dict:
    """
    Computes the final confidence score, combining the CoT confidence + the verification.
    """
    # The CoT's own base score (if the model provided one)
    cot_confidence = parsed_output.confianza or 0.7  # Default 0.7 when there's no score
    
    # Adjustment based on the verification
    if verification_result.estado == "VERIFIED":
        verificacion_factor = 1.0
    elif verification_result.estado == "UNDETERMINED":
        verificacion_factor = 0.8
    else:  # ERROR
        verificacion_factor = 0.2
    
    # Adjustment based on the extraction method
    parse_factor = {
        "explicit_label": 1.0,
        "conclusion_pattern": 0.9,
        "last_number_fallback": 0.7,
        "none": 0.1
    }.get(parsed_output.metodo_extraccion, 0.7)
    
    # Final weighted score
    score_final = cot_confidence * 0.4 + verificacion_factor * 0.4 + parse_factor * 0.2
    score_final = min(1.0, max(0.0, score_final))  # Clamp to [0, 1]
    
    nivel = (
        "VERY_HIGH" if score_final >= 0.9 else
        "HIGH" if score_final >= 0.7 else
        "MEDIUM" if score_final >= 0.5 else
        "LOW"
    )
    
    return {
        "score": round(score_final, 3),
        "nivel": nivel,
        "factores": {
            "cot_confidence": cot_confidence,
            "verificacion_factor": verificacion_factor,
            "parse_factor": parse_factor
        }
    }

Main Implementation: engine.py

# engine.py
"""The main reasoning engine with verifiable CoT."""
import time
import re
from dataclasses import dataclass, field
from enum import Enum
from openai import OpenAI

# Import the project modules
# (In a real file, import from the corresponding modules)
client = OpenAI()


class TipoProblema(Enum):
    MATH = "math"
    LOGIC = "logic"
    GENERAL = "general"


class ModoVerificacion(Enum):
    NONE = "none"
    SELF = "self"
    BACKWARD = "backward"
    TWO_PASS = "two_pass"


@dataclass
class VerificationInfo:
    metodo: str
    estado: str
    detalles: str
    respuesta_corregida: str | None = None


@dataclass
class ResultadoReasoning:
    problema: str
    tipo: TipoProblema
    razonamiento: str
    respuesta: str | None
    verificacion: VerificationInfo | None
    confianza: dict
    tokens_usados: int
    tiempo_ms: int
    modo_cot: str
    parse_exitoso: bool


class ReasoningEngine:
    """
    A reasoning engine with verifiable CoT.
    
    Supports multiple problem types and verification modes,
    and produces calibrated confidence scores.
    
    Usage example:
        engine = ReasoningEngine()
        resultado = engine.resolver("What is 17 × 23?")
        print(resultado.respuesta)  # "391"
        print(resultado.confianza["nivel"])  # "VERY_HIGH"
    """
    
    PROMPTS = {
        TipoProblema.MATH: """Solve the math problem step by step.

For each step:
1. Describe the operation
2. Show the calculation  
3. Write the intermediate result

At the end, write EXACTLY:
ANSWER: [number or expression]
CONFIDENCE: [number between 0 and 1]

Problem: {problema}
""",
        TipoProblema.LOGIC: """Analyze the logical argument step by step.

1. List the PREMISES
2. Identify the CONCLUSION
3. Determine whether the conclusion follows necessarily
4. Look for counterexamples if you suspect it's invalid

At the end, write EXACTLY:
ANSWER: VALID / INVALID
REASONING: [one line]
CONFIDENCE: [number between 0 and 1]

Argument: {problema}
""",
        TipoProblema.GENERAL: """Solve it step by step.

At the end, write EXACTLY:
ANSWER: [your answer]
CONFIDENCE: [number between 0 and 1]

Problem: {problema}
"""
    }
    
    def __init__(
        self,
        model: str = "gpt-4o-mini",
        temperatura: float = 0.0,
        max_tokens: int = 700
    ):
        self.model = model
        self.temperatura = temperatura
        self.max_tokens = max_tokens
    
    def detectar_tipo(self, problema: str) -> TipoProblema:
        """Automatically detects the type of problem."""
        problema_lower = problema.lower()
        
        keywords_math = [
            "how much", "calculate", "solve", "how many", "sum", "subtract",
            "multiply", "divide", "percent", "discount", "price",
            "equation", "algebra", "integral", "derivative", "+", "-", "×"
        ]
        keywords_logic = [
            "if...then", "therefore", "implies", "valid", "invalid",
            "premise", "conclusion", "all the", "some", "no ",
            "logic", "argument", "does it follow"
        ]
        
        score_math = sum(1 for k in keywords_math if k in problema_lower)
        score_logic = sum(1 for k in keywords_logic if k in problema_lower)
        
        if score_math > score_logic:
            return TipoProblema.MATH
        elif score_logic > 0:
            return TipoProblema.LOGIC
        else:
            return TipoProblema.GENERAL
    
    def _construir_prompt_cot(self, problema: str, tipo: TipoProblema) -> str:
        """Builds the CoT prompt appropriate for the problem type."""
        template = self.PROMPTS[tipo]
        return template.format(problema=problema)
    
    def _parsear_output(self, output: str, tipo: TipoProblema) -> tuple[str | None, float | None, bool]:
        """
        Extracts the answer and the confidence from the CoT output.
        
        Returns:
            tuple of (answer, confidence, parse_succeeded)
        """
        # Look for the ANSWER: label
        match_resp = re.search(
            r'^ANSWER:\s*(.+?)$',
            output,
            re.MULTILINE | re.IGNORECASE
        )
        
        # Look for the CONFIDENCE: label
        match_conf = re.search(
            r'^CONFIDENCE:\s*(0\.\d+|1\.0|1)',
            output,
            re.MULTILINE | re.IGNORECASE
        )
        
        respuesta = None
        confianza = None
        parse_exitoso = False
        
        if match_resp:
            respuesta = match_resp.group(1).strip()
            parse_exitoso = True
        else:
            # Fallback: look for conclusion patterns
            if tipo == TipoProblema.MATH:
                numeros = re.findall(r'\b\d+(?:\.\d+)?\b', output)
                respuesta = numeros[-1] if numeros else None
                parse_exitoso = respuesta is not None
            elif tipo == TipoProblema.LOGIC:
                ultima = output[-300:].upper()
                if "INVALID" in ultima:
                    respuesta = "INVALID"
                    parse_exitoso = True
                elif "VALID" in ultima:
                    respuesta = "VALID"
                    parse_exitoso = True
        
        if match_conf:
            try:
                confianza = float(match_conf.group(1))
            except ValueError:
                confianza = None
        
        return respuesta, confianza, parse_exitoso
    
    def _verificar_respuesta(
        self,
        problema: str,
        respuesta: str,
        tipo: TipoProblema,
        modo: ModoVerificacion
    ) -> VerificationInfo | None:
        """Verifies the answer according to the selected mode."""
        if modo == ModoVerificacion.NONE:
            return None
        
        if modo == ModoVerificacion.SELF:
            prompt = f"""Verify whether this answer is correct.
Problem: {problema}
Proposed answer: {respuesta}
Is it correct? Respond VERIFIED or ERROR. Explain briefly."""
            
            r = client.chat.completions.create(
                model=self.model,
                messages=[{"role": "user", "content": prompt}],
                temperature=0, max_tokens=200
            )
            output_v = r.choices[0].message.content
            estado = "VERIFIED" if "verified" in output_v.lower() else "UNDETERMINED"
            return VerificationInfo("self_verification", estado, output_v)
        
        elif modo == ModoVerificacion.BACKWARD:
            if tipo == TipoProblema.MATH:
                prompt = f"""Verify by substituting the answer back into the original problem.
Problem: {problema}
Answer: {respuesta}
Substitute and verify. Write VERIFIED ✓ or ERROR ✗ + the correction if it applies."""
            else:
                prompt = f"""Verify this logical argument by looking for counterexamples.
Argument: {problema}
Conclusion: {respuesta}
Is there a counterexample? Write VERIFIED ✓ or ERROR ✗."""
            
            r = client.chat.completions.create(
                model=self.model,
                messages=[{"role": "user", "content": prompt}],
                temperature=0, max_tokens=300
            )
            output_v = r.choices[0].message.content
            
            if "✓" in output_v or "verified" in output_v.lower():
                estado = "VERIFIED"
            elif "✗" in output_v or "error" in output_v.lower():
                estado = "ERROR"
                # Try to extract the correction
                match_corr = re.search(r'(?:correction|correct)[:\s]+(.+?)(?:\n|$)', output_v, re.IGNORECASE)
                correccion = match_corr.group(1).strip() if match_corr else None
                return VerificationInfo("backward", estado, output_v, correccion)
            else:
                estado = "UNDETERMINED"
            
            return VerificationInfo("backward", estado, output_v)
        
        elif modo == ModoVerificacion.TWO_PASS:
            prompt = f"""You are an independent auditor. Solve the problem YOURSELF 
and check whether the proposed solution is correct.

Problem: {problema}
Proposed solution: {respuesta}

Your independent solution:
[reason here]

Verdict: CORRECT or INCORRECT
If INCORRECT: CORRECT_SOLUTION: [answer]"""
            
            r = client.chat.completions.create(
                model=self.model,
                messages=[{"role": "user", "content": prompt}],
                temperature=0, max_tokens=400
            )
            output_v = r.choices[0].message.content
            
            if "correct" in output_v.lower() and "incorrect" not in output_v.lower():
                estado = "VERIFIED"
            elif "incorrect" in output_v.lower():
                estado = "ERROR"
                match_sol = re.search(r'CORRECT_SOLUTION:\s*(.+?)(?:\n|$)', output_v, re.IGNORECASE)
                correccion = match_sol.group(1).strip() if match_sol else None
                return VerificationInfo("two_pass", estado, output_v, correccion)
            else:
                estado = "UNDETERMINED"
            
            return VerificationInfo("two_pass", estado, output_v)
        
        return None
    
    def _calcular_confianza(
        self,
        confianza_cot: float | None,
        verificacion: VerificationInfo | None,
        parse_exitoso: bool
    ) -> dict:
        """Computes the final confidence score."""
        base = confianza_cot or 0.65
        
        verificacion_factor = 1.0
        if verificacion:
            if verificacion.estado == "VERIFIED":
                verificacion_factor = 1.0
            elif verificacion.estado == "UNDETERMINED":
                verificacion_factor = 0.8
            else:  # ERROR
                verificacion_factor = 0.2
        
        parse_factor = 1.0 if parse_exitoso else 0.4
        
        score = base * 0.4 + verificacion_factor * 0.4 + parse_factor * 0.2
        score = round(min(1.0, max(0.0, score)), 3)
        
        nivel = (
            "VERY_HIGH" if score >= 0.9 else
            "HIGH" if score >= 0.7 else
            "MEDIUM" if score >= 0.5 else
            "LOW"
        )
        
        return {
            "score": score,
            "nivel": nivel,
            "factores": {
                "cot_confidence": base,
                "verificacion": verificacion_factor,
                "parse": parse_factor
            }
        }
    
    def resolver(
        self,
        problema: str,
        tipo: TipoProblema | None = None,
        verificacion: ModoVerificacion = ModoVerificacion.BACKWARD,
        usar_cot: bool = True
    ) -> ResultadoReasoning:
        """
        Solves a problem with optional CoT and verification.
        
        Args:
            problema: The problem to solve
            tipo: The problem type (auto-detected if None)
            verificacion: The verification mode to use
            usar_cot: If False, answers directly with no CoT
        
        Returns:
            ResultadoReasoning with every detail
        """
        t_inicio = time.time()
        
        # Detect the type if it wasn't specified
        tipo_detectado = tipo or self.detectar_tipo(problema)
        
        # Build the prompt
        if usar_cot:
            prompt = self._construir_prompt_cot(problema, tipo_detectado)
            modo_cot = "chain_of_thought"
        else:
            prompt = f"Answer directly: {problema}"
            modo_cot = "direct"
        
        # Call the model
        response = client.chat.completions.create(
            model=self.model,
            messages=[{"role": "user", "content": prompt}],
            temperature=self.temperatura,
            max_tokens=self.max_tokens
        )
        
        output = response.choices[0].message.content
        tokens = response.usage.total_tokens
        
        # Parse the answer
        respuesta, confianza_cot, parse_ok = self._parsear_output(output, tipo_detectado)
        
        # Verify if CoT is active and there is an answer
        verificacion_result = None
        tokens_verificacion = 0
        if usar_cot and respuesta and verificacion != ModoVerificacion.NONE:
            verificacion_result = self._verificar_respuesta(
                problema, respuesta, tipo_detectado, verificacion
            )
            # Use the corrected answer if the verification found an error
            if verificacion_result and verificacion_result.respuesta_corregida:
                respuesta = verificacion_result.respuesta_corregida
        
        # Compute the final confidence
        confianza_info = self._calcular_confianza(confianza_cot, verificacion_result, parse_ok)
        
        tiempo_ms = int((time.time() - t_inicio) * 1000)
        
        return ResultadoReasoning(
            problema=problema,
            tipo=tipo_detectado,
            razonamiento=output,
            respuesta=respuesta,
            verificacion=verificacion_result,
            confianza=confianza_info,
            tokens_usados=tokens + tokens_verificacion,
            tiempo_ms=tiempo_ms,
            modo_cot=modo_cot,
            parse_exitoso=parse_ok
        )

Implementation: benchmark.py

# benchmark.py
"""Benchmark system for comparing CoT vs. no-CoT."""
from dataclasses import dataclass
from engine import ReasoningEngine, TipoProblema, ModoVerificacion


@dataclass
class ResultadoBenchmark:
    total_problemas: int
    correctos_cot: int
    correctos_sin_cot: int
    accuracy_cot: float
    accuracy_sin_cot: float
    mejora_accuracy: float
    tokens_promedio_cot: float
    tokens_promedio_sin_cot: float
    multiplicador_tokens: float
    tiempo_promedio_cot_ms: float
    tiempo_promedio_sin_cot_ms: float


# Dataset of problems with known answers
PROBLEMAS_MATH = [
    # (problem, correct_answer)
    ("What is 17 × 23?", "391"),
    ("A product costs $80 with a 20% discount. What is the original price?", "100"),
    ("If A finishes in 6h and B in 4h, how long do they take together?", "2.4"),
    ("What is 15% of $240?", "36"),
    ("Train A leaves at 80 km/h. Train B leaves 30 min later at 120 km/h. When does B catch A?", "1"),
    ("How many integers from 1 to 50 are divisible by 3 or by 5?", "23"),
    ("I invest $1000 at 5% compounded annually for 2 years. How much do I have?", "1102.5"),
    ("The hypotenuse of a right triangle with legs 6 and 8 is:", "10"),
]

PROBLEMAS_LOGIC = [
    # (problem, correct_answer)
    ("If P then Q. P is true. Is Q true?", "VALID"),
    ("If it rains, the ground gets wet. The ground is wet. Is it raining?", "INVALID"),
    ("All cats are animals. Misi is a cat. Is Misi an animal?", "VALID"),
    ("Some dogs bark. Rex is a dog. Does Rex bark?", "INVALID"),
    ("If I don't study, I fail. I don't fail. Did I study?", "VALID"),
]


def ejecutar_benchmark(
    n_problemas_math: int = 5,
    n_problemas_logic: int = 3,
    modo_verificacion: ModoVerificacion = ModoVerificacion.BACKWARD
) -> ResultadoBenchmark:
    """
    Runs the complete CoT vs. no-CoT benchmark.
    
    Args:
        n_problemas_math: Number of math problems to use
        n_problemas_logic: Number of logic problems to use
        modo_verificacion: The verification mode for the CoT engine
    
    Returns:
        ResultadoBenchmark with detailed statistics
    """
    engine = ReasoningEngine()
    
    todos_problemas = (
        [(p, r, TipoProblema.MATH) for p, r in PROBLEMAS_MATH[:n_problemas_math]] +
        [(p, r, TipoProblema.LOGIC) for p, r in PROBLEMAS_LOGIC[:n_problemas_logic]]
    )
    
    correctos_cot = 0
    correctos_sin_cot = 0
    tokens_cot_total = 0
    tokens_sin_cot_total = 0
    tiempos_cot = []
    tiempos_sin_cot = []
    
    print(f"\n{'='*70}")
    print(f"BENCHMARK: CoT vs. No CoT ({len(todos_problemas)} problems)")
    print(f"{'='*70}")
    
    for problema, respuesta_correcta, tipo in todos_problemas:
        print(f"\n► {problema[:60]}...")
        
        # With CoT
        r_cot = engine.resolver(
            problema,
            tipo=tipo,
            verificacion=modo_verificacion,
            usar_cot=True
        )
        
        # Without CoT
        r_sin = engine.resolver(
            problema,
            tipo=tipo,
            verificacion=ModoVerificacion.NONE,
            usar_cot=False
        )
        
        # Check whether they're correct
        cot_ok = respuesta_correcta in str(r_cot.respuesta or "")
        sin_ok = respuesta_correcta in str(r_sin.respuesta or "")
        
        if cot_ok:
            correctos_cot += 1
        if sin_ok:
            correctos_sin_cot += 1
        
        tokens_cot_total += r_cot.tokens_usados
        tokens_sin_cot_total += r_sin.tokens_usados
        tiempos_cot.append(r_cot.tiempo_ms)
        tiempos_sin_cot.append(r_sin.tiempo_ms)
        
        print(f"  With CoT:    {'✓' if cot_ok else '✗'} | Ans: {r_cot.respuesta} | Confidence: {r_cot.confianza['nivel']} | {r_cot.tokens_usados} tokens")
        print(f"  Without CoT: {'✓' if sin_ok else '✗'} | Ans: {r_sin.respuesta} | {r_sin.tokens_usados} tokens")
    
    n = len(todos_problemas)
    acc_cot = correctos_cot / n
    acc_sin = correctos_sin_cot / n
    
    resultado = ResultadoBenchmark(
        total_problemas=n,
        correctos_cot=correctos_cot,
        correctos_sin_cot=correctos_sin_cot,
        accuracy_cot=acc_cot,
        accuracy_sin_cot=acc_sin,
        mejora_accuracy=acc_cot - acc_sin,
        tokens_promedio_cot=tokens_cot_total / n,
        tokens_promedio_sin_cot=tokens_sin_cot_total / n,
        multiplicador_tokens=(tokens_cot_total / n) / max(tokens_sin_cot_total / n, 1),
        tiempo_promedio_cot_ms=sum(tiempos_cot) / n,
        tiempo_promedio_sin_cot_ms=sum(tiempos_sin_cot) / n
    )
    
    print(f"\n{'='*70}")
    print("BENCHMARK SUMMARY")
    print(f"{'='*70}")
    print(f"Accuracy with CoT:    {acc_cot:.1%} ({correctos_cot}/{n})")
    print(f"Accuracy without CoT: {acc_sin:.1%} ({correctos_sin_cot}/{n})")
    print(f"CoT improvement:      +{resultado.mejora_accuracy:.1%}")
    print(f"Tokens with CoT:      {resultado.tokens_promedio_cot:.0f} (average)")
    print(f"Tokens without CoT:   {resultado.tokens_promedio_sin_cot:.0f} (average)")
    print(f"CoT multiplier:       {resultado.multiplicador_tokens:.1f}x more tokens")
    
    return resultado

Implementation: cli.py

# cli.py
"""Command-line interface for the Reasoning Engine."""
import sys
from engine import ReasoningEngine, TipoProblema, ModoVerificacion


def main():
    """Interactive CLI for the Reasoning Engine."""
    engine = ReasoningEngine()
    
    print("\n" + "="*60)
    print("REASONING ENGINE - Verifiable CoT")
    print("="*60)
    print("Commands:")
    print("  [problem]      → Solve with the default configuration")
    print("  :type math     → Set the type (math/logic/general)")
    print("  :verification backward → Mode (none/self/backward/two_pass)")
    print("  :benchmark     → Run the complete benchmark")
    print("  :exit          → Exit")
    print("="*60)
    
    tipo_actual = None
    verificacion_actual = ModoVerificacion.BACKWARD
    
    while True:
        try:
            entrada = input("\n> ").strip()
        except (KeyboardInterrupt, EOFError):
            print("\nSee you!")
            sys.exit(0)
        
        if not entrada:
            continue
        
        if entrada.lower() == ":exit":
            print("See you!")
            break
        
        elif entrada.lower().startswith(":type "):
            tipo_str = entrada.split(" ", 1)[1].strip()
            mapa = {"math": TipoProblema.MATH, "logic": TipoProblema.LOGIC, "general": TipoProblema.GENERAL}
            if tipo_str in mapa:
                tipo_actual = mapa[tipo_str]
                print(f"Type set to: {tipo_actual.value}")
            else:
                print(f"Type not recognized. Options: {list(mapa.keys())}")
        
        elif entrada.lower().startswith(":verification "):
            modo_str = entrada.split(" ", 1)[1].strip()
            mapa_v = {
                "none": ModoVerificacion.NONE,
                "self": ModoVerificacion.SELF,
                "backward": ModoVerificacion.BACKWARD,
                "two_pass": ModoVerificacion.TWO_PASS
            }
            if modo_str in mapa_v:
                verificacion_actual = mapa_v[modo_str]
                print(f"Verification: {verificacion_actual.value}")
            else:
                print(f"Mode not recognized. Options: {list(mapa_v.keys())}")
        
        elif entrada.lower() == ":benchmark":
            from benchmark import ejecutar_benchmark
            ejecutar_benchmark(n_problemas_math=5, n_problemas_logic=3)
        
        else:
            # Solve the problem
            print(f"\n[Solving with CoT + {verificacion_actual.value} verification...]")
            
            resultado = engine.resolver(
                entrada,
                tipo=tipo_actual,
                verificacion=verificacion_actual
            )
            
            print(f"\n── REASONING ──")
            print(resultado.razonamiento)
            
            print(f"\n── RESULT ──")
            print(f"Detected type:     {resultado.tipo.value}")
            print(f"Answer:            {resultado.respuesta}")
            print(f"Confidence:        {resultado.confianza['score']} ({resultado.confianza['nivel']})")
            
            if resultado.verificacion:
                estado_emoji = "✓" if resultado.verificacion.estado == "VERIFIED" else "✗" if resultado.verificacion.estado == "ERROR" else "?"
                print(f"Verification:      {estado_emoji} {resultado.verificacion.estado} ({resultado.verificacion.metodo})")
            
            print(f"Tokens used:       {resultado.tokens_usados}")
            print(f"Time:              {resultado.tiempo_ms}ms")


if __name__ == "__main__":
    main()

Running It and Demo

# demo.py - Run this to see the engine in action
from engine import ReasoningEngine, TipoProblema, ModoVerificacion
from benchmark import ejecutar_benchmark

def demo_basico():
    """Demonstrates the basic usage of the Reasoning Engine."""
    engine = ReasoningEngine()
    
    print("=" * 60)
    print("DEMO: Reasoning Engine with Verifiable CoT")
    print("=" * 60)
    
    # A math problem
    print("\n1. MATH PROBLEM")
    resultado = engine.resolver(
        "If I invest $5,000 at 8% compounded annually for 3 years, how much do I have?",
        tipo=TipoProblema.MATH,
        verificacion=ModoVerificacion.BACKWARD
    )
    print(f"Answer: {resultado.respuesta}")
    print(f"Confidence: {resultado.confianza['score']} ({resultado.confianza['nivel']})")
    verificacion_estado = resultado.verificacion.estado if resultado.verificacion else "N/A"
    print(f"Verification: {verificacion_estado}")
    # Expected answer: $6,298.56 (approximately)
    
    # A logic problem
    print("\n2. LOGIC PROBLEM")
    resultado_logico = engine.resolver(
        "If you study, you pass the exam. You didn't pass the exam. Did you study?",
        tipo=TipoProblema.LOGIC,
        verificacion=ModoVerificacion.BACKWARD
    )
    print(f"Answer: {resultado_logico.respuesta}")
    print(f"Confidence: {resultado_logico.confianza['score']}")
    # Expected answer: VALID (modus tollens)
    
    # CoT vs. no-CoT comparison
    print("\n3. COMPARISON CoT vs. No CoT")
    
    problema_test = "What is 23% of $450?"
    
    r_cot = engine.resolver(problema_test, usar_cot=True, verificacion=ModoVerificacion.NONE)
    r_sin = engine.resolver(problema_test, usar_cot=False, verificacion=ModoVerificacion.NONE)
    
    print(f"With CoT:    {r_cot.respuesta} ({r_cot.tokens_usados} tokens)")
    print(f"Without CoT: {r_sin.respuesta} ({r_sin.tokens_usados} tokens)")
    print(f"Expected answer: 103.5")


def demo_benchmark():
    """Runs the complete benchmark."""
    resultados = ejecutar_benchmark(
        n_problemas_math=5,
        n_problemas_logic=3,
        modo_verificacion=ModoVerificacion.BACKWARD
    )
    
    print(f"\nCONCLUSION:")
    if resultados.mejora_accuracy > 0.1:
        print(f"CoT improves accuracy significantly (+{resultados.mejora_accuracy:.0%})")
        print(f"The extra cost ({resultados.multiplicador_tokens:.1f}x tokens) is justified for this kind of problem.")
    elif resultados.mejora_accuracy > 0:
        print(f"CoT improves accuracy modestly (+{resultados.mejora_accuracy:.0%})")
        print(f"Consider whether the extra cost is worth it for your use case.")
    else:
        print(f"CoT did not improve accuracy in this benchmark.")
        print(f"Check whether these problems are appropriate for CoT.")


if __name__ == "__main__":
    print("Running the basic demo...")
    demo_basico()
    
    print("\n\nRunning the benchmark...")
    demo_benchmark()

Project Success Criteria

Check off each criterion as you complete it:

  • Basic solving: The engine correctly solves 8/10 simple math problems
  • Problem types: It auto-detects the type (math/logic) with ≥80% accuracy
  • Backward verification: The verification detects errors and corrects the answer when possible
  • Confidence score: The score reflects the real difficulty (simple problems → high confidence)
  • Benchmark: It shows a measurable improvement of CoT vs. no CoT on the dataset's problems
  • Working CLI: The CLI accepts free-text problems and shows formatted results
  • Error handling: The engine doesn't crash if the API doesn't respond or if it can't parse the answer
  • Extensibility: It's easy to add a new problem type (e.g. PROBABILITY)

Optional Extensions

Extension 1: Answer Cache

import hashlib
import json
import os

class ReasoningEngineConCache(ReasoningEngine):
    """An extension of the engine with an on-disk cache for development."""
    
    def __init__(self, cache_path: str = ".reasoning_cache.json", **kwargs):
        super().__init__(**kwargs)
        self.cache_path = cache_path
        self._cache = self._cargar_cache()
    
    def _cargar_cache(self) -> dict:
        if os.path.exists(self.cache_path):
            with open(self.cache_path, 'r') as f:
                return json.load(f)
        return {}
    
    def _guardar_cache(self):
        with open(self.cache_path, 'w') as f:
            json.dump(self._cache, f, indent=2, ensure_ascii=False)
    
    def resolver(self, problema: str, **kwargs) -> 'ResultadoReasoning':
        cache_key = hashlib.md5(f"{problema}{kwargs}".encode()).hexdigest()
        
        if cache_key in self._cache:
            print("[From cache]")
            # Here you'd rebuild the ResultadoReasoning from the cache
            # Simplified for this example
        
        resultado = super().resolver(problema, **kwargs)
        
        # Store in the cache
        self._cache[cache_key] = {
            "problema": problema,
            "respuesta": resultado.respuesta,
            "confianza": resultado.confianza,
            "tokens": resultado.tokens_usados
        }
        self._guardar_cache()
        
        return resultado

Extension 2: Batch Processing

from concurrent.futures import ThreadPoolExecutor, as_completed

def resolver_batch(
    problemas: list[str],
    engine: ReasoningEngine,
    max_workers: int = 3
) -> list[ResultadoReasoning]:
    """
    Solves multiple problems in parallel using ThreadPoolExecutor.
    
    Note: The OpenAI API has rate limits; max_workers should stay conservative.
    """
    resultados = [None] * len(problemas)
    
    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        futuras = {
            executor.submit(engine.resolver, p): i 
            for i, p in enumerate(problemas)
        }
        
        for futura in as_completed(futuras):
            idx = futuras[futura]
            try:
                resultados[idx] = futura.result()
                print(f"✓ Problem {idx + 1} complete")
            except Exception as e:
                print(f"✗ Problem {idx + 1} failed: {e}")
    
    return [r for r in resultados if r is not None]

Project Summary

  • ReasoningEngine integrates Zero-Shot CoT, type detection, and multiple verification modes
  • Verification: none → free; self → cheap; backward → balanced; two_pass → more robust but 2x the cost
  • Confidence scoring: Combines the CoT's own score + the verification result + the quality of the parsing
  • Benchmark: Measures the real improvement CoT gives you on your specific dataset
  • CLI: Enables interactive experimentation without writing code
  • Extensibility: The architecture makes it easy to add new problem types and verification modes

Additional resources

  1. Chain-of-Thought Prompting (Wei et al., 2022)
  2. Large Language Models are Zero-Shot Reasoners (Kojima et al., 2022)
  3. Self-Consistency Improves CoT (Wang et al., 2022)
  4. GSM8K Dataset - Grade School Math Problems
  5. OpenAI Python SDK - async support
  6. Pydantic v2 Documentation
  7. Python dataclasses - Official Docs
← Previous7. Limitations and Anti-Patterns of CoTNext →1. Introduction: Beyond Chain-of-Thought