1. Introduction: Why LLMs Need to Think Step by Step

Overview

Chain-of-Thought (CoT) is a technique that dramatically improves an LLM's reasoning by asking it to explain its process step by step. In this capsule you'll understand the original paper (Wei et al., 2022), the paradox of models with 100B+ parameters that fail at simple arithmetic without CoT, and how CoT unlocks latent capabilities the model already has but doesn't express.

Why it matters: Without CoT, a model can give you the wrong answer to a logic or math problem without you knowing why. With CoT, the reasoning is visible, verifiable, and it fixes many errors by forcing the model to generate intermediate tokens that act as a "mental scratchpad".

Estimated time: 45-60 minutes

The Paradox of the Giant Model That Fails at Arithmetic

Picture this: GPT-4, a model trained on trillions of tokens of human text, capable of writing code, explaining quantum physics and debating philosophy... fails to compute 17 × 23.

How is that possible?

Models like GPT-4 can fail at:

• 15 + 27 = ? (sometimes they answer 42, sometimes 43)
• "If A implies B, and B is false, what happens to A?"
• Multi-step chained problems
• Age, speed and percentage problems

But when you ask them to "Think step by step", accuracy improves 15-30% on reasoning tasks.

That's the paradox: the model knows how to solve the problem, but the token-by-token prediction format makes it "take shortcuts". CoT forces the model to show its work, like a student who can't just write the final answer on an exam.

Why Does This Happen? The Underlying Mechanism

LLMs generate tokens one by one, and each token depends on the previous ones. When you answer directly:

Question: What is 17 × 23?
Answer: [token for number] → 391

The model tries to "guess" the right number based on statistical patterns from its training.

But with CoT:

Question: What is 17 × 23? Think step by step.
Answer:
- Break it down: 17 × 23 = 17 × 20 + 17 × 3
- 17 × 20 = 340
- 17 × 3 = 51
- 340 + 51 = 391
- Therefore: 391

Here, each step generates tokens that serve as enriched context for the tokens that follow. The intermediate tokens ("340", "51") are already in context when the model computes the final sum, which reduces errors.

Human analogy: It's the difference between doing math in your head and writing it down on paper. The paper (the intermediate tokens) acts as external memory.

The Original Paper: Chain-of-Thought Prompting (2022)

Reference: Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., ... & Zhou, D. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022.

In this foundational paper, Wei et al. showed that adding examples with explicit reasoning significantly improves model capabilities in:

• Arithmetic: GSM8K, MATH benchmark
• Logical reasoning: SVAMP, MultiArith
• Common sense: CommonsenseQA, StrategyQA
• Symbolic reasoning: Coin flip, last letter concatenation

Key Findings from the Paper

Example from the Paper

The paper showed examples like this one:

Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls.
   Each can has 3 tennis balls. How many tennis balls does he have now?

A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls.
   5 + 6 = 11. The answer is 11.

Compared to the standard approach that would only say "11", this format forces the model to show every step.

Zero-Shot CoT: Kojima's Discovery (2022)

A year later, Kojima et al. (2022) made a surprising discovery: you don't need examples. Just adding the magic phrase is enough:

"Let's think step by step."

This is called Zero-Shot CoT and it works because:

1. The model has seen thousands of texts where explicit reasoning precedes correct answers
2. The phrase activates that generation pattern
3. It doesn't require designing domain-specific examples

Reported improvements (Zero-Shot CoT vs. standard Zero-Shot):

• MultiArith: 17.7% → 78.7%
• GSM8K: 10.4% → 40.7%
• SVAMP: 67.7% → 74.2%

CoT vs. Standard Prompting: Side by Side

The 4 Types of CoT

In this module we'll explore four main variants:

1. Zero-Shot CoT

The simplest one. You just add "Let's think step by step".

prompt = f"{question}\n\nLet's think step by step."

2. Manual CoT (Few-Shot CoT)

You provide examples with explicit reasoning in the Input → Reasoning → Answer format.

prompt = f"""
Example:
Input: {example_1}
Reasoning: {reasoning_1}
Answer: {answer_1}

Input: {new_question}
Reasoning:
"""

3. Self-Consistency CoT

You generate multiple reasoning chains and pick the most common answer (majority vote).

4. Automatic CoT

The model automatically generates the reasoning examples using clustering.

When Should You Use CoT?

✅ Use CoT when:

• The problem requires multiple steps (arithmetic, algebra, geometry)
• There's logical reasoning involved (implications, deductions)
• You need to verify the process, not just the answer
• The problem is ambiguous and you want to see how the model interprets it
• You're analyzing code or debugging

❌ Do NOT use CoT when:

• The task is creative (poetry, brainstorming, metaphors)
• It's simple classification with clear criteria
• The answer is a direct fact ("Capital of Spain?")
• Cost matters and the task is easy
• Latency is critical and you have thousands of calls

Minimal Implementation with OpenAI

Here's the most basic code to start experimenting with CoT:

from openai import OpenAI

client = OpenAI()  # Make sure you have OPENAI_API_KEY in your environment

def compare_with_without_cot(question: str) -> dict:
    """
    Compares a model's answer with and without CoT.
    
    Args:
        question: The problem to solve
    
    Returns:
        dict with the 'without_cot' and 'with_cot' answers
    """
    # Without CoT
    direct_response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": question}],
        temperature=0,
        max_tokens=100
    )
    
    # With CoT
    cot_response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"{question}\n\nLet's think step by step."}],
        temperature=0,
        max_tokens=500
    )
    
    return {
        "without_cot": direct_response.choices[0].message.content,
        "with_cot": cot_response.choices[0].message.content
    }


# Example usage
if __name__ == "__main__":
    problems = [
        "If I have 17 apples and give 30% of them to Maria, how many do I have left?",
        "A train leaves A at 60 km/h. Another leaves B (300 km away) at 90 km/h in the opposite direction. When do they meet?",
        "If all cats are mammals, and some mammals fly, can some cats fly?"
    ]
    
    for problem in problems:
        print(f"\nProblem: {problem}")
        result = compare_with_without_cot(problem)
        print(f"\nWithout CoT:\n{result['without_cot']}")
        print(f"\nWith CoT:\n{result['with_cot']}")
        print("-" * 60)

Emergence with Scale: An Important Phenomenon

Wei et al. discovered something fascinating: CoT only emerges as useful in large models.

In small models (< 8B parameters), adding CoT sometimes hurts the results, because the model can't generate coherent reasoning.

Practical implication: If you use GPT-4o-mini, CoT works well. If you use smaller local models, verify that the model is capable enough before relying on CoT.

Module 4 Roadmap

This module takes you from the basics to advanced CoT prompting:

By the end of the module you'll know how to:

• Decide when and how to apply CoT
• Design prompts with explicit reasoning per domain
• Verify and validate the model's reasoning
• Build robust multi-stage pipelines
• Detect and avoid common anti-patterns

Exercises for This Capsule

Exercise 1: Your first CoT experiment

Pick 3 arithmetic problems of different difficulty. Run each one with and without "Let's think step by step". Record whether CoT improves the result or not.

from openai import OpenAI

client = OpenAI()

problems_with_answer = [
    ("What is 15 + 27?", "42"),
    ("What is 17 × 23?", "391"),
    ("If I have $150 and spend 40%, how much do I have left?", "90"),
]

def solve(question: str, cot: bool) -> str:
    content = question
    if cot:
        content += "\n\nLet's think step by step."
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": content}],
        temperature=0,
        max_tokens=300
    )
    return response.choices[0].message.content

hits_without_cot = 0
hits_with_cot = 0

for question, correct in problems_with_answer:
    r_without = solve(question, cot=False)
    r_with = solve(question, cot=True)
    
    if correct in r_without:
        hits_without_cot += 1
    if correct in r_with:
        hits_with_cot += 1
    
    print(f"\n=== {question} ===")
    print(f"Without CoT: {r_without[:80]}...")
    print(f"With CoT: {r_with[:80]}...")

print(f"\nAccuracy without CoT: {hits_without_cot}/{len(problems_with_answer)}")
print(f"Accuracy with CoT: {hits_with_cot}/{len(problems_with_answer)}")

Exercise 2: Identify the mechanism

Given this CoT output, identify the "intermediate tokens" that act as working memory and explain how each one helps the next step.

Problem: If there are 5 teams in a tournament and each team plays every other team once, how many matches are there in total?

Answer with CoT:
- Team 1 plays against 4 teams: 4 matches
- Team 2 plays against the 3 remaining ones (it already played team 1): 3 matches  
- Team 3 plays against the 2 remaining ones: 2 matches
- Team 4 plays against the 1 remaining one: 1 match
- Total: 4 + 3 + 2 + 1 = 10 matches

Exercise 3: Design a CoT use case in your own domain

Think of a problem from your job or field of interest where CoT could help. Describe:

• The type of problem
• Why it requires multiple steps
• What you expect to gain from CoT

Exercise 4: The limits of CoT

Try CoT on these 3 tasks and watch what happens. Which ones improve? Which ones don't?

1. "Write a haiku about autumn"
2. "Is this argument valid: All dogs are animals. Rex is an animal. Therefore Rex is a dog?"
3. "What's the capital of Japan?"

Key Concepts from This Capsule

Summary

• CoT forces the model to generate intermediate tokens that act as a mental scratchpad
• The paradox exists because models take shortcuts without CoT, even when they have the knowledge
• The mechanism is simple: more relevant context in the input of the next token = better prediction
• Typical improvement: +15-30% on math/logic with CoT
• Limits: It only works on models that are large enough; it doesn't apply to creative or simple tasks

Additional resources

1. Chain-of-Thought Prompting Elicits Reasoning in LLMs (Wei et al., 2022)
2. Large Language Models are Zero-Shot Reasoners (Kojima et al., 2022)
3. OpenAI: Prompt Engineering Guide
4. Learn Prompting: Chain of Thought
5. Anthropic: Chain-of-Thought Prompting
6. GSM8K Dataset (Grade School Math)