Module 2: How LLMs Work (What You Need to Know as a Developer)

Description

Module 01 explained WHAT changed. This module explains the HOW: what's inside the box that makes coding agents work?

But it's not a Machine Learning class. You're not going to see backpropagation, attention mechanisms, or linear algebra. That's for ML engineers and researchers. What you need as a developer is different: you need to understand the concepts that directly affect how you use these tools.

When a coding agent invents an API that doesn't exist, why does it do that? When the same prompt produces different results, why? When the agent "forgets" something you told it 5 minutes ago, why? The answers are in how LLMs work — and you're going to understand them in this module.

By the end of the 5 capsules, you'll be able to predict which tasks are going to fail before you ask the agent for them, calibrate your confidence in the output based on the type of task, and diagnose strange behaviors without resorting to trial and error.

Where We Are in the Guide

Module 01: The paradigm shift                ✅ Completed
Module 02: How LLMs work                      ← YOU ARE HERE
Module 03: From chatbots to coding agents
Module 04: The agent's toolbox
Module 05: The developer as director
Module 06: The fundamental workflow
Module 07: Project: Build a mini-agent

This is Module 02 of 7 — the foundational technical module. Everything that comes after (how agents work, why you need to verify, how to direct them) is built on these concepts.

The Engine Analogy

YOU CAN DRIVE A CAR WITHOUT KNOWING HOW THE ENGINE WORKS.

But if you want to:
→ Understand why the car sometimes skids
→ Know when it's safe to accelerate and when to brake
→ Diagnose problems when something goes wrong
→ Predict how it will behave in different conditions

... you need to know the BASICS of the engine.

You do NOT need to know:
→ The details of internal combustion
→ The thermodynamics of the Otto cycle
→ The design of the intake valve

You DO need to know:
→ It uses gasoline (tokens)
→ It has a tank of limited size (context window)
→ It generates power probabilistically, not deterministically (inference)
→ It sometimes fails in predictable ways (hallucinations)

Each of the four elements of the car's engine has a direct equivalent in the "engine" of the coding agent. Learning them doesn't make you a mechanic — it makes you a better driver.

A Day in the Life: Two Developers, the Same Task

To anchor why this module matters, consider a concrete scenario. It's Monday morning, and two developers get the same task:

"Replicate the authentication pattern we already have in users-service for the new payments-service. There are 4 layers: middleware, validators, controllers, tests."

Both use the same coding agent (Claude Code, Cursor, whatever). Both have 2 hours to deliver.

Developer A: Doesn't understand how LLMs work

09:00 → Opens the agent and tells it:
        "Replicate the auth pattern from users-service in payments-service.
         Here's the whole codebase."
        Pastes 30 files into the prompt.

09:15 → The agent generates code. It looks reasonable. Accepts it.

09:45 → Runs the tests. They fail because the agent imported
        `from app.security.middleware import JWTGuard` —
        but JWTGuard doesn't exist in the project, it's called JWTMiddleware.

10:15 → "Fix it," they tell the agent.
        The agent "fixes" it by changing the name, but now
        the method doesn't exist either (the agent invented .verify_with_secret()
        when the real method is .validate()).

11:00 → 2 hours later, the code compiles but the tests are still red.
        The developer doesn't know why the agent "doesn't understand" their project.

Developer B: Understands the fundamentals of Module 02

09:00 → Knows the model can't process 30 full files
        (context window). Decides to give it only 4 relevant files:
        one middleware, one validator, one controller, one test —
        examples of the pattern to copy.

09:05 → Knows the agent can invent APIs (hallucinations).
        Explicitly asks it: "List the classes and methods you're going to
        use before generating code. I'll verify them."

09:15 → The agent lists 12 imports. Developer B finds that 2 are
        invented (they don't exist in their codebase). They fix them before
        generating code.

09:45 → Knows the output is probabilistic. Asks the agent to generate
        a version, reviews it, asks it to regenerate the part they didn't
        like. They don't expect the first version to be the best.

10:30 → Tests pass. Developer B finishes and spends the last 30 minutes
        documenting the pattern for future use.

Same tools. Same task. Radically different results.

The difference wasn't the tool. It was that Developer B understood:

Why pasting 30 files doesn't work (context window — Capsule 02)
Why the agent invents imports (hallucinations — Capsule 04)
Why the first answer isn't the "truth" (probabilistic inference — Capsule 03)
What to trust and what not to (capabilities and limitations — Capsule 05)

This module trains you to be Developer B.

What You WILL Learn

Five concepts that directly affect your daily work with coding agents:

1. TOKENS AND TOKENIZATION
   "What does the model 'see' when it reads my code?"
   → Why code uses more tokens than text
   → How it affects cost and speed

2. CONTEXT WINDOWS
   "How much can the model 'remember'?"
   → Why the agent "forgets" things
   → How the size of the context window limits what you can do

3. NEXT-TOKEN PREDICTION
   "How does the model generate code?"
   → One token at a time, based on probabilities
   → Why the same prompt gives different results

4. HALLUCINATIONS
   "Why does the model invent things?"
   → Why plausible ≠ correct
   → Specific kinds of errors in code

5. CAPABILITIES AND LIMITATIONS
   "What do I trust and what don't I?"
   → Pattern matching vs understanding
   → Ideal vs dangerous tasks for LLMs

What You Will NOT Learn

We don't cover	Why not	Where to find it
Backpropagation	It doesn't affect how you USE the model	ML/DL courses
Attention mechanisms	An architecture detail, not a usage detail	The "Attention is All You Need" paper
Transformer architecture	Relevant for researchers	NLP courses
Fine-tuning	You don't do fine-tuning as a user	Provider documentation
RLHF/DPO	A training process	Anthropic/OpenAI blog posts
Linear algebra	The model's internal math	Math-for-ML courses

The rule: If it doesn't affect how you USE a coding agent, it's not in this module.

Prerequisites

Required knowledge:

✅ Module 01 completed (context of the paradigm shift)
✅ No Machine Learning knowledge required

Module Roadmap

Capsule 01 — Module introduction (this capsule)

Context and motivation. What you'll learn and what you won't. The scenario that anchors the module.

Capsule 02 — Tokens and context windows

Tokenization: how the model converts text and code into numbers. BPE (Byte-Pair Encoding). Counting tokens. Context windows: current sizes (4K → 1M), why they matter, what happens when they fill up.

Capsule 03 — Inference and next-token prediction

How the model generates text: one token at a time, predicting the next one. Autoregressive generation. Temperature and top-p. Why the same prompt gives different results — and how to control it.

Capsule 04 — Hallucinations and limitations

Why LLMs invent information. Kinds of hallucinations in code: APIs that don't exist, incorrect versions, plausible but incorrect logic. Training data cutoff and its consequences. Why more parameters does NOT = more correct.

Capsule 05 — What they can and cannot do

Pattern matching vs real understanding. Tasks where LLMs shine (boilerplate, common patterns, explanations) vs tasks where they fail (complex logic, global state, architecture decisions). The "almost right" problem (66% of devs per Stack Overflow 2025).

Progression map

Capsule 01 (this)     → Why you need to know this
Capsule 02            → Tokens and context windows (input)
Capsule 03            → Next-token prediction (process)
Capsule 04            → Hallucinations (errors)
Capsule 05            → Capabilities and limitations (calibration)

Difficulty: ⭐ ──────────────────────────▶ ⭐⭐⭐

What You'll Achieve in This Module

By completing the 5 capsules:

Understand tokenization and why your code consumes more tokens than prose
Know what the context window is and how it affects your workflow
Understand code generation (it's probabilistic, not deterministic)
Predict when an LLM is going to hallucinate and what kind of error it will produce
Calibrate your confidence in the model's output based on the task

The before and after

BEFORE the module:
→ "The model understands my code"
→ "I don't know why it sometimes invents things"
→ "The context window is... a big number?"
→ "Tokens are... words?"

AFTER the module:
→ The model processes tokens, it doesn't understand like a human
→ It hallucinates because it predicts the "most probable," not the "correct"
→ The context window limits how much it can handle per session
→ Tokens are pieces of text — and code uses more than prose

Why These Concepts Matter for the Following Modules

MODULE 03 (Agents):
→ You need to understand tokens to understand tool calling
→ You need to understand context windows to understand the agentic loop

MODULE 04 (Toolbox):
→ The agent's tools consume tokens from the context window
→ Each iteration of the loop reduces the available space

MODULE 05 (Developer as director):
→ Trust calibration is based on knowing when the LLM is reliable
→ Knowing about hallucinations informs when to verify

MODULE 06 (Workflow):
→ R→P→E→V is a response to the model's limitations
→ Verification exists because LLMs can be "almost right"

MODULE 07 (Project):
→ You'll implement a mini-agent that uses an LLM's API
→ You'll configure temperature, tokens, and context window
→ You'll see from the inside how the responses are generated

Traps to Avoid While Taking This Module

Five predictable misunderstandings that can slow down your learning. Anticipate them before starting.

1. "I need ML depth to understand this"

No. If you find yourself searching for "how attention works exactly" or "what is backpropagation," you got off track. Those details matter for building models, not for using them. The module is designed so that a developer with no ML background comes out with practical mastery. If your curiosity asks for more, the resources at the end provide the next step — but you don't need it to finish the guide.

2. "The model understands my code"

It's the most expensive trap. The model processes patterns, it doesn't understand intent, domain, or business purpose. When the agent gets the code right, it's because it saw similar patterns millions of times during training — not because it "understood" what you want. This distinction saves you hours of debugging when something goes wrong: it's not that the model "got confused," it's that the pattern it saw doesn't match what you need.

3. "Capsule 03 sounds mathematical, I'll skip it"

Capsule 03 (inference and next-token prediction) is the most useful one for understanding why agents behave the way they do. It's not math — it's two analogies and a couple of parameters (temperature, top-p) that you're going to use in the Module 07 project. If you skip it, capsules 04 and 05 lose context.

4. "More tokens = better answer"

No. Large context windows (1M tokens) are useful for specific tasks (extensive codebases), but filling them without judgment degrades the quality of the answer. The model spends "attention" on irrelevant information. Capsule 02 explains why selecting the files well is more effective than pasting them all.

5. "The context window is like the model's memory between conversations"

It's not persistent. Every new conversation (or every context compaction) starts from zero. The model doesn't "remember" what you told it yesterday — only what's in the current context window. This confusion makes developers surprised when the agent "forgets" decisions from the previous session. Capsule 02 clarifies it.

Diagnosis: What's Your Starting Point?

Before starting the technical capsules, answer these questions mentally (or in writing). They give you an honest diagnosis of what you need to absorb with more attention.

Question 1: Have you seen a coding agent invent a function, method, or library that doesn't exist?

If yes: Note a concrete case before starting. You'll understand it in Capsule 04 (hallucinations) and you'll have a framework to detect it before accepting the code.

If no: You probably haven't reviewed the output carefully enough. Capsule 04 teaches you what to look for.

Question 2: If you give the same prompt to the same model twice, do you expect the same answer?

If yes: You're missing the concept of probabilistic inference (Capsule 03). The same prompt can give different answers — and that's not a bug, it's by design.

If no: Do you know why? If your answer is "because of temperature," excellent. If it's "just because," Capsule 03 gives you the mental model.

Question 3: Could you estimate how many tokens fit in a Claude or ChatGPT conversation before starting it?

If yes: Do you distinguish between input tokens, output tokens, and total context window? Capsule 02 formalizes that distinction.

If no: This is the most practical thing you're going to learn. It saves you unexpected conversation cutoffs and helps you budget long prompts.

Question 4: On which tasks would you blindly trust an LLM and on which tasks would you verify every line?

If you have an answer: Excellent. Capsule 05 gives you the formal framework to validate your intuitions (and possibly correct some).

If you don't have an answer: This is the heart of professional calibration. Capsule 05 gives you a taxonomy: what kind of task shines, which fails, which is "almost right" (the worst category).

Question 5: What does "the model has a training cutoff" mean?

If you know: You'll go deeper into how that affects queries about recent libraries, API versions, and post-cutoff changes.

If not: It's a critical concept. It determines why the agent may "not know" about libraries or features that exist — and why it can invent information about them.

If you hesitated on 3 or more: this module is calibrated exactly for you. If you answered them all with confidence, use it as a review and focus on Capsule 05 (the most practical one).

How to Work Through This Module

Capsule 02 is the most practical. You can experiment counting tokens in real time with an online tokenizer.
Capsule 03 is the most "technical." But it's explained with analogies, not math.
Capsule 04 is the most immediately useful. Understanding hallucinations changes how you verify.
Capsule 05 closes the module with a framework for deciding when to trust and when not to.

Estimated time:

Capsule 01 (this)  →  5 min
Capsule 02         → 12 min
Capsule 03         → 12 min
Capsule 04         → 10 min
Capsule 05         → 10 min + experimentation

Total: ~50-60 minutes

Evidence of Success

Before moving on to Module 03 (From chatbots to coding agents), you should be able to:

✅ Explain in one sentence what a token is and why your code uses more tokens than prose
✅ Estimate off the top of your head whether a conversation will fit in a given model's context window
✅ Predict which kind of task will produce hallucinations with the highest probability
✅ Decide on a reasonable temperature value based on the type of task (deterministic vs creative)
✅ Diagnose why the same prompt gave different results in two sessions
✅ Classify a task from your real work as "ideal for LLM" / "dangerous for LLM" / "almost right risk"

If any of these aren't met at the end, go back to the corresponding capsule before moving on. Module 03 assumes that the 6 points are in place.

Summary

This module gives you the fundamentals for understanding the behavior of coding agents.

What it covers:

Tokens and tokenization (what the model "sees")
Context windows (how much it "remembers")
Next-token prediction (how it "generates")
Hallucinations (why it "invents")
Capabilities and limitations (what to trust)

What it produces:

The ability to predict the agent's behaviors before they happen
Confidence calibration based on the type of task, not on intuition
Technical vocabulary to diagnose problems (instead of "the agent doesn't work")
A foundation for modules 03-07 (each one assumes these concepts)

Why it matters:

You can't direct what you don't understand
These concepts explain 90% of the strange behaviors
Module 07 will ask you to configure these parameters directly
The difference between Developer A and Developer B in the opening scenario is exactly this module

Next capsule: 02 — Tokens and context windows. We start with the engine's input: how your code turns into something the model can process, and why the "tank" (context window) has a size that matters more than you think.

Additional Resources

Andrej Karpathy: Intro to LLMs (1hr talk) — The best introduction to LLMs for developers, no math
OpenAI Tokenizer — Interactive tool to see tokenization in real time
Anthropic Token Counting Tool — How to count tokens with the Claude API
Anthropic: Building Effective Agents — How the concepts in this module apply to agent design
Simon Willison: Understanding LLMs — Practical, demystifying articles about the real behavior of LLMs
Stack Overflow Developer Survey 2025 — AI Section — The "66% almost right" data point that motivates Capsule 05

← PreviousAgentic Development vs Vibe Coding vs Traditional Development Next →Tokens and Context Windows