Module 2: How LLMs Work (What You Need to Know as a Developer)
Inference and Next-Token Prediction
Description
When you ask an LLM to "write a function that validates emails," the model doesn't "think" the whole function and write it. It generates the answer one token at a time, predicting what the most probable next token is given everything before it. This process is called inference, and the central mechanism is next-token prediction.
Understanding this changes how you interpret a coding agent's output — and explains why the same prompt can give different results.
The Central Concept: Predicting the Next Token
How an LLM generates text
GENERATION PROCESS (simplified):
INPUT: "function validate"
Step 1: What's the most probable token after "function validate"?
→ "Email" (probability: 15%)
→ "Input" (probability: 12%)
→ "User" (probability: 8%)
→ ... (thousands of options with decreasing probabilities)
→ PICKS: "Email"
Step 2: What's the most probable token after "function validateEmail"?
→ "(" (probability: 85%)
→ "Address" (probability: 5%)
→ ...
→ PICKS: "("
Step 3: What's the most probable token after "function validateEmail("?
→ "email" (probability: 40%)
→ "str" (probability: 15%)
→ "input" (probability: 10%)
→ PICKS: "email"
... and so on token by token until it generates the whole function.
Generation is sequential, not parallel
THE MODEL DOES NOT:
→ "Think" the whole function internally
→ Generate the whole answer at once
→ Have a "plan" of what it's going to write
→ "Know" how the function will end
THE MODEL DOES:
→ Generate a token
→ Add it to the context
→ Predict the next token given everything before
→ Repeat until a stop token or the limit
THIS IS CALLED:
AUTOREGRESSIVE generation
(each output becomes input for the next step)
Visualizing the process
INPUT: "Write a Python function to validate an email"
Step-by-step generation:
"def"
"def validate"
"def validate_"
"def validate_email"
"def validate_email("
"def validate_email(email"
"def validate_email(email:"
"def validate_email(email: str"
"def validate_email(email: str)"
"def validate_email(email: str) ->"
"def validate_email(email: str) -> bool"
"def validate_email(email: str) -> bool:"
"def validate_email(email: str) -> bool:\n"
"def validate_email(email: str) -> bool:\n "
"def validate_email(email: str) -> bool:\n import"
"def validate_email(email: str) -> bool:\n import re"
...
EACH LINE IS ONE ITERATION.
Each token is generated based on ALL the previous tokens.
Probability Distributions: It's Not Deterministic
The model doesn't pick "the correct answer"
At each step, the model generates a probability distribution over all possible tokens in its vocabulary. There's no "correct" answer — there are more or less probable answers:
AFTER "def validate_email(email: str) -> "
Possible token: Probability:
"bool" 35%
"str" 15%
"dict" 8%
"None" 7%
"Optional" 6%
"bool" --
... ...
THE MODEL DOESN'T "KNOW" IT SHOULD RETURN bool.
It calculates that "bool" is the MOST PROBABLE token
given the context of "validate email."
If the context said "parse email," the probabilities
would be different — "dict" or "str" would go up.
Why the same prompt gives different results
PROMPT: "Write a function to sort an array"
RUN 1:
def sort_array(arr):
return sorted(arr)
RUN 2:
def sort_array(arr):
arr.sort()
return arr
RUN 3:
def sort_array(arr: list) -> list:
return sorted(arr)
THREE VALID ANSWERS TO THE SAME PROMPT.
Why?
→ At each step, there are MULTIPLE probable tokens
→ The model doesn't always pick the most probable one
→ There's an element of controlled randomness
→ That randomness is controlled with TEMPERATURE
Temperature: Controlling the Randomness
What temperature is
Temperature is a parameter that controls how much randomness the generation has. It goes from 0 to ~2 (depending on the model):
TEMPERATURE = 0 (deterministic)
→ ALWAYS picks the most probable token
→ Reproducible results (same input = same output)
→ Less creative, more predictable
→ Ideal for: code, structured data, tasks with one correct answer
TEMPERATURE = 0.7 (balanced)
→ Generally picks probable tokens, but with variation
→ Slightly different results each time
→ Balance between consistency and variety
→ Default in many models
TEMPERATURE = 1.5+ (creative)
→ Distributes more evenly among possible tokens
→ Very varied results
→ More "creative" but less reliable
→ Ideal for: brainstorming, creative text
→ Dangerous for: code (introduces random errors)
Visual analogy
TEMPERATURE 0:
Probabilities: ████████░░ ██░░░░░░░░ █░░░░░░░░░
"bool" "str" "dict"
↑
ALWAYS picks this one
TEMPERATURE 0.7:
Probabilities: ██████░░░░ ████░░░░░░ ██░░░░░░░░
"bool" "str" "dict"
↑ ↑
Usually Sometimes this one
TEMPERATURE 1.5:
Probabilities: ████░░░░░░ ███░░░░░░░ ███░░░░░░░
"bool" "str" "dict"
↑ ↑ ↑
Any of them can be picked
Temperature in coding agents
MOST CODING AGENTS use LOW temperature for code:
→ Claude Code: temperature close to 0 for code generation
→ Cursor: configurable, low default for code generation
→ Copilot: low for autocomplete
WHY LOW?
→ For code, you want CONSISTENCY
→ A variable with a different name each time is confusing
→ "Creativity" in code is usually errors
→ You want the same pattern to produce the same result
Top-p (Nucleus Sampling)
What top-p is
Top-p is another way of controlling the randomness. Instead of scaling the probabilities (temperature), it limits how many tokens it picks from:
TOP-P = 0.1 (very restrictive):
→ Only picks from the top 10% of most probable tokens
→ Very predictable, very consistent
TOP-P = 0.9 (standard):
→ Picks from the set of tokens that add up to 90% probability
→ Balance between variety and coherence
TOP-P = 1.0 (anything goes):
→ All tokens are candidates
→ Maximum variety (but it can be incoherent)
EXAMPLE WITH TOP-P = 0.9:
Token: Probability: Cumulative:
"bool" 35% 35% ← included
"str" 15% 50% ← included
"dict" 8% 58% ← included
"None" 7% 65% ← included
"Optional" 6% 71% ← included
"int" 5% 76% ← included
"list" 4% 80% ← included
"tuple" 3% 83% ← included
"Any" 3% 86% ← included
"Union" 2% 88% ← included
"float" 2% 90% ← CUTOFF (top-p = 0.9)
"bytes" 1% 91% ← excluded
...
Temperature vs Top-p
TEMPERATURE:
→ Controls HOW MUCH to redistribute the probabilities
→ Temperature 0 = always the most probable
→ More temperature = more uniform distribution
TOP-P:
→ Controls HOW MANY tokens to pick from
→ Low top-p = few candidates
→ High top-p = many candidates
IN PRACTICE:
→ Most APIs let you configure both
→ For code: low temperature + moderate top-p
→ You generally don't need to adjust top-p manually
→ Temperature is the more important parameter of the two
Implications for Coding Agents
Why the agent sometimes generates code different from what you expected
SITUATION:
You ask the agent to "add a GET /users endpoint"
You expected it to use the Express Router.
The agent used app.get() directly.
EXPLANATION:
→ The model predicted that app.get() was the most probable next token
→ Given the context (your prompt + files read), app.get() had
a high probability
→ It didn't "decide" to ignore the Router — its prediction was different from your expectation
→ If the prompt mentioned "use the Router pattern that already exists in the project,"
the probability of Router would go up dramatically
IMPLICATION:
→ More context in the prompt → more aligned predictions
→ Less context → more variation → more surprises
→ This connects with the "prompt-and-pray" anti-pattern from Module 06
Why autoregressive generation matters
THE MODEL CAN'T "GO BACK"
Once it generates a token, that token becomes
part of the context. If the first token was a
bad decision, the rest builds on that base.
EXAMPLE:
"def validate_email(email):" ← no type hint
Once generated, the model continues WITHOUT type hints
because the context now has no hints.
If it had started with:
"def validate_email(email: str) -> bool:" ← with type hints
The rest of the code would have consistent type hints.
IMPLICATION FOR DEVELOPERS:
→ The start of the generation matters a lot
→ A good prompt "anchors" the first decisions
→ If the agent starts badly, it's better to restart than to patch
→ Connects with the "rule of three" from Module 06
The "reasoning tokens" (thinking tokens)
MODERN MODELS have "extended thinking":
Before generating the visible answer, the model
generates "reasoning tokens" — internal thinking.
EXAMPLE:
Input: "Fix the bug in auth.ts"
[Reasoning tokens — NOT visible or partially visible]:
"The user wants me to fix a bug. I need to:
1. Understand what auth.ts does
2. Find the bug
3. Fix it
Let me start by reading the file..."
[Output tokens — visible]:
"Let me read auth.ts first."
→ tool_call(file_read, "auth.ts")
THE REASONING TOKENS:
→ Consume context window (they count as output)
→ Improve the quality of the decisions
→ Are the basis of the ReAct pattern (Module 03)
→ Their cost is included in the output tokens
Inference in Practice: The API
When you use an LLM's API (Module 07), you'll see these parameters directly:
# Simplified example of an API call
response = client.messages.create(
model="claude-4-sonnet",
max_tokens=4096, # Maximum tokens to generate
temperature=0.2, # Low for code
messages=[
{
"role": "user",
"content": "Write a function to validate emails"
}
]
)
# The response includes token metadata:
print(response.usage)
# {
# "input_tokens": 12, ← tokens from your prompt
# "output_tokens": 85, ← generated tokens
# }The parameters you control
GENERATION PARAMETERS:
max_tokens: Maximum tokens the model can generate
temperature: How much randomness (0 = deterministic)
top_p: How many tokens to pick from (nucleus sampling)
stop: Tokens that stop the generation (e.g., "\n\n")
PARAMETERS YOU DON'T CONTROL:
The model: Trained on fixed data (training data cutoff)
The weights: The model's "intelligence" is fixed
The tokenizer: How the tokens are split is fixed
Practical Exercise
Exercise 1: Experiment with temperature
If you have access to an API or playground:
PROMPT: "Write a Python function to reverse a string"
Temperature 0:
→ Result: ________________________________________
→ Run it 3 times. Always the same? □ Yes □ No
Temperature 0.7:
→ Result: ________________________________________
→ Run it 3 times. Always the same? □ Yes □ No
Temperature 1.5:
→ Result: ________________________________________
→ Run it 3 times. Always the same? □ Yes □ No
What differences do you observe?
→ _______________________________________________
→ Which one do you prefer for code? _______________
See solution
Expected results:
-
Temperature 0: The 3 runs should produce identical or nearly identical results. The code will be predictable — for example,
return s[::-1]in Python. It's the ideal option for code generation because it guarantees consistency. -
Temperature 0.7: There will be variations between runs. You might see
return s[::-1]once,return ''.join(reversed(s))another, and maybe a loop-based solution the third time. All valid, but different. -
Temperature 1.5: The results will vary significantly and could include errors. Unusual variable names, strange approaches, or even code that doesn't work. The high randomness introduces "creativity" that in code is usually counterproductive.
Conclusion: For code generation, low temperature (0-0.3) is recommended. The "creativity" you need in code doesn't come from randomness — it comes from a prompt with good context.
Exercise 2: Observe sequential generation
If your coding agent shows the generation in real time (streaming):
1. Give it a task that requires generating ~20 lines of code
2. WATCH how the text appears — token by token
3. Can you see that it generates left to right, top to bottom?
4. At any point does it "go back" and correct something it already generated?
□ Yes □ No (spoiler: the answer is No — it's autoregressive)
See solution
Expected observation:
-
The text appears sequentially, from left to right and top to bottom, exactly the way you read a book.
-
No, the model never "goes back." The generation is autoregressive: each token is generated in sequence and, once produced, becomes fixed context for the following tokens. If the model makes an error at token 5, tokens 6, 7, 8... build on that error.
-
This explains why when an agent starts generating a solution with a wrong approach, the entire solution tends to follow that approach. It's more effective to restart the generation (a new answer) than to wait for the model to "correct itself" midway.
Exercise 3: The impact of context on prediction
PROMPT 1 (no context):
"Write a function to get users"
→ What did it generate? ___________________________________
→ Did it use your framework? □ Yes □ No
PROMPT 2 (with codebase context):
"Write a function to get users. The project uses
Express with TypeScript and Prisma as the ORM.
Follow the pattern in src/controllers/ProductController.ts"
→ What did it generate? ___________________________________
→ Did it use your framework? □ Yes □ No
Did the context improve the prediction?
→ _______________________________________________
See solution
Expected result:
-
Prompt 1 (no context): The model probably generated a generic function — maybe with
sqlite3,requests, or even an in-memory dictionary. It picked the globally most probable pattern, which rarely matches your stack. -
Prompt 2 (with context): By mentioning Express, TypeScript, and Prisma, the model should generate code using those technologies. By mentioning a reference file (
ProductController.ts), the probabilities "anchor" to that specific pattern — function names, import structure, and project conventions. -
Conclusion: Context transforms the model's predictions. Without context, you get "average internet code." With specific context, you get code aligned with your project. This is the basis of why coding agents read files before generating: they're feeding the model with context that improves the probabilities.
Common Mistakes
| Mistake | Reality |
|---|---|
| "The model thinks the whole answer" | It generates one token at a time, sequentially |
| "High temperature = smarter" | High temperature = more random, not smarter |
| "The model decides which is the correct answer" | It predicts which is the most probable, not the most correct |
| "I can get the same result every time" | Only with temperature 0, and even then there can be variation from the implementation |
| "The reasoning tokens are free" | They consume context window and count as output tokens |
Summary
NEXT-TOKEN PREDICTION:
→ The model generates ONE TOKEN at a time
→ Each token is based on ALL the previous ones
→ It's autoregressive generation (the output feeds the input)
→ It doesn't "think" the whole answer beforehand
PROBABILITIES:
→ At each step there are THOUSANDS of possible tokens
→ Each one with a different probability
→ The model picks based on probabilities, not "correctness"
→ That's why the same prompt can give different results
TEMPERATURE:
→ Controls the randomness of the selection
→ Low (0-0.3) = predictable, consistent (ideal for code)
→ High (1.0+) = varied, creative (dangerous for code)
IMPLICATIONS:
→ More context → better predictions
→ The start of the generation matters a lot
→ If it starts badly, it's better to restart than to patch
→ The reasoning tokens improve quality but cost tokens
Next capsule: 04 - Hallucinations and limitations — why the model invents things and what kinds of errors it produces in code.
Resources
- Andrej Karpathy: Intro to LLMs — Visual explanation of next-token prediction
- OpenAI: API Parameters — Documentation of temperature, top-p, etc.
- Anthropic: API Reference — Generation parameters in Claude
- Jay Alammar: Illustrated GPT-2 — Detailed visualization of the generation process
- Chip Huyen: Designing ML Systems — Context on inference in production