Module 2: How LLMs Work (What You Need to Know as a Developer)

Hallucinations and Limitations

Next →

Description

In the previous capsule you learned that the model generates text by predicting the most probable token. This mechanic has an inevitable consequence: sometimes, the most probable token isn't the most correct one. When the model generates information that looks correct but isn't, we call that a hallucination.

In code, hallucinations aren't a minor inconvenience. They're bugs that look like correct code, compile without errors, and pass a superficial review — but don't work. This capsule explains why they happen, what specific kinds they produce in code, and how to detect them.

Why LLMs Hallucinate

The root cause

THE MODEL DOESN'T "KNOW" ANYTHING.

The model has processed patterns from trillions of tokens of text.
When it generates, it produces the most PROBABLE pattern.

PROBABLE ≠ CORRECT

Example:
→ It has seen thousands of times: "import pandas as pd"
→ It knows this is a valid pattern
→ But if you ask it for "import pandasx as pd"
→ It could generate it if the context suggests it
→ Because it looks PLAUSIBLE, even though "pandasx" doesn't exist

THE MODEL DOESN'T VERIFY WHAT IT GENERATES.
It doesn't have a database of "true facts."
It only has statistical patterns.

Analogy: advanced autocomplete

YOUR PHONE has predictive autocomplete:
→ "See you at" → suggests "8" (frequent pattern)
→ But the meeting is at 3
→ Autocomplete predicted the most PROBABLE, not the CORRECT

AN LLM is the same, but at massive scale:
→ It generates code that LOOKS correct
→ Based on patterns from millions of code files
→ But it can be incorrect for YOUR specific case
→ Because it works with probabilities, not with truth

The three factors of hallucinations

1. TRAINING DATA DISTRIBUTION
   → If something is rare in the training data, the model handles it worse
   → New APIs, rarely used libraries, uncommon patterns
   → The model "fills in" with the closest thing it knows

2. CONFIDENCE CALIBRATION
   → The model doesn't know what it doesn't know
   → It generates correct and incorrect text with the same confidence
   → It doesn't say "I'm not sure" — it generates the most probable
   → You can't use the model's "tone" as an indicator of correctness

3. PATTERN COMPLETION vs FACTUAL RECALL
   → The model completes patterns, it doesn't remember facts
   → "The X API has the method..." → completes the pattern
   → But it can complete with a method that doesn't exist
   → Because the PATTERN of "API has method" is correct
     even though the specific FACT is incorrect

Types of Hallucinations in Code

Type 1: APIs and methods that don't exist

EXAMPLE:
"Use response.json.parse() to parse the response"

REALITY:
→ response.json() exists in the fetch API
→ JSON.parse() exists as a global function
→ response.json.parse() does NOT exist
→ It's a MIX of two real things

WHY IT HAPPENS:
→ The model has seen "response.json" and "JSON.parse" thousands of times
→ The combination looks plausible
→ The pattern "object.method.method()" is valid in JS
→ But this specific case doesn't exist

HOW TO DETECT:
→ Verify methods in the official documentation
→ If a method seems new or unfamiliar to you, look it up
→ The agent can invent methods that "should exist"

Type 2: Incorrect or deprecated versions

EXAMPLE:
"Install react-router-dom v5 and use <Switch>"

REALITY:
→ <Switch> existed in v5 but was replaced by <Routes> in v6
→ If the project uses v6, this code doesn't work
→ If the project doesn't specify a version, the model can mix APIs

WHY IT HAPPENS:
→ The training data contains code from MULTIPLE versions
→ The model doesn't distinguish between v5 and v6 "automatically"
→ If the context doesn't specify the version, it picks the most frequent
→ Which may not be the most recent

HOW TO DETECT:
→ Verify the version of the dependencies in your project
→ If the generated code uses an API you don't recognize, it may be from another version
→ The training data cutoff date matters here

Type 3: Plausible but incorrect logic

EXAMPLE:
function isLeapYear(year) {
  return year % 4 === 0;
}

REALITY:
→ Leap years aren't just divisible by 4
→ They must also be divisible by 400 OR not divisible by 100
→ The real rule: (year % 4 === 0 && year % 100 !== 0) || year % 400 === 0
→ The generated code is ALMOST correct (works for 96% of years)
→ But it fails for years like 1900, 2100

WHY IT HAPPENS:
→ "Divisible by 4" is the most frequent pattern for leap year
→ The simplified version appears more in the training data
→ The model generates the most PROBABLE, not the most COMPLETE
→ It's the "almost right" problem

HOW TO DETECT:
→ Business logic always needs complete verification
→ "It looks correct" isn't enough for logic
→ Tests with edge cases reveal these errors

Type 4: Packages and imports that don't exist

EXAMPLE:
from utils.validators import EmailValidator

REALITY:
→ utils/validators.py doesn't exist in your project
→ The model ASSUMED it existed because the pattern is common
→ It created an import to something that doesn't exist

WHY IT HAPPENS:
→ "from utils.something import Something" is an extremely common pattern
→ The model generates it because it's statistically probable
→ It doesn't verify that the file actually exists

HOW TO DETECT:
→ If the agent imports something you don't recognize, verify it exists
→ Coding agents are BETTER at this because they can search
→ But ChatGPT (chatbot) hallucinates imports frequently

Type 5: Syntactically correct, semantically incorrect code

EXAMPLE:
async function getUser(id) {
  const user = await db.query("SELECT * FROM users WHERE id = " + id);
  return user;
}

REALITY:
→ The code WORKS
→ It compiles and runs
→ It returns the correct user
→ But it has SQL INJECTION
→ It's a critical security bug

WHY IT HAPPENS:
→ String concatenation in SQL is a pattern that exists in the training data
→ The model doesn't "understand" security — it sees patterns
→ The pattern "SELECT * WHERE id = " + variable is frequent (especially in tutorials)
→ The model doesn't evaluate the security implications

HOW TO DETECT:
→ Security-focused code review
→ NEVER trust AI-generated security code without review
→ The "adversarial prompt" from Module 06 helps here

Training Data Cutoff

What it is

THE MODEL WAS TRAINED ON DATA UP TO A SPECIFIC DATE.

After that date:
→ It doesn't know new APIs
→ It doesn't know new library versions
→ It doesn't know bugs reported afterward
→ It doesn't know changes in best practices

THIS MEANS:
→ If an API changed after the cutoff, the model uses the old one
→ If a method was deprecated afterward, the model suggests it
→ If a vulnerability was discovered afterward, the model doesn't know it

Practical impact

COMMON SCENARIOS:

1. "Use the OpenAI API to..."
   → The model may use the v3 API when v4 already exists
   → The methods and parameters may have changed

2. "Configure Next.js..."
   → Next.js changes significantly between versions
   → The model can mix App Router and Pages Router patterns

3. "Implement auth with Supabase..."
   → If Supabase changed its API after the cutoff
   → The generated code may use methods that no longer exist

MITIGATION:
→ Specify versions in your prompt
→ Include relevant documentation as context
→ Verify against the current official documentation
→ Coding agents can search the web (web_search tool)

The Reliability Scale by Task Type

Not all tasks are equally prone to hallucinations:

MORE RELIABLE ──────────────────────────── LESS RELIABLE

Common          Standard       Specific      Complex      Global
patterns        code           APIs          business     state
                                             logic

"for loop"     "Express       "Stripe v3    "Tax        "Multi-tenant
"if/else"       middleware"    webhook"      calc"        permission
"map/filter"   "basic CRUD"   "specific                  system"
                              AWS SDK"

Rarely         Occasionally   Frequently    Likely       Very
hallucinates   hallucinates   hallucinates  hallucinates likely
                                                         hallucinates

Why this scale

COMMON PATTERNS (high reliability):
→ They appear MILLIONS of times in the training data
→ for, if, map, filter → almost impossible to get wrong
→ The pattern is so frequent that the prediction is robust

STANDARD CODE (good reliability):
→ Express middleware, React components, basic SQL queries
→ Well-established and frequent patterns
→ Errors possible but generally minor

SPECIFIC APIs (variable):
→ Depends on how much they appear in the training data
→ Popular APIs (Stripe, AWS) → relatively reliable
→ Niche or very new APIs → more hallucinations

BUSINESS LOGIC (low reliability):
→ The model does NOT know YOUR domain
→ "Calculate income tax in Mexico" → probably incorrect
→ It needs explicit specifications, it can't "guess" the rules

GLOBAL STATE (very low reliability):
→ The model sees individual files, not the complete state
→ Interactions between systems, race conditions, side effects
→ Too much complexity for pattern matching

Hallucinations in Coding Agents vs Chatbots

CODING AGENTS hallucinate LESS than chatbots
on certain kinds of errors:

WHY?
→ The agent can VERIFY what it generates
→ file_read: it can check whether a file exists
→ shell_execute: it can run tests
→ search: it can search whether a function exists in the codebase

BUT:
→ The agent still uses an LLM as its engine
→ The LLM's hallucinations still occur
→ The agent can generate a file_write with hallucinated code
→ VERIFICATION is what reduces the impact

THAT'S WHY:
→ Verification isn't optional (Module 06)
→ Trust calibration depends on the task type (Module 05)
→ The developer is the final circuit breaker

How to Detect Hallucinations

The warning signs

1. CODE THAT "LOOKS" CORRECT BUT YOU DON'T RECOGNIZE
   → If you don't recognize a method, verify it exists
   → If you don't recognize a pattern, research before accepting
   → "It sounds familiar" isn't enough

2. IMPORTS TO UNFAMILIAR MODULES
   → If you don't know the library, verify it exists
   → If the import path doesn't sound right, verify it

3. UNSPECIFIED VERSIONS
   → If the code assumes a version but doesn't specify it
   → It may be using an old or future API

4. LOGIC THAT SEEMS TOO SIMPLE
   → If a complex task produces simple code
   → It's probably missing edge case handling
   → "Too good to be true" is a signal

5. SPECIFIC DATA OR FACTS
   → URLs, version numbers, dates, statistics
   → The model can invent plausible data
   → Always verify factual data

Detection techniques

1. EXECUTION
   → Run the code. Does it work?
   → If it doesn't compile or fails at runtime: probable hallucination

2. TESTS
   → Write tests for edge cases
   → If the tests fail: the logic may be hallucinated

3. DOCUMENTATION CHECK
   → Verify methods and APIs against the official documentation
   → If the method doesn't exist in the docs: hallucination

4. ADVERSARIAL THINKING
   → "What would happen if the input is empty?"
   → "What would happen if there are 1 million records?"
   → If the code doesn't handle these cases: it may be incomplete

5. ASKING THE AGENT TO REVIEW
   → "Review this code and look for errors"
   → Switch the role from "generator" to "reviewer"
   → Sometimes it finds its own hallucinations

Practical Exercise

Exercise 1: Hunting hallucinations

Ask a chatbot (not a coding agent) to generate code for a task you know well:

TASK: _______________________________________________

1. Does the generated code compile? □ Yes □ No
2. Do all the imports exist? □ Yes □ No □ Don't know
3. Do the methods/functions used exist? □ Yes □ No □ Don't know
4. Is the logic correct for ALL edge cases? □ Yes □ No
5. Is the version of the APIs correct? □ Yes □ No □ Don't know

HALLUCINATIONS FOUND:
1. _______________________________________________
2. _______________________________________________
3. _______________________________________________

Would you have accepted this code without review? □ Yes □ No
See solution

Typical hallucinations you might find:

  1. Nonexistent imports: The model can import modules or functions that don't exist in your project or in the mentioned library (e.g., from utils.helpers import sanitize_input when that file doesn't exist).

  2. Mixed methods: It combines methods from different versions or libraries. For example, mixing the fetch API with the axios one in the same function.

  3. Ignored edge cases: The logic works for the most common case but omits validations of null, empty strings, empty arrays, or negative values.

  4. Invented parameters: It uses configuration parameters that don't exist in the real API (e.g., response.json(strict=True) when that parameter doesn't exist).

Key reflection: If you answered "Yes" to whether you would have accepted the code without review, this exercise demonstrates why verification isn't optional. AI-generated code "looks" correct by design — the model generates plausible text.

Exercise 2: The invented API test

Ask the model:
"Give me an example of how to use the 'fluxinator' library 
in Python to process data"

NOTE: 'fluxinator' DOES NOT EXIST.

Did the model...
□ Say it doesn't exist and not generate code
□ Generate code with an invented library
□ Confuse it with another library

THIS EXERCISE demonstrates that the model generates
what looks probable, not what is true.
See solution

Expected result:

In the vast majority of cases, the model will generate code for the invented library as if it existed. It can:

  • Invent a plausible API: from fluxinator import DataPipeline with methods like .process(), .transform(), .output()
  • Give installation instructions: pip install fluxinator
  • Even invent documentation or configuration arguments

Why? Because the pattern "import library → instantiate → process data" is extremely frequent in the training data. The model completes the pattern without verifying whether the library is real. The name "fluxinator" sounds plausible (similar to "transformers", "accelerator", etc.), which increases the probability of generation.

Lesson: The model doesn't have a registry of "real libraries." It generates what is statistically plausible. Always verify that the packages, APIs, and methods actually exist.

Exercise 3: Compare versions

Ask the model:
"Show how to create a server with the most 
recent API of [framework you use]"

1. Did it use the most recent version? □ Yes □ No
2. If not, which version did it use? _______________
3. Are there differences with the current version? _______________

Would this have caused a bug in your project?
→ _______________________________________________
See solution

Expected result:

The model will probably not use the most recent version, especially if the framework has had significant changes since the training data cutoff date. Common examples:

  • Next.js: It can mix App Router (v13+) with Pages Router (v12 and earlier).
  • React: It can use class components or deprecated APIs.
  • FastAPI / Express: It can use patterns from earlier versions.

Why? The training data contains code from ALL versions, and the older versions have more accumulated examples. The model picks the version with the most statistical representation, not the most recent.

Practical mitigation: Always specify the version in your prompt: "Use Next.js 15 with App Router" instead of "Use Next.js." Including a file from your project as context (e.g., package.json) helps the model detect the correct version.

Common Mistakes

MistakeReality
"If the code compiles, it's correct"Compiling only verifies syntax, not logic or security
"The model would tell me if it doesn't know"The model generates the most probable, it doesn't say "I don't know"
"Hallucinations are rare"They're frequent in specific APIs, business logic, and versions
"Coding agents don't hallucinate"They hallucinate less (they can verify) but the underlying LLM is the same
"I can trust the model's data/facts"URLs, versions, statistics must ALWAYS be verified

Summary

WHY THEY HALLUCINATE:
→ The model predicts the PROBABLE, not the CORRECT
→ It doesn't verify what it generates
→ It doesn't "know" anything — it completes patterns
→ It doesn't distinguish between real and invented information

TYPES IN CODE:
1. APIs and methods that don't exist
2. Incorrect or deprecated versions
3. Plausible but incorrect logic
4. Invented packages and imports
5. Syntactically correct, semantically wrong code

RELIABILITY SCALE:
Common patterns > Standard code > Specific APIs > 
Business logic > Global state

TRAINING DATA CUTOFF:
→ The model doesn't know anything after its training date
→ New APIs, new versions, new vulnerabilities
→ Specify versions and verify against current docs

DETECTION:
→ Run the code
→ Tests with edge cases
→ Verify against documentation
→ Adversarial thinking
→ Ask the agent to review its own code

Next capsule: 05 - What LLMs can and cannot do — the guide to calibrating your confidence.

Resources

  1. Anthropic: Understanding Hallucinations — The provider's perspective
  2. Veracode: State of Software Security 2025 — Data on vulnerabilities in AI-generated code
  3. Ji et al.: Survey of Hallucination in NLG — Academic survey on hallucinations
  4. GitHub Blog: AI Code Generation Research — Research on generated-code quality
  5. Simon Willison: Hallucinations in LLMs — Practical analysis and examples
← PreviousInference and Next-Token PredictionNext →What LLMs Can and Cannot Do