Module 2: How LLMs Work (What You Need to Know as a Developer)

Tokens and Context Windows

Next →

Description

When you send code to an LLM, the model doesn't see letters, words, or lines of code the way you see them. It sees tokens — fragments of text converted into numbers. And it can only process a limited amount at a time: the context window. These two concepts are fundamental to understanding why coding agents behave the way they do.

What a Token Is

The simple definition

A token is a piece of text the model processes as a unit. It's not exactly a word, nor exactly a character. It's something in between:

ENGLISH TEXT:
"Hello world" → ["Hello", " world"]  → 2 tokens

SPANISH TEXT:
"Hola mundo" → ["Hola", " mundo"]  → 2 tokens

CODE:
"console.log('hello')" → ["console", ".", "log", "('", "hello", "')"]  → 6 tokens

NOTE: The exact tokenization varies by model.
These are simplified examples to illustrate the concept.

Why they aren't simply words

ONE WORD CAN BE MULTIPLE TOKENS:

"authentication" → ["authentic", "ation"]  → 2 tokens
"implementation" → ["implement", "ation"]  → 2 tokens
"backgroundColor" → ["background", "Color"]  → 2 tokens

ONE TOKEN CAN BE LESS THAN A WORD:

"a" → ["a"]  → 1 token
"  " (spaces) → ["  "]  → 1 token
"\n" (newline) → ["\n"]  → 1 token

ONE TOKEN CAN INCLUDE SPACES:

" the" (with a leading space) → [" the"]  → 1 token
Spaces are part of the tokens, not separators

How tokenization works (BPE)

Modern models use Byte-Pair Encoding (BPE) to tokenize. You don't need to implement BPE, but understanding the idea helps:

BPE IN 30 SECONDS:

1. Start with individual characters: h, e, l, l, o
2. Look for the most frequent pairs in the training data
3. Combine the frequent pairs into larger tokens
4. Repeat until you have a fixed-size vocabulary

RESULT:
→ Common words = 1 token ("the", "and", "for")
→ Less common words = 2-3 tokens
→ Rare words = many tokens
→ Code = depends on the language's patterns

THE FINAL VOCABULARY:
→ GPT-4: ~100,000 tokens in its vocabulary
→ Claude: similar size
→ Each model has its own tokenizer

Tokens and Code: Why It Matters

Code uses more tokens than prose

PROSE (100 words):
"The quick brown fox jumps over the lazy dog.
 This is a simple sentence that most people can
 understand easily."
→ ~25-30 tokens

CODE (equivalent in complexity):
function validateEmail(email) {
  const regex = /^[a-zA-Z0-9._]+@[a-zA-Z0-9]+\.[a-zA-Z]{2,}$/;
  return regex.test(email);
}
→ ~40-50 tokens

WHY MORE?
→ Special symbols: { } ( ) ; = / ^ $ + . * [ ]
→ Each symbol can be a separate token
→ camelCase and snake_case split into multiple tokens
→ Regex is especially token-expensive
→ Indentation (spaces/tabs) consumes tokens

Practical impact: TypeScript code vs Python

TYPESCRIPT (verbose):
interface UserProfile {
  id: string;
  email: string;
  name: string;
  createdAt: Date;
}

export async function getUserProfile(
  userId: string
): Promise<UserProfile | null> {
  const user = await db.users.findUnique({
    where: { id: userId }
  });
  return user;
}

→ ~80-100 tokens

PYTHON (concise):
async def get_user_profile(user_id: str) -> dict | None:
    user = await db.users.find_unique(
        where={"id": user_id}
    )
    return user

→ ~40-55 tokens

SAME FUNCTIONALITY, DIFFERENT TOKENS.
→ TypeScript: more types, more syntax → more tokens
→ Python: less boilerplate → fewer tokens
→ This affects how much code fits in the context window

Counting tokens in practice

GENERAL RULE (approximate):

English: 1 token ≈ 4 characters ≈ 0.75 words
Code: 1 token ≈ 3 characters (more symbols = fewer chars per token)

TO ESTIMATE QUICKLY:
→ A 100-line file of code ≈ 500-800 tokens
→ A 500-line file of code ≈ 2,500-4,000 tokens
→ A complete project (thousands of files) ≈ millions of tokens

TOOLS FOR COUNTING:
→ OpenAI Tokenizer: https://platform.openai.com/tokenizer
→ tiktoken (Python library): pip install tiktoken
→ Most APIs report the tokens used in the response

Context Windows: How Much the Model Can Process

What the context window is

The context window is the total token limit the model can process in a single interaction. It includes EVERYTHING:

CONTEXT WINDOW = INPUT + OUTPUT

INPUT includes:
→ System prompt (instructions to the model)
→ Conversation history (previous messages)
→ Your current message
→ Tool definitions (if it's an agent)
→ Results of previous tool calls
→ Files the agent read

OUTPUT includes:
→ The model's reasoning (thinking)
→ The generated response
→ Generated tool calls

EVERYTHING COMPETES FOR THE SAME SPACE.

Current context window sizes (2026)

┌─────────────────────────┬───────────────┬──────────────┐
│ MODEL                   │ CONTEXT       │ APPROX.      │
│                         │ WINDOW        │ EQUIVALENT   │
├─────────────────────────┼───────────────┼──────────────┤
│ Claude 4 (Anthropic)    │ 200K tokens   │ ~150K words  │
│                         │               │ ~500 pages   │
├─────────────────────────┼───────────────┼──────────────┤
│ GPT-4.5 (OpenAI)        │ 128K tokens   │ ~96K words   │
│                         │               │ ~320 pages   │
├─────────────────────────┼───────────────┼──────────────┤
│ Gemini 2.5 (Google)     │ 1M tokens     │ ~750K words  │
│                         │               │ ~2,500 pages │
├─────────────────────────┼───────────────┼──────────────┤
│ Codestral (Mistral)     │ 256K tokens   │ ~192K words  │
│                         │               │ ~640 pages   │
└─────────────────────────┴───────────────┴──────────────┘

NOTE: These numbers change frequently.
What matters isn't memorizing the exact number
but understanding the CONCEPT that there's a limit.

It seems huge. What's the problem?

200K tokens sounds like a lot. But...

A TYPICAL CODING AGENT USES:

System prompt:                    ~2,000-5,000 tokens
Tool definitions:                 ~1,000-3,000 tokens
Conversation history:             ~varies (grows with each turn)
Files read by the agent:          ~500-4,000 per file
Command results:                  ~varies

EXAMPLE: After 10 iterations of the agentic loop

System prompt:                     3,000 tokens
Tool definitions:                  2,000 tokens
10 conversation messages:         15,000 tokens
5 files read:                     12,000 tokens
3 shell results:                   4,000 tokens
Model reasoning:                   8,000 tokens
──────────────────────────────────────────────
Total:                            44,000 tokens ← 22% of 200K

After 30 iterations:             ~130,000 tokens ← 65% of 200K
After 50 iterations:             ~200,000 tokens ← FULL

THE CONTEXT WINDOW FILLS UP FASTER THAN YOU THINK.

What Happens When the Context Window Fills Up

Three strategies agents use

1. TRUNCATION (cutting)
   → The oldest messages get removed
   → You lose context from the start of the conversation
   → "The agent forgets what you told it at the beginning"

2. COMPACTION (summarizing)
   → The previous conversation gets summarized into fewer tokens
   → Detail is lost but the essence is kept
   → Claude Code and Cursor use this strategy

3. SLIDING WINDOW
   → The N most recent messages are kept
   → The oldest ones get discarded
   → Similar to truncation but by messages, not by tokens

Practical impact

A FULL CONTEXT WINDOW MEANS:

1. LOSS OF COHERENCE
   → The agent can contradict itself
   → It "doesn't remember" decisions it made 20 turns ago
   → The initial plan can get lost

2. QUALITY DEGRADATION
   → The quality of the reasoning drops
   → More iterations ≠ more quality after a certain point
   → The "sweet spot" is before filling up 70-80%

3. A SIGNAL FOR YOU (THE DEVELOPER)
   → If the agent starts repeating itself or contradicting itself
   → The context window is probably full
   → It's time to start a new session

Tokens and Cost

How tokens affect cost

PROVIDERS CHARGE PER TOKEN:

Claude (example pricing, it varies):
→ Input: $X per million tokens
→ Output: $Y per million tokens (generally more expensive)

GPT-4 (example):
→ Input: $X per million tokens
→ Output: $Y per million tokens

THIS MEANS:
→ A large file the agent reads = more cost
→ A long prompt = more cost
→ A full context window = maximum cost per iteration
→ More loop iterations = more accumulated cost

PRACTICAL IMPLICATION:
→ Precise prompts → fewer iterations → less cost
→ Context curation → fewer unnecessary tokens → less cost
→ R→P→E→V → less rework → less total cost

Input vs output tokens

INPUT TOKENS (the ones the model RECEIVES):
→ Your prompt
→ System prompt
→ History
→ Files read
→ Tool definitions

OUTPUT TOKENS (the ones the model GENERATES):
→ Reasoning
→ Answers
→ Tool calls
→ Generated code

OUTPUT IS MORE EXPENSIVE than INPUT (generally 2-5x).
This means an agent that generates a lot of code
costs more than one that reads a lot of code.

Tokens and Coding Agents: The Connection

Why this matters for your workflow

1. EACH TOOL CALL CONSUMES TOKENS
   → file_read("large_file.ts") → the whole file enters the context
   → If the file has 1,000 lines → ~4,000 tokens
   → The agent reads 5 files → 20,000 tokens just in files

2. THE HISTORY GROWS WITH EACH ITERATION
   → Iteration 1: 5,000 tokens
   → Iteration 10: 50,000 tokens
   → Iteration 30: 150,000 tokens
   → The quality starts to degrade

3. THE CONTEXT WINDOW DETERMINES THE AGENT'S "MEMORY"
   → The agent "remembers" what fits in the context window
   → It has no permanent memory between sessions
   → Each new session starts with an empty context window

4. YOU CONTROL HOW MANY TOKENS GET USED
   → More precise prompts → fewer iterations → fewer tokens
   → Context curation → only what's relevant
   → R→P→E→V → less rework → fewer wasted tokens

Example: how the context window affects a real task

TASK: Refactor a 3-file module

APPROACH A (without a workflow):
→ "Refactor these 3 files" (vague prompt)
→ The agent reads the 3 files: 12,000 tokens
→ Generates refactoring: 8,000 tokens output
→ Tests fail → debug: 5 more iterations: 30,000 tokens
→ More debug: 3 iterations: 18,000 tokens
→ TOTAL: ~70,000 tokens

APPROACH B (with R→P→E→V):
→ Research: "Read these 3 files" → 12,000 tokens
→ Plan: "Design the refactoring" → 3,000 tokens
→ [Developer reviews the plan before continuing]
→ Execute step by step: 3 steps → 15,000 tokens
→ Validate: tests → 5,000 tokens
→ TOTAL: ~35,000 tokens

HALF THE TOKENS, BETTER RESULT.

Practical Exercise

Exercise 1: Count tokens

Use the OpenAI Tokenizer to count tokens:

1. Paste a paragraph of English text → how many tokens?
   Result: _____ tokens

2. Paste the same paragraph translated to Spanish → more or fewer tokens?
   Result: _____ tokens
   Difference: _____

3. Paste 20 lines of JavaScript code → how many tokens?
   Result: _____ tokens

4. Paste the same 20 lines rewritten in Python → how many?
   Result: _____ tokens

5. Paste a complex regex → how many tokens?
   Result: _____ tokens (it will probably surprise you)

WHAT DID YOU LEARN?
→ _______________________________________________
See solution

Expected observations:

  1. English vs. Spanish: Spanish text usually consumes between 10-20% more tokens than the English equivalent, because many words in Spanish are longer and less frequent in the tokenizer's vocabulary (trained mostly on English text).

  2. JavaScript vs. Python: JavaScript tends to use more tokens due to the syntax symbols ({, }, ;, =>). Python is more concise and generates fewer tokens for the same functionality.

  3. Regex: A single line of complex regex can consume 20-40 tokens because each special character (^, $, [, ], +, *, \) is usually an individual token.

Key conclusion: Not all text costs the same in tokens. Code with many symbols and text in languages other than English consume more tokens, which impacts how much fits in the context window.

Exercise 2: Estimate the context window usage

For your current project:

1. How many files does your project have? _____
2. How many lines in the largest file? _____
3. Estimate: largest file ≈ _____ tokens
4. If the agent reads 5 key files ≈ _____ tokens
5. What percentage of the context window (200K) is that? ____%

REFLECTION:
→ Does your whole project fit in the context window? _____
→ Why can't the agent "see everything" at once? _____
→ How does this affect the way you give instructions? _____
See solution

Guided estimation:

  • Estimation rule: A typical code file has ~500-800 tokens per 100 lines. A 300-line file ≈ 1,500-2,400 tokens.
  • 5 key files: If each file averages 200 lines, that's ~5,000-8,000 tokens just in files.
  • Percentage of the context window: With 200K tokens available, 8,000 tokens = only 4%. It seems like little, but remember that the system prompt, conversation history, tool definitions, and the model's response also compete for that space.
  • Does your whole project fit? Most real projects (thousands of files) do NOT fit entirely in a context window. That's why coding agents use selective search instead of loading everything.
  • Impact on instructions: You should be specific about which files are relevant. "Refactor the whole project" forces the agent to read many files. "Refactor the validateUser function in src/auth.ts" minimizes the tokens consumed.

Exercise 3: Experiment with the agent's "memory"

1. Start a conversation with your coding agent
2. Give it an instruction at the beginning: "Remember: X"
3. Ask it 10-15 questions/tasks about other topics
4. After 15 turns, ask: "What did I ask you to remember?"

Did it remember? □ Yes □ No □ Partially

If it didn't remember:
→ It's because the context window filled up
→ The first messages were truncated/summarized
→ It's not a bug — it's a fundamental limitation
See solution

Expected result:

  • After 10-15 turns, the agent will probably remember partially or not remember. The result depends on the model and the length of the intermediate turns.
  • If it remembered: Your turns were short and the context window didn't fill up. The original message is still in the history.
  • If it didn't remember or was partial: The context window filled up and the system applied truncation or compaction. The first messages were removed or summarized.
  • Takeaway: An LLM's "memory" isn't persistent — it's only what fits in the current context window. This explains why in long sessions the agent "forgets" instructions from the beginning, and why sometimes it's worth starting a new session instead of continuing to accumulate context.

Common Mistakes

MistakeReality
"One token = one word"A token can be part of a word, a word, or multiple characters
"200K tokens = I can paste my whole project"The context window is shared with the system prompt, history, tools, and output
"More context tokens = better"More context window helps, but quality degrades in the more distant parts
"The agent has memory between sessions"The context window resets with each new session
"Code and prose use the same amount of tokens"Code generally uses more tokens due to the amount of special symbols

Summary

TOKENS:
→ They're pieces of text the model processes as units
→ They aren't words — they can be more or less than a word
→ Code uses more tokens than prose (symbols, indentation)
→ Providers charge per token

CONTEXT WINDOW:
→ Total token limit the model can process
→ Includes EVERYTHING: system prompt, history, files, output
→ It fills up faster than you think with coding agents
→ When it fills up: truncation, compaction, or degradation

PRACTICAL IMPACT:
→ Each tool call consumes context window
→ The history grows with each iteration
→ Precise prompts = fewer tokens = better result
→ R→P→E→V optimizes token usage

FOR MODULE 07:
→ You'll configure max_tokens in the API
→ You'll see how many tokens your mini-agent consumes
→ You'll understand the real cost of each iteration

Next capsule: 03 - Inference and next-token prediction — how the model generates code, one token at a time.

Resources

  1. OpenAI Tokenizer — Interactive tool to visualize tokens
  2. Andrej Karpathy: Let's build GPT tokenizer — Detailed video on BPE
  3. Anthropic: Context Windows Guide — Context windows documentation
  4. tiktoken (Python library) — Library for counting tokens programmatically
  5. Hugging Face: Tokenizers — Advanced tokenization library
← PreviousModule 2: How LLMs Work (What You Need to Know as a Developer)Next →Inference and Next-Token Prediction