Module 5: LLMs (Large Language Models) - GPT, Claude and More

5. Context Window: LLMs' Memory Limit

Next →

Description

Context window: The maximum number of tokens an LLM can process in a single request (input + output).

It's one of LLMs' most important limitations. In this lesson you'll understand:

  • What the context window is (a memory limit).
  • Why it exists (attention scales quadratically).
  • A comparison of LLMs (GPT-4: 128K, Claude 3: 200K).
  • Solutions (RAG, sliding window, summarizing).

What the Context Window is

Definition: The maximum number of tokens an LLM can process in a single conversation (prompt + accumulated response).

Example (GPT-4):

  • Context window: 128,000 tokens (~96,000 words in English).
  • If your prompt uses 100K tokens, only 28K tokens are left for the response.
  • If you exceed 128K tokens → an error or truncation.

Comparison of Context Windows

ModelContext WindowEquivalent (English)Year
GPT-34K tokens~3K words (~10 pages)2020
GPT-3.516K tokens~12K words (~40 pages)2022
GPT-48K / 32K / 128K tokens~6K / 24K / 96K words2023
Claude 2100K tokens~75K words (~250 pages)2023
Claude 3200K tokens~150K words (~500 pages)2024
Llama 38K / 32K tokens~6K / 24K words2024
Gemini 1.51M tokens~750K words (~2,500 pages)2024

Moral: Context windows have grown rapidly (4K → 1M in 4 years), but they're still limited.

Why the Limit Exists

Attention scales quadratically

Reminder from Module 4: Attention compares each token with all the other tokens (N × N comparisons).

Consequence:

  • 4K tokens: 16M comparisons.
  • 128K tokens: 16B comparisons (1,000× more).
  • 1M tokens: 1T comparisons (62,500× more).

Implication: Large context windows are slow and expensive (they require massive GPUs).

Example: Exceeding the Context Window

Scenario: A chatbot that analyzes a 200-page document (~150K words → ~200K tokens in English).

Problem: GPT-4 (128K tokens) can't process the complete document + the prompt.

What happens:

  1. Option A: The API rejects the request with a context_length_exceeded error.
  2. Option B: The model truncates the input (it only processes the first 128K tokens) → information is lost.

Solution: See the solutions section below.

Solutions to the Context Window Limit

Solution 1: RAG (Retrieval-Augmented Generation)

Idea: Instead of passing the complete document, search for the relevant sections and pass only those.

Process:

  1. Index: Split the document into chunks (e.g. 1K tokens each), generate embeddings.
  2. Search: Given a query, look for the most similar chunks (semantic search).
  3. Generate: Pass the top-K relevant chunks + the query to the LLM.

Advantage: You only use ~10-20K tokens (the relevant chunks) instead of 200K (the complete document).

Solution 2: Sliding Window

Idea: Process the document in overlapping windows.

Process:

  1. Split the document into chunks of 100K tokens each (with a 10K overlap).
  2. Process chunk 1 → generate a partial summary/answer.
  3. Process chunk 2 → generate a partial summary/answer.
  4. Combine the partial answers.

Disadvantage: It loses the global context (each chunk is processed independently).

Solution 3: Summarize Iteratively

Idea: Summarize the document first, then process the summary.

Process:

  1. Split the document into chunks.
  2. Summarize each chunk (e.g. from 50K tokens to 5K tokens).
  3. Combine the summaries → a summarized document (e.g. 30K tokens).
  4. Process the summarized document with the LLM.

Disadvantage: It loses details.

Solution 4: Use an LLM with a Larger Context Window

Options:

  • Claude 3: 200K tokens (vs GPT-4: 128K).
  • Gemini 1.5: 1M tokens.

Advantage: You process more without splitting.

Disadvantage: It can be more expensive or less capable (depending on the model).

Implications for Applications

1. Chatbots with Long History

Problem: A long conversation (50+ messages) → the history grows → it exceeds the context window.

Solution:

  • Summarize the history: Every 10 messages, summarize the earlier conversation.
  • Sliding window: Keep only the last N messages (e.g. the last 20).

2. Document Analysis

Problem: Long documents (contracts, papers) → they exceed the context window.

Solution: RAG (searching for the relevant sections based on the question).

3. Code Assistants

Problem: A large codebase (100K+ lines) → it doesn't fit in the context window.

Solution:

  • RAG: Index the functions/classes, search for the relevant ones based on the query.
  • Pass only the relevant files: Instead of the whole codebase, pass only the files the user is editing + their imports.

Why this matters for an AI Engineer

1. Application design

Question: Does your app need to process long documents?

  • Yes: You need RAG or an LLM with a large context window.
  • No: GPT-3.5 (16K tokens) may be enough (cheaper).

2. Debugging

A common error: context_length_exceeded

Diagnosis:

import tiktoken

encoding = tiktoken.encoding_for_model("gpt-4")
prompt_tokens = len(encoding.encode(prompt))
document_tokens = len(encoding.encode(document))
total = prompt_tokens + document_tokens

print(f"Total: {total} tokens")
if total > 128000:
    print("Exceeds GPT-4's context window")

3. Cost optimization

Example: You use Claude 3 (200K tokens) when you only need 16K.

Problem: Claude 3 is more expensive than GPT-3.5.

Solution: Use GPT-3.5 (16K tokens, cheaper) if your use case doesn't require 200K.

Summary

Context window:

  • The limit on tokens an LLM can process (input + output).
  • GPT-4: 128K, Claude 3: 200K, Gemini 1.5: 1M.

Why it exists:

  • Attention scales quadratically (N² comparisons).
  • Large context windows are slow and expensive.

Solutions:

  1. RAG: Search for the relevant chunks (don't pass the complete document).
  2. Sliding window: Process in overlapping windows.
  3. Summarize: Compress the document before processing.
  4. Use an LLM with a larger context window: Claude 3, Gemini 1.5.

Why it matters:

  • It limits which documents you can process.
  • RAG is the key technique for overcoming the limitation.
  • LLM selection depends on the context window you need.

Next step: Lesson 06: Model Parameters — Temperature, top_p, max_tokens → controlling generation.

← Previous4. Embeddings: Vector Representations of MeaningNext →6. Model Parameters: Controlling Generation