5. Context Window Management

Overview

The context window is one of the most critical and limited resources when you work with LLMs. Understanding how it works, how to measure it precisely, and which strategies to use when your content exceeds the limit is fundamental to building robust systems.

This capsule covers the three main strategies: summarization chains, sliding window, and hierarchical summarization, along with criteria for choosing when to use each one.

The context window problem

gpt-4o-mini: 128K tokens of context
gpt-4o: 128K tokens of context
claude-3-5-haiku: 200K tokens of context

How much is a token? Roughly:
- 1 token ≈ 0.75 words in English
- 1 token ≈ 0.5 words in Spanish (more tokens per character)
- 1 A4 page of text ≈ 500 tokens
- A book (200 pages) ≈ 100,000 tokens

The real problem:
- A company's annual report: 150,000 tokens → DOES NOT FIT in a single prompt
- A 10K-line codebase: ~100,000 tokens → Doesn't fit alongside the analysis prompt
- A 2-hour chat conversation: ~30,000 tokens → May run out of context

But even when it DOES fit, there's another problem: "Lost in the Middle"
The model has worse recall for information in the MIDDLE of the context window.
Best remembered: the beginning and the end.

Counting tokens accurately

from openai import OpenAI
import tiktoken
from typing import Union

client = OpenAI()

def count_tokens(text: str, model: str = "gpt-4o-mini") -> int:
    """
    Counts tokens accurately using tiktoken.
    Far more precise than the word-based heuristic.
    
    Args:
        text: Text to tokenize
        model: OpenAI model to count tokens for
    
    Returns:
        Exact number of tokens
    """
    try:
        encoding = tiktoken.encoding_for_model(model)
    except KeyError:
        # Fallback for new models
        encoding = tiktoken.get_encoding("cl100k_base")
    
    return len(encoding.encode(text))

def count_message_tokens(messages: list[dict], model: str = "gpt-4o-mini") -> int:
    """
    Counts tokens for a list of messages (chat format).
    Includes the overhead tokens of the chat format.
    """
    try:
        encoding = tiktoken.encoding_for_model(model)
    except KeyError:
        encoding = tiktoken.get_encoding("cl100k_base")
    
    # Overhead per message (approximate for GPT-4)
    tokens = 3  # Base system
    for msg in messages:
        tokens += 4  # Overhead per message
        tokens += len(encoding.encode(msg.get("content", "")))
        tokens += len(encoding.encode(msg.get("role", "")))
    
    return tokens

def chunks_by_tokens(
    text: str,
    max_tokens_per_chunk: int = 4000,
    overlap_tokens: int = 200,
    model: str = "gpt-4o-mini"
) -> list[str]:
    """
    Splits a text into chunks of a maximum token size.
    Includes overlap to keep coherence between chunks.
    
    Args:
        text: Text to split
        max_tokens_per_chunk: Maximum tokens per chunk
        overlap_tokens: Overlap tokens between consecutive chunks
        model: Model used to count tokens
    
    Returns:
        List of text chunks
    """
    try:
        encoding = tiktoken.encoding_for_model(model)
    except KeyError:
        encoding = tiktoken.get_encoding("cl100k_base")
    
    tokens = encoding.encode(text)
    
    chunks = []
    start = 0
    
    while start < len(tokens):
        end = min(start + max_tokens_per_chunk, len(tokens))
        
        chunk_tokens = tokens[start:end]
        chunk_text = encoding.decode(chunk_tokens)
        chunks.append(chunk_text)
        
        start = end - overlap_tokens  # Overlap for continuity
        
        if start >= len(tokens):
            break
    
    return chunks


# Example usage:
if __name__ == "__main__":
    long_text = "This is a sample text. " * 1000  # ~6K tokens
    
    total_tokens = count_tokens(long_text)
    print(f"Total tokens: {total_tokens:,}")
    
    chunks = chunks_by_tokens(long_text, max_tokens_per_chunk=1000, overlap_tokens=100)
    print(f"Number of chunks: {len(chunks)}")
    for i, chunk in enumerate(chunks[:3]):
        print(f"Chunk {i+1}: {count_tokens(chunk)} tokens")

Strategy 1: Summarization chain

The simplest strategy: split the text into chunks, summarize each one, and concatenate the summaries.

def summarization_chain(
    text: str,
    max_tokens_chunk: int = 4000,
    tokens_per_summary: int = 200,
    extra_instruction: str = "",
    verbose: bool = False
) -> str:
    """
    Basic summarization strategy:
    1. Split into chunks
    2. Summarize each chunk
    3. Concatenate the summaries
    
    Best for: Documents where a summary of each section is enough.
    Not ideal when: You need to preserve specific details.
    
    Args:
        text: Long text to summarize
        max_tokens_chunk: Maximum tokens per chunk
        tokens_per_summary: Tokens for each chunk's summary
        extra_instruction: What to preserve in the summary (e.g. "keep exact figures")
    
    Returns:
        Concatenated summarized text
    """
    chunks = chunks_by_tokens(text, max_tokens_chunk)
    
    if len(chunks) == 1:
        return text  # No need to split
    
    if verbose:
        print(f"Split into {len(chunks)} chunks for summarization")
    
    summaries = []
    for i, chunk in enumerate(chunks):
        if verbose:
            print(f"  Summarizing chunk {i+1}/{len(chunks)}...")
        
        prompt = f"""Summarize the following text fragment in {tokens_per_summary} words.
{f'Additional instruction: {extra_instruction}' if extra_instruction else ''}
Preserve: proper nouns, dates, specific numbers, main ideas.

Text:
{chunk}"""
        
        summary = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
            temperature=0,
            max_tokens=tokens_per_summary * 2
        ).choices[0].message.content
        
        summaries.append(summary)
    
    return "\n\n".join(summaries)


# Example with a specific instruction:
report_text = "TechCorp's annual report..." * 500  # Simulate a long text
summary = summarization_chain(
    report_text,
    extra_instruction="keep every financial figure and percentage",
    verbose=True
)
print(f"Original text: {count_tokens(report_text):,} tokens")
print(f"Summary: {count_tokens(summary):,} tokens")

Strategy 2: Sliding window

Processes the text in sliding windows with overlap. Useful when you need to keep contextual coherence.

def sliding_window_analysis(
    text: str,
    question: str,
    window_tokens: int = 3000,
    overlap_tokens: int = 300,
    combine: bool = True
) -> Union[list[str], str]:
    """
    Sliding window analysis.
    Useful for: finding specific information, coherent local analysis.
    
    Args:
        text: Long text to analyze
        question: The question or task to apply on each window
        window_tokens: Window size in tokens
        overlap_tokens: Overlap tokens between windows (for coherence)
        combine: If True, combines results into a final synthesis
    
    Returns:
        List of results per window, or a combined synthesis
    """
    chunks = chunks_by_tokens(text, window_tokens, overlap_tokens)
    
    results = []
    
    for i, chunk in enumerate(chunks):
        # Tell the model the context of the window
        context = f"[Fragment {i+1} of {len(chunks)} of the document]"
        
        prompt = f"""{context}
        
{question}

Fragment:
{chunk}

Answer based ONLY on this fragment. If the information is not in this fragment, say "Not found in this fragment"."""
        
        result = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
            temperature=0,
            max_tokens=400
        ).choices[0].message.content
        
        results.append({
            "fragment": i + 1,
            "total_fragments": len(chunks),
            "result": result
        })
    
    if not combine:
        return results
    
    # Combine results from all windows
    results_str = "\n\n".join([
        f"Fragment {r['fragment']}:\n{r['result']}"
        for r in results
        if "Not found" not in r['result']
    ])
    
    if not results_str.strip():
        return "The requested information was not found in any fragment of the document."
    
    synthesis = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"""Synthesize these per-fragment analysis results:

{results_str}

Produce a coherent answer with no repetition for: {question}"""}],
        temperature=0,
        max_tokens=500
    ).choices[0].message.content
    
    return synthesis

# Example: Find specific information in a long document
document = "Long contract text..." * 300
question = "What are the penalties for breach mentioned in the document?"
result = sliding_window_analysis(document, question, window_tokens=2000)
print(result)

Strategy 3: Hierarchical summarization

For very long documents, summarize across multiple levels.

def hierarchical_summarize(
    text: str,
    level_1_tokens: int = 3000,
    level_1_summary_tokens: int = 300,
    level_2_summary_tokens: int = 500,
    goal: str = "general summary"
) -> dict:
    """
    Hierarchical summarization for very long documents.
    
    Level 1: Summarize each chunk (fine granularity)
    Level 2: Summarize the level 1 summaries (medium granularity)
    Level 3: Final summary of the level 2 summaries (coarse granularity)
    
    Args:
        text: Very long text
        level_1_tokens: Chunk size for level 1
        level_1_summary_tokens: Tokens per summary at level 1
        level_2_summary_tokens: Tokens for the level 2 summary
        goal: The goal of the summary (steers the focus)
    
    Returns:
        dict with the summaries of each level
    """
    # LEVEL 1: Summarize each chunk
    print("Level 1: Summarizing individual chunks...")
    chunks_n1 = chunks_by_tokens(text, level_1_tokens)
    summaries_n1 = []
    
    for i, chunk in enumerate(chunks_n1):
        print(f"  Chunk {i+1}/{len(chunks_n1)}")
        summary = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"""Summarize this fragment for the goal: {goal}
Be concise but preserve key data (numbers, names, dates).
Target: {level_1_summary_tokens} words.

Fragment:
{chunk}"""}],
            temperature=0,
            max_tokens=level_1_summary_tokens * 2
        ).choices[0].message.content
        summaries_n1.append(summary)
    
    # Check whether level 1 is already small enough
    text_n1 = "\n\n---\n\n".join(summaries_n1)
    tokens_n1 = count_tokens(text_n1)
    
    if tokens_n1 <= 4000:
        # Level 1 already fits in context, do just one more level
        print(f"Level 2: Final synthesis (level 1 is already manageable: {tokens_n1} tokens)")
        final_summary = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"""Goal: {goal}
            
Based on these summaries of the document's sections, produce a coherent final summary:

{text_n1}

The final summary must be {level_2_summary_tokens} words with no repetition."""}],
            temperature=0,
            max_tokens=level_2_summary_tokens * 2
        ).choices[0].message.content
        
        return {
            "levels": 2,
            "n_chunks_level1": len(chunks_n1),
            "summaries_level1": summaries_n1,
            "final_summary": final_summary
        }
    
    # LEVEL 2: We need another summarization level
    print(f"Level 2: The level 1 summaries are still large ({tokens_n1} tokens). Summarizing level 2...")
    chunks_n2 = chunks_by_tokens(text_n1, level_1_tokens)
    summaries_n2 = []
    
    for i, chunk in enumerate(chunks_n2):
        summary = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"Synthesize these partial summaries (goal: {goal}):\n\n{chunk}"}],
            temperature=0,
            max_tokens=level_2_summary_tokens
        ).choices[0].message.content
        summaries_n2.append(summary)
    
    # FINAL LEVEL: Synthesis of level 2
    text_n2 = "\n\n".join(summaries_n2)
    final_summary = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"""Produce the final executive summary of the document.
Goal: {goal}

Based on:
{text_n2}

Be thorough but concise. Highlight the most important points."""}],
        temperature=0,
        max_tokens=level_2_summary_tokens * 2
    ).choices[0].message.content
    
    return {
        "levels": 3,
        "n_chunks_level1": len(chunks_n1),
        "n_chunks_level2": len(chunks_n2),
        "summaries_level1": summaries_n1,
        "summaries_level2": summaries_n2,
        "final_summary": final_summary
    }

When to use each strategy

def select_context_strategy(
    text: str,
    task: str,
    model: str = "gpt-4o-mini"
) -> dict:
    """
    Recommends the context management strategy based on the size of the text
    and the type of task.
    
    Returns:
        dict with the recommended strategy and its rationale
    """
    text_tokens = count_tokens(text, model)
    
    MODEL_LIMITS = {
        "gpt-4o-mini": 128_000,
        "gpt-4o": 128_000,
        "claude-3-5-haiku-20241022": 200_000
    }
    
    limit = MODEL_LIMITS.get(model, 128_000)
    available_tokens = limit - 2000  # Reserve room for the prompt and the answer
    
    # Task types that require querying section by section
    search_tasks = ["find", "search", "extract specific", "where", "when", "who"]
    is_search = any(t in task.lower() for t in search_tasks)
    
    if text_tokens <= available_tokens * 0.8:
        # The text fits comfortably in context
        return {
            "strategy": "direct_context",
            "description": "The text fits in the context window. Use it directly.",
            "text_tokens": text_tokens,
            "available_tokens": available_tokens,
            "estimated_chunks": 1,
            "estimated_calls": 1
        }
    
    if text_tokens <= available_tokens * 2:
        # Slightly exceeds the context
        if is_search:
            return {
                "strategy": "sliding_window",
                "description": "Section-by-section search with overlap to find specific information.",
                "text_tokens": text_tokens,
                "estimated_chunks": 2,
                "estimated_calls": 3  # 2 chunks + 1 synthesis
            }
        return {
            "strategy": "summarization_chain",
            "description": "Summarize into 2-3 chunks, concatenate.",
            "text_tokens": text_tokens,
            "estimated_chunks": 2,
            "estimated_calls": 3
        }
    
    if text_tokens <= available_tokens * 10:
        # Moderately long
        if is_search:
            return {
                "strategy": "sliding_window",
                "description": "Sliding window with a final synthesis.",
                "text_tokens": text_tokens,
                "estimated_chunks": text_tokens // 3000,
                "estimated_calls": (text_tokens // 3000) + 1
            }
        return {
            "strategy": "hierarchical_summarization_2_levels",
            "description": "Summarize chunks, then synthesize the summaries.",
            "text_tokens": text_tokens,
            "estimated_chunks": text_tokens // 3000,
            "estimated_calls": text_tokens // 3000 + 1
        }
    
    # Very long: needs the full hierarchy
    return {
        "strategy": "hierarchical_summarization_3_levels",
        "description": "Summarization across 3 hierarchical levels.",
        "text_tokens": text_tokens,
        "estimated_chunks": text_tokens // 3000,
        "estimated_calls": text_tokens // 2000  # Rough estimate
    }

Strategy comparison table

Integrating with Anthropic (Claude)

import anthropic

client_anthropic = anthropic.Anthropic()

def summarization_chain_claude(
    text: str,
    max_chars_chunk: int = 15000,
    instruction: str = "Executive summary preserving key data"
) -> str:
    """
    Summarization chain using Claude.
    Claude has a 200K token window, but the same strategy
    applies for extremely long documents.
    """
    # Claude-3-5-haiku supports up to 200K tokens of context
    if len(text) < 150000:  # Roughly 100K tokens
        # The text fits directly
        message = client_anthropic.messages.create(
            model="claude-3-5-haiku-20241022",
            max_tokens=1000,
            messages=[{"role": "user", "content": f"{instruction}:\n\n{text}"}]
        )
        return message.content[0].text
    
    # Split into chunks if it's too long
    chunks = [text[i:i+max_chars_chunk] for i in range(0, len(text), max_chars_chunk)]
    summaries = []
    
    for chunk in chunks:
        message = client_anthropic.messages.create(
            model="claude-3-5-haiku-20241022",
            max_tokens=400,
            messages=[{"role": "user", "content": f"Summarize in 300 words, preserving important data:\n\n{chunk}"}]
        )
        summaries.append(message.content[0].text)
    
    # Final synthesis
    synthesis_input = "\n\n---\n\n".join(summaries)
    synthesis_msg = client_anthropic.messages.create(
        model="claude-3-5-haiku-20241022",
        max_tokens=800,
        messages=[{"role": "user", "content": f"Synthesize these partial summaries into a coherent executive report:\n\n{synthesis_input}"}]
    )
    return synthesis_msg.content[0].text

Handling "Lost in the Middle"

def process_with_optimal_position(
    long_context: str,
    question: str,
    n_key_fragments: int = 3
) -> str:
    """
    Mitigates the "Lost in the Middle" problem by placing the most relevant
    information at the beginning and at the end of the context.
    
    The model remembers the start and the end of the context window best.
    """
    # Step 1: Identify the most relevant fragments
    chunks = chunks_by_tokens(long_context, max_tokens_per_chunk=1000)
    
    relevance = []
    for i, chunk in enumerate(chunks):
        score_resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"How relevant is this fragment for answering: '{question}'?\nScore 0-10. The number only.\n\nFragment: {chunk[:300]}"}],
            temperature=0,
            max_tokens=5
        ).choices[0].message.content.strip()
        
        try:
            score = float(score_resp[:3])
        except ValueError:
            score = 5.0
        relevance.append((score, i, chunk))
    
    # Sort by relevance and take the best ones
    relevance.sort(reverse=True)
    key_fragments = relevance[:n_key_fragments]
    
    # Step 2: Place the key fragments at the beginning of the context
    optimized_context = "MOST RELEVANT FRAGMENTS (for your reference):\n\n"
    for score, i, chunk in sorted(key_fragments, key=lambda x: x[0], reverse=True):
        optimized_context += f"[Fragment {i+1}, relevance {score:.1f}/10]:\n{chunk}\n\n"
    
    # Step 3: Answer with the optimized context
    answer = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": f"""{optimized_context}

Based on these relevant fragments, answer:
{question}"""
        }],
        temperature=0,
        max_tokens=500
    ).choices[0].message.content
    
    return answer

Troubleshooting

Problem 1: Information lost in the chunks

Symptom: A chunk's summary doesn't mention important data that was in it.

Cause: The summary prompt is too generic and the model decides what to preserve.

Solution:

def summarize_preserving_specifics(
    chunk: str,
    types_to_preserve: list[str] = None
) -> str:
    """Summary that guarantees preservation of specific data types."""
    if types_to_preserve is None:
        types_to_preserve = ["numbers and percentages", "proper nouns", "dates", "URLs or references"]
    
    types_str = ", ".join(types_to_preserve)
    
    return client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"""Summarize the following text (max 200 words).
        
CRITICAL: Preserve LITERALLY every: {types_str}
If you can't fit them into the summary, list them at the end under "Key data:"

Text: {chunk}"""}],
        temperature=0,
        max_tokens=350
    ).choices[0].message.content

Problem 2: Incoherence between summaries of consecutive chunks

Symptom: The final summary reads disjointed; each chunk seems unrelated to the previous one.

Solution: Pass the previous chunk's context into the next one:

def summarization_chain_with_context(text: str, max_tokens_chunk: int = 3000) -> str:
    """Summarization where each chunk knows the context of the previous one."""
    chunks = chunks_by_tokens(text, max_tokens_chunk)
    summaries = []
    previous_context = ""
    
    for i, chunk in enumerate(chunks):
        prompt = f"""Summarize this fragment ({i+1} of {len(chunks)}).
{'Context from previous fragments: ' + previous_context[:300] if previous_context else ''}

Current fragment:
{chunk}

Produce a 150-200 word summary that is coherent with the previous context."""
        
        summary = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}],
            temperature=0,
            max_tokens=300
        ).choices[0].message.content
        
        summaries.append(summary)
        previous_context = summary  # The current summary becomes context for the next one
    
    return "\n\n".join(summaries)

Problem 3: Very high cost with many chunks

Symptom: A 50K-token document with 2K chunks = 25 calls just for the first level.

Solution: Use larger chunks and summarize in 2 passes if needed:

def efficient_summarization(text: str, call_budget: int = 10) -> str:
    """
    Summarization that respects a budget of API calls.
    Adjusts the chunk size automatically.
    """
    total_tokens = count_tokens(text)
    tokens_per_call = total_tokens // call_budget
    tokens_per_call = max(tokens_per_call, 2000)  # Minimum 2K per chunk
    
    print(f"Text: {total_tokens:,} tokens, Budget: {call_budget} calls")
    print(f"Adjusted chunk size: {tokens_per_call:,} tokens")
    
    return summarization_chain(text, max_tokens_chunk=tokens_per_call)

Exercises

Exercise 1: Build an extractor for specific information

Write a function that uses a sliding window to extract ALL mentions of dates, amounts, and people's names from a long document. The result must be a deduplicated, sorted list.

import re

def extract_entities_sliding_window(document: str) -> dict:
    """Extracts specific entities using a sliding window."""
    chunks = chunks_by_tokens(document, max_tokens_per_chunk=2000, overlap_tokens=200)
    
    dates = set()
    amounts = set()
    people = set()
    
    for i, chunk in enumerate(chunks):
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"""Extract from the text:
1. Dates (format: DD/MM/YYYY or text such as "January 2026")
2. Monetary amounts (€/$) with the exact value
3. People's names

Respond in JSON: {{"dates": [], "amounts": [], "people": []}}

Text: {chunk}"""}],
            temperature=0,
            response_format={"type": "json_object"}
        ).choices[0].message.content
        
        try:
            data = json.loads(response)
            dates.update(data.get("dates", []))
            amounts.update(data.get("amounts", []))
            people.update(data.get("people", []))
        except Exception:
            pass
    
    return {
        "dates": sorted(list(dates)),
        "amounts": sorted(list(amounts)),
        "people": sorted(list(people))
    }

Exercise 2: Compare strategies

For a ~10K token document, measure and compare:

• Direct context (if it fits)
• Summarization chain (2K chunks)
• Hierarchical (2 levels)

Compare: summary quality, number of calls, tokens used.

import time

def compare_strategies(text: str, evaluation_question: str) -> dict:
    """Compares the 3 strategies on the same document."""
    tokens = count_tokens(text)
    results = {}
    
    # Strategy 1: Direct (if it fits)
    if tokens < 100000:
        t0 = time.time()
        resp = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": f"Summarize: {text}"}],
            temperature=0, max_tokens=400
        ).choices[0].message.content
        results["direct"] = {
            "summary": resp, "time": time.time()-t0, "calls": 1,
            "tokens_input": tokens
        }
    
    # Strategy 2: Summarization chain
    t0 = time.time()
    summary_sc = summarization_chain(text, max_tokens_chunk=2000, verbose=False)
    results["summarization_chain"] = {
        "summary": summary_sc, "time": time.time()-t0,
        "calls": len(chunks_by_tokens(text, 2000)),
        "tokens_input": tokens
    }
    
    # Compare
    print(f"Original tokens: {tokens:,}")
    for strategy, data in results.items():
        print(f"\n{strategy}:")
        print(f"  Time: {data['time']:.2f}s")
        print(f"  Calls: {data['calls']}")
        print(f"  Summary: {data['summary'][:100]}...")
    
    return results

Summary

• Direct context: Whenever the text fits (~80% of the limit to leave headroom). Best quality.
• Summarization chain: For moderately long documents (2-3x the limit). Simple and effective.
• Sliding window: For searching specific information. Preserves local detail.
• Hierarchical: For very long documents (>10x the limit). Loses some detail at each level.
• Counting tokens: Use tiktoken, not heuristic estimates.
• Lost in the Middle: Place critical information at the start or the end of the context.

Additional resources

1. tiktoken - OpenAI tokenizer
2. Lost in the Middle (Liu et al., 2023) - The original paper on the phenomenon
3. OpenAI Token counting cookbook
4. Anthropic Claude context window documentation
5. Long Context Best Practices - Anthropic
6. LangChain text splitters