Local Models, Caching and Rate Limiting

Capsule overview

So far you've worked exclusively with models in the cloud: every call to invoke() travels across the internet to OpenAI's, Anthropic's or Google's servers. That works, but it comes with three real problems: it costs money per token, you're subject to rate limits that can block your application, and you depend on an internet connection. In this capsule you'll learn to solve all three.

First you'll explore Ollama to run models locally on your machine — no API keys, no costs, no internet connection. Then you'll look at prompt caching to cut costs when you repeat the same prompt prefixes across multiple calls. Next you'll use InMemoryRateLimiter to keep your calls from getting blocked for exceeding API quotas. And you'll learn to monitor exactly how many tokens each call consumes with usage_metadata.

The capsule closes with a practical section you'll reach for often: a mapping from legacy APIs to modern LangChain APIs. When you run into old tutorials that use LLMChain or AgentExecutor, you'll know exactly what to replace them with in LangChain v1.2+.

Local models with Ollama

Why run models locally?

Every call to a cloud model has a cost. If you're prototyping, experimenting with prompts, or processing sensitive data you can't send to external servers, running a model locally eliminates those three problems in one shot.

Ollama is the simplest way to run language models on your machine. It acts as a local server that exposes a compatible API — and LangChain connects to it exactly the same way it connects to OpenAI or Anthropic.

Install and configure Ollama

# Install: macOS
brew install ollama
# Linux: curl -fsSL https://ollama.ai/install.sh | sh
# Windows: download from https://ollama.ai/download

# Download a model
ollama pull llama3.2     # Lightweight model (~2GB), good for development
ollama pull llama3.1     # More capable model (~5GB), better quality

# Check which models are available
ollama list

# Check that the server is running (it starts automatically)
ollama serve
# Output: Listening on 127.0.0.1:11434

Using local models with LangChain

Once Ollama is running, you use init_chat_model with the ollama: prefix — identical to any other provider:

from langchain.chat_models import init_chat_model

model = init_chat_model("ollama:llama3.2")

response = model.invoke("Explain what a REST API is in one sentence")
print(response.content)
# Expected output: A REST API is an interface that lets applications
# communicate with each other over the HTTP protocol using standard
# operations like GET, POST, PUT and DELETE.

You don't need an API key. You don't need .env. You don't need an internet connection.

Cloud vs Local: when to use each

Rule of thumb:

• ✅ Use cloud for production, complex tasks, or when quality is critical
• ✅ Use local for development, prototyping, sensitive data, or when you want to iterate without cost

Prompt caching

How it works

When you call a model, the provider processes the entire prompt from scratch every time — including the system prompt. If you send the same 2,000-token system prompt in 100 consecutive calls, you pay to process those 2,000 tokens 100 times.

Prompt caching solves this: the provider detects that a prefix of the prompt was already processed recently and reuses the cached result. It only processes (and charges for) the new tokens.

Without caching:
  Call 1: [System prompt: 2000 tokens] + [User: 50 tokens] → Charges 2050 tokens
  Call 2: [System prompt: 2000 tokens] + [User: 30 tokens] → Charges 2030 tokens
  Call 3: [System prompt: 2000 tokens] + [User: 45 tokens] → Charges 2045 tokens
  Total processed: 6,125 tokens

With caching:
  Call 1: [System prompt: 2000 tokens] + [User: 50 tokens] → Charges 2050 tokens (cache miss)
  Call 2: [System prompt: CACHED]      + [User: 30 tokens] → Charges ~530 tokens (cache hit)
  Call 3: [System prompt: CACHED]      + [User: 45 tokens] → Charges ~545 tokens (cache hit)
  Total processed: ~3,125 tokens (~50% savings)

Support by provider

When it helps

Caching pays off when:

• ✅ You use the same long system prompt across multiple calls
• ✅ You process many documents with the same analysis instructions
• ✅ You run an agent that keeps a constant base prompt between iterations

It's not worth it when:

• ❌ Every call has a completely different prompt
• ❌ Your system prompt is short (< 500 tokens)
• ❌ You make few calls (the cache expires quickly)

Note: Prompt caching is a provider-side optimization that works transparently. You don't need to change your code — it simply happens when your usage pattern allows it. What matters is understanding when it benefits you, so you can design your applications to take advantage of it. We'll go deeper into prompt caching with practical examples in Module 12 (LangSmith and Production).

Rate limiting with InMemoryRateLimiter

The problem: rate limit errors

Every API provider has rate limits. If you send too many requests per minute, you get a 429 Too Many Requests error and your application stops.

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model

model = init_chat_model("openai:gpt-4.1-mini")

# This can fail with a rate limit error if you fire off many calls quickly
for i in range(50):
    response = model.invoke(f"Question number {i}")
    print(f"{i}: {response.content[:30]}...")

# Possible error:
# openai.RateLimitError: Error code: 429 - Rate limit reached

You can use max_retries to retry (capsule 03), but that doesn't fix the root problem: you're sending requests faster than the API allows.

The solution: InMemoryRateLimiter

InMemoryRateLimiter controls the pace of your calls before they leave your application. Instead of firing 50 instant requests and hoping the API accepts them, it spaces them out automatically.

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model
from langchain_core.rate_limiters import InMemoryRateLimiter

rate_limiter = InMemoryRateLimiter(requests_per_second=1)

model = init_chat_model(
    "openai:gpt-4.1-mini",
    rate_limiter=rate_limiter
)

response = model.invoke("What is rate limiting?")
print(response.content)
# Expected output: Rate limiting is a technique that controls how many
# requests a client can make to a service within a given period
# of time.

You can tune the parameters: requests_per_second (maximum pace), check_every_n_seconds (how often it checks, default 0.1), and max_bucket_size (allowed burst, default 10).

Practical example: batch processing with rate limiting

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model
from langchain_core.rate_limiters import InMemoryRateLimiter
import time

rate_limiter = InMemoryRateLimiter(requests_per_second=2)

model = init_chat_model(
    "openai:gpt-4.1-mini",
    rate_limiter=rate_limiter
)

questions = [
    "What is Python?",
    "What is JavaScript?",
    "What is Rust?",
    "What is Go?",
    "What is TypeScript?",
]

start = time.time()

for i, question in enumerate(questions):
    response = model.invoke(question)
    elapsed = time.time() - start
    print(f"[{elapsed:.1f}s] {question} → {response.content[:50]}...")

# Expected output (notice the spacing in time):
# [0.5s] What is Python? → Python is a programming language...
# [1.1s] What is JavaScript? → JavaScript is a programming langu...
# [1.6s] What is Rust? → Rust is a systems programming language...
# [2.2s] What is Go? → Go is a programming language created by...
# [2.7s] What is TypeScript? → TypeScript is a typed superset...

The calls get spaced out automatically to ~0.5s between each one (2 requests/second).

Token usage tracking

Monitoring token consumption

Every model response includes metadata about how many tokens were consumed. You access it through response.usage_metadata:

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model

model = init_chat_model("openai:gpt-4.1-mini")

response = model.invoke("Hi, how are you?")
print(response.content)
print(response.usage_metadata)
# Expected output:
# Hi there! I'm doing well, thanks for asking. How can I help you?
# {'input_tokens': 8, 'output_tokens': 15, 'total_tokens': 23}

Calculating costs

Every provider has per-token pricing. You can compute the real cost of each call:

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model

PRICING = {
    "gpt-4.1-mini": {"input": 0.40 / 1_000_000, "output": 1.60 / 1_000_000},
    "gpt-4.1":      {"input": 2.00 / 1_000_000, "output": 8.00 / 1_000_000},
    "gpt-4.1-nano": {"input": 0.10 / 1_000_000, "output": 0.40 / 1_000_000},
}

def invoke_with_cost(model_name: str, question: str) -> dict:
    """Invokes the model and computes the cost of the call."""
    model = init_chat_model(f"openai:{model_name}")
    response = model.invoke(question)

    tokens = response.usage_metadata
    pricing = PRICING[model_name]

    cost_input = tokens["input_tokens"] * pricing["input"]
    cost_output = tokens["output_tokens"] * pricing["output"]
    total_cost = cost_input + cost_output

    return {
        "content": response.content,
        "input_tokens": tokens["input_tokens"],
        "output_tokens": tokens["output_tokens"],
        "cost_usd": total_cost,
    }

result = invoke_with_cost("gpt-4.1-mini", "What is Docker in one sentence?")
print(f"Response: {result['content']}")
print(f"Tokens: {result['input_tokens']} input + {result['output_tokens']} output")
print(f"Cost: ${result['cost_usd']:.6f} USD")
# Expected output:
# Response: Docker is a platform that lets you package applications
# into isolated containers so they run consistently in
# any environment.
# Tokens: 12 input + 25 output
# Cost: $0.000045 USD

In Exercise 3 of this capsule you'll build a CostTracker class that accumulates tokens and costs across multiple calls — an essential pattern for production.

Mapping: Legacy API → Modern API

Why you need this table

LangChain went through a major transformation with version 1.0+ (October 2025). Many of the APIs you'll find in tutorials, on Stack Overflow, and in old documentation are deprecated or replaced. If you copy code from a 2024 tutorial, it probably won't work — or it'll work with deprecation warnings.

This table gives you the modern equivalent of every legacy API. Use it as a reference every time you run into old code.

Equivalence table

Example: LLMChain → model.invoke()

# ❌ Legacy (pre-v1.0) — DON'T use this
# from langchain.llms import OpenAI
# from langchain.chains import LLMChain
# from langchain.prompts import PromptTemplate
#
# llm = OpenAI(temperature=0.7)
# prompt = PromptTemplate(
#     input_variables=["topic"],
#     template="Explain {topic} in one sentence."
# )
# chain = LLMChain(llm=llm, prompt=prompt)
# result = chain.run(topic="Docker")

# ✅ Modern (v1.2+) — Use this
from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model

model = init_chat_model("openai:gpt-4.1-mini", temperature=0.7)
response = model.invoke("Explain Docker in one sentence.")
print(response.content)
# Expected output: Docker is a container platform that packages
# applications with all their dependencies so they run
# consistently in any environment.

Example: OutputParser → with_structured_output

# ❌ Legacy — manual OutputParser
# from langchain.output_parsers import PydanticOutputParser
# parser = PydanticOutputParser(pydantic_object=Movie)
# prompt = prompt_template.format(format_instructions=parser.get_format_instructions())

# ✅ Modern — native structured output
from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model
from pydantic import BaseModel, Field

class Movie(BaseModel):
    title: str = Field(description="The movie's title")
    year: int = Field(description="Release year")
    genre: str = Field(description="Main genre")

model = init_chat_model("openai:gpt-4.1-mini")
structured_model = model.with_structured_output(Movie)

result = structured_model.invoke("Give me information about Inception")
print(f"{result.title} ({result.year}) - {result.genre}")
# Expected output: Inception (2010) - Science Fiction

Connection with the project

In this module's project (Capsule 08), you'll build a multi-provider chat with automatic fallback. The concepts from this capsule apply directly:

• Ollama as the last-resort fallback: if OpenAI and Anthropic fail (API down, rate limit), the system can fall back to a local model with Ollama. No cost, no dependency on the internet.
• Rate limiting: when the system gets a lot of requests, InMemoryRateLimiter keeps it from getting blocked for exceeding the cloud providers' quotas.
• Token tracking: the project reports metadata for each response, including tokens consumed and the provider used — computed with usage_metadata.
• Modern APIs: the whole project uses init_chat_model, invoke(), and with_structured_output — the modern APIs you learned in this module.

Troubleshooting

Problem 1: "Connection refused" when using Ollama

ConnectionError: Connection refused - connect(2) for "127.0.0.1" port 11434

Cause: The Ollama server isn't running.

Fix:

# Start the server
ollama serve

# Check that it responds
curl http://localhost:11434
# It should return: "Ollama is running"

Problem 2: Ollama model not found

Error: model "llama3.1" not found, try pulling it first

Cause: You didn't download the model before using it.

Fix:

# Download the model first
ollama pull llama3.1

# Check which models are available
ollama list

Problem 3: Rate limit error even though you're using InMemoryRateLimiter

Cause: Your requests_per_second is higher than the real limit of your API tier. Fix: Lower the value (e.g. requests_per_second=0.5) until the errors go away.

Problem 4: usage_metadata is None

Cause: Some providers don't return usage metadata (Ollama, local models). Fix: Check before you access it: if response.usage_metadata:.

Problem 5: Slow responses with Ollama

Cause: Not enough hardware for the model. Fix: Use a smaller model (ollama:llama3.2 instead of llama3.1), check the GPU with ollama ps, and close apps that are eating memory.

Exercises

Exercise 1: Your first local model (Basic)

Install Ollama, download the llama3.2 model, and make a call with init_chat_model. Print the response and confirm that no API key was needed.

from langchain.chat_models import init_chat_model

model = init_chat_model("ollama:llama3.2")

response = model.invoke("What's the difference between a list and a tuple in Python?")
print(response.content)
print(f"\nResponse type: {type(response)}")
print(f"Usage metadata: {response.usage_metadata}")
# Expected output:
# The main difference is that lists are mutable (you can modify
# their elements) while tuples are immutable (once created,
# you can't change their elements). Lists use square brackets [] and
# tuples use parentheses ().
#
# Response type: <class 'langchain_core.messages.ai.AIMessage'>
# Usage metadata: None (or a dict with tokens if the model supports it)

Exercise 2: Rate limiter for batch processing (Basic)

Create a model with a rate limiter of 1 request/second and process a list of 5 questions. Measure the total time to confirm that the rate limiting is working.

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model
from langchain_core.rate_limiters import InMemoryRateLimiter
import time

rate_limiter = InMemoryRateLimiter(requests_per_second=1)
model = init_chat_model("openai:gpt-4.1-mini", rate_limiter=rate_limiter)

questions = [
    "What is HTML?",
    "What is CSS?",
    "What is JavaScript?",
    "What is React?",
    "What is Node.js?",
]

start = time.time()

for question in questions:
    response = model.invoke(question)
    elapsed = time.time() - start
    print(f"[{elapsed:.1f}s] {question} → {response.content[:40]}...")

total = time.time() - start
print(f"\nTotal time: {total:.1f}s (expected: ~5s at 1 req/s)")
# Expected output:
# [0.8s] What is HTML? → HTML is the standard markup langua...
# [1.9s] What is CSS? → CSS is a style sheet language...
# [2.9s] What is JavaScript? → JavaScript is a programming lan...
# [4.0s] What is React? → React is a JavaScript library...
# [5.1s] What is Node.js? → Node.js is a runtime environment...
#
# Total time: 5.1s (expected: ~5s at 1 req/s)

Exercise 3: Per-session cost tracker (Medium)

Create a CostTracker class that accumulates the tokens and costs of multiple calls. It should have track(response) and summary() methods.

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model

class CostTracker:
    PRICING = {
        "gpt-4.1-mini": {"input": 0.40 / 1_000_000, "output": 1.60 / 1_000_000},
        "gpt-4.1":      {"input": 2.00 / 1_000_000, "output": 8.00 / 1_000_000},
    }

    def __init__(self, model_name: str = "gpt-4.1-mini"):
        self.model_name = model_name
        self.total_input = 0
        self.total_output = 0
        self.call_count = 0

    def track(self, response) -> None:
        """Records the tokens from a response."""
        if response.usage_metadata:
            self.total_input += response.usage_metadata["input_tokens"]
            self.total_output += response.usage_metadata["output_tokens"]
            self.call_count += 1

    def summary(self) -> dict:
        """Returns a summary of usage and cost."""
        pricing = self.PRICING.get(self.model_name, {"input": 0, "output": 0})
        cost = (
            self.total_input * pricing["input"]
            + self.total_output * pricing["output"]
        )
        return {
            "calls": self.call_count,
            "input_tokens": self.total_input,
            "output_tokens": self.total_output,
            "total_tokens": self.total_input + self.total_output,
            "cost_usd": cost,
        }

tracker = CostTracker("gpt-4.1-mini")
model = init_chat_model("openai:gpt-4.1-mini")

questions = ["What is Git?", "What is Docker?", "What is Kubernetes?"]

for question in questions:
    response = model.invoke(question)
    tracker.track(response)
    print(f"{question} → {response.usage_metadata['total_tokens']} tokens")

s = tracker.summary()
print(f"\n--- Summary ---")
print(f"Calls: {s['calls']}")
print(f"Total tokens: {s['total_tokens']}")
print(f"Estimated cost: ${s['cost_usd']:.6f} USD")
# Expected output:
# What is Git? → 48 tokens
# What is Docker? → 52 tokens
# What is Kubernetes? → 61 tokens
#
# --- Summary ---
# Calls: 3
# Total tokens: 161
# Estimated cost: $0.000212 USD

Exercise 4: Migrate legacy code to the modern API (Medium)

Given the following legacy code, rewrite it using the modern LangChain v1.2+ APIs.

# Legacy code to migrate:
# from langchain.llms import OpenAI
# from langchain.chains import LLMChain
# from langchain.prompts import PromptTemplate
# from langchain.output_parsers import PydanticOutputParser
#
# llm = OpenAI(temperature=0.5)
# parser = PydanticOutputParser(pydantic_object=Recipe)
# prompt = PromptTemplate(
#     template="Give me a recipe for {dish}. {format_instructions}",
#     input_variables=["dish"],
#     partial_variables={"format_instructions": parser.get_format_instructions()}
# )
# chain = LLMChain(llm=llm, prompt=prompt)
# result = chain.run(dish="tacos")
# recipe = parser.parse(result)

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model
from pydantic import BaseModel, Field

class Recipe(BaseModel):
    name: str = Field(description="Name of the recipe")
    ingredients: list[str] = Field(description="List of ingredients")
    steps: list[str] = Field(description="Preparation steps")
    prep_time_minutes: int = Field(description="Preparation time in minutes")

model = init_chat_model("openai:gpt-4.1-mini", temperature=0.5)
structured_model = model.with_structured_output(Recipe)

recipe = structured_model.invoke("Give me a recipe for tacos")

print(f"Recipe: {recipe.name}")
print(f"Time: {recipe.prep_time_minutes} minutes")
print(f"Ingredients: {', '.join(recipe.ingredients)}")
for i, step in enumerate(recipe.steps, 1):
    print(f"  {i}. {step}")
# Expected output:
# Recipe: Carne asada tacos
# Time: 30 minutes
# Ingredients: corn tortillas, beef, onion, cilantro, lime, salt
#   1. Marinate the beef with salt and lime for 15 minutes
#   2. Grill the beef over high heat for 3-4 minutes per side
#   3. Finely chop the onion and cilantro
#   4. Cut the beef into small pieces
#   5. Warm the tortillas
#   6. Serve the beef in the tortillas with onion, cilantro and lime

Exercise 5: Robust system with rate limiting and tracking (Hard)

Create a robust_batch_process function that takes a list of questions and processes them with: rate limiting (2 req/s), token tracking, and error handling. It should return a summary with the responses, total tokens, and any errors it hit.

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model
from langchain_core.rate_limiters import InMemoryRateLimiter
import time

def robust_batch_process(
    questions: list[str],
    model_name: str = "openai:gpt-4.1-mini",
    requests_per_second: float = 2,
) -> dict:
    """Processes questions in batch with rate limiting, tracking and error handling."""
    rate_limiter = InMemoryRateLimiter(requests_per_second=requests_per_second)
    model = init_chat_model(model_name, rate_limiter=rate_limiter)

    results = []
    errors = []
    total_input = 0
    total_output = 0
    start = time.time()

    for i, question in enumerate(questions):
        try:
            response = model.invoke(question)
            tokens = response.usage_metadata or {}
            total_input += tokens.get("input_tokens", 0)
            total_output += tokens.get("output_tokens", 0)

            results.append({
                "question": question,
                "answer": response.content[:100],
                "tokens": tokens.get("total_tokens", 0),
            })
        except Exception as e:
            errors.append({
                "question": question,
                "error": str(e),
            })

        elapsed = time.time() - start
        status = "OK" if not errors or errors[-1]["question"] != question else "ERROR"
        print(f"[{elapsed:.1f}s] ({i+1}/{len(questions)}) {status}: {question[:40]}")

    return {
        "results": results,
        "errors": errors,
        "total_input_tokens": total_input,
        "total_output_tokens": total_output,
        "total_tokens": total_input + total_output,
        "elapsed_seconds": time.time() - start,
        "success_rate": len(results) / len(questions) * 100,
    }

questions = [
    "What is Python?",
    "What is FastAPI?",
    "What is SQLAlchemy?",
    "What is Redis?",
    "What is PostgreSQL?",
    "What is Docker Compose?",
]

summary = robust_batch_process(questions)

print(f"\n--- Summary ---")
print(f"Successful: {len(summary['results'])}/{len(questions)}")
print(f"Errors: {len(summary['errors'])}")
print(f"Total tokens: {summary['total_tokens']}")
print(f"Time: {summary['elapsed_seconds']:.1f}s")
print(f"Success rate: {summary['success_rate']:.0f}%")
# Expected output:
# [0.6s] (1/6) OK: What is Python?
# [1.2s] (2/6) OK: What is FastAPI?
# [1.7s] (3/6) OK: What is SQLAlchemy?
# [2.3s] (4/6) OK: What is Redis?
# [2.8s] (5/6) OK: What is PostgreSQL?
# [3.4s] (6/6) OK: What is Docker Compose?
#
# --- Summary ---
# Successful: 6/6
# Errors: 0
# Total tokens: 312
# Time: 3.4s
# Success rate: 100%

Summary

In this capsule you learned:

• Ollama lets you run models locally — no API keys, no costs, no internet
• init_chat_model("ollama:llama3.2") is used exactly like any cloud provider
• Prompt caching cuts costs when you repeat the same prompt prefix across multiple calls
• InMemoryRateLimiter controls the pace of your calls to avoid 429 errors
• response.usage_metadata reports the tokens consumed (input, output, total) per call
• The legacy APIs (LLMChain, AgentExecutor, OutputParser) have modern equivalents in LangChain v1.2+
• invoke() replaces chains, with_structured_output replaces parsers, create_agent replaces AgentExecutor
• In production: always configure rate limiting and token tracking

Next capsule: Project — you'll build a multi-provider chat with automatic fallback that pulls together everything you learned in this module: init_chat_model, streaming, structured output, rate limiting, and fallback to a local Ollama model.

Additional resources

1. Ollama Official Site - Installation, available models and documentation
2. LangChain + Ollama Integration - Official integration guide for Ollama
3. Anthropic Prompt Caching - Caching documentation for Anthropic
4. OpenAI Rate Limits - Limits per tier and strategies for handling them
5. InMemoryRateLimiter API Reference - Reference for the rate limiter class
6. LangChain Migration Guide - Official guide for migrating chains to LCEL
7. OpenAI Pricing - Current per-model prices for calculating costs
8. Token Usage Tracking - How-to for tracking tokens in LangChain

Module 1 — LangChain & LangGraph: From Chains to Agents