Module 12: LangSmith and Production

Introduction: From Prototype to Production

Next →

Overview

Your AI Research Assistant is a multi-agent system with automatic planning, a virtual filesystem, persistent memory, human supervision, and Deep Agent capabilities. You built it across 11 modules. It is, without exaggeration, a production system.

But you can't deploy it yet.

Not because it's incomplete. Because you can't answer four questions that any engineering team would ask you before putting an agent in production:

  1. What is my agent doing? — In development, you watch the terminal. In production, your agent runs 24/7. Nobody is watching the terminal. If the agent makes a wrong decision at 3am, how do you find out?

  2. How much is it costing? — A research run that costs $0.12 is viable. One that costs $4.80 destroys your business model. Do you know what each execution costs? Which agent burns the most tokens? Which operation is the most expensive?

  3. Where does it fail? — Your agent handles 200 queries a day. 195 go fine. 5 produce bad answers. Which ones? Why? At exactly which step did it go off the rails?

  4. Are its answers any good? — "Seems to work" is not evaluation. Are the answers relevant? Complete? Accurate? How do you measure that systematically instead of anecdotally?

This module gives you the tools to answer all four. LangSmith for tracing and visual debugging, automated evaluation with datasets and evaluators, and the production practices that turn a prototype into a system you can trust.

You made it. Now let's polish what you built so it's ready for the real world.

Where are we in the guide?

This is Module 12 of the guide LangChain & LangGraph: From Chains to Agents. It's the last module — the close of Block 4 (Production) and of the whole guide.

Block 1: LangChain Core (Modules 1-4)             ✅ Done
Block 2: LangGraph Fundamentals (Modules 5-7)     ✅ Done
Block 3: Advanced LangGraph (Modules 8-10)        ✅ Done
Block 4: Production (Modules 11-12)               ← YOU ARE HERE (Module 12 — FINAL)

Your progress:

Block 1 — LangChain Core                    ✅ Done
    │
    │  Module 1: Models and Providers        ✅
    │  Module 2: Tools and Tool Calling      ✅
    │  Module 3: Agents (create_agent)       ✅
    │  Module 4: Middleware and Customization ✅
    │
    ▼
Block 2 — LangGraph Fundamentals            ✅ Done
    │
    │  Module 5: Introduction to LangGraph   ✅
    │  Module 6: Functional API              ✅
    │  Module 7: Advanced Flows              ✅
    │
    ▼
Block 3 — Advanced LangGraph                ✅ Done
    │
    │  Module 8: Memory and Persistence      ✅
    │  Module 9: Human-in-the-Loop           ✅
    │  Module 10: Multi-Agent Systems        ✅
    │
    ▼
Block 4 — Production                        ← YOU ARE HERE
    │
    │  Module 11: Deep Agents                ✅ Done
    │  Module 12: LangSmith and Production   ← FINAL MODULE

Look at that progress. 11 modules done. You went from invoking a model with one line of code to building a multi-agent system with automatic planning, persistent memory, and human supervision. This module is the last step: giving everything you built observability and measurable quality.

The bridge from Module 11

What you already have

Your Research Agent v6 is a complete Deep Agent:

  • ✅ Three specialized agents: researcher, analyst, writer
  • ✅ A supervisor that coordinates the flow between agents
  • ✅ Automatic planning with write_todos
  • ✅ A virtual filesystem for intermediate files
  • ✅ Persistent, multi-user memory with thread_id
  • ✅ Approval gates before expensive actions
  • ✅ Retry with exponential backoff and graceful degradation
  • ✅ Parallel search across multiple sources

The production gap

Your agent is technically complete. But in production, the situation changes radically:

In development:                         In production:
  You watch the terminal                 Nobody watches anything
  One user (you)                         Hundreds of users
  If it fails, you see it right away     If it fails, nobody notices
  Cost doesn't matter                    Cost is operating expense #1
  "Works on my machine"                  "Does it work at 3am on a Sunday?"
  You evaluate by reading the output     You evaluate with automated metrics

Most AI projects don't fail because the agent is bad. They fail because there's no observability. The agent produces a wrong answer, nobody catches it, users lose trust, and the project gets cancelled. Not for lack of technical capability — for lack of monitoring.

LangSmith: the observability platform

LangSmith is the observability platform for applications built with LangChain and LangGraph. Three main capabilities:

Tracing

Every operation your agent performs gets recorded: model calls, tool executions, state changes, routing decisions. All of it on a visual timeline you can explore step by step.

Without tracing:
  Input: "Research AI in finance"
  Output: "Here's your report..."
  What happened in between? 🤷

With tracing:
  Input: "Research AI in finance"
  ├─ Supervisor → delegates to Researcher
  │   ├─ web_search("AI fintech trends 2025") → 3 results, 1.2s, 450 tokens
  │   ├─ arxiv_search("AI financial systems") → 2 papers, 0.8s, 320 tokens
  │   └─ Return → researcher done
  ├─ Supervisor → delegates to Analyst
  │   ├─ gpt-4.1 → analysis of 5 sources, 2.1s, 1200 tokens
  │   └─ Return → analyst done
  ├─ Supervisor → delegates to Writer
  │   ├─ gpt-4.1-mini → executive report, 1.5s, 800 tokens
  │   └─ Return → writer done
  └─ Output: "Here's your report..."
     Total: 5.6s, 2770 tokens, $0.008

Evaluation

Measuring answer quality systematically. Not "does it look good?" but "does it meet specific criteria for relevance, completeness, accuracy, and format?"

Monitoring

Dashboards with production metrics: latency per operation, cost per execution, error rate, distribution of quality scores.

The full journey: from Module 1 to Module 12

Before we get into the technical content, look at what you built:

Module 1:  model.invoke("What is AI?")
              One line. One model. One answer.

Module 2:  Model + tools. The agent can DO things.

Module 3:  create_agent: a full ReAct loop.

Module 4:  Middleware: customizing and controlling the pipeline.

Module 5:  LangGraph: state graphs, nodes, edges.

Module 6:  Functional API: @entrypoint + @task.

Module 7:  Retry, branching, error handling, parallelism.

Module 8:  Memory: checkpointing, crash recovery, long-term memory.

Module 9:  Human-in-the-loop: approvals, review, feedback.

Module 10: Multi-agent: supervisor, handoffs, 3 specialized agents.

Module 11: Deep Agents: automatic planning, virtual filesystem.

Module 12: Observability, evaluation, production.
              ↑ YOU ARE HERE

From one line of code to a production multi-agent system with full observability. That's an AI Engineer.

What changes in production

Observability = confidence

Without tracing, deploying an agent is an act of faith. "I think it works. I hope it doesn't break." With tracing, you know exactly what your agent did, what it cost, and where it failed. Observability is what lets you deploy and still sleep at night.

Evaluation as a continuous practice

It's not something you do once at the end. It's something you run with every prompt change, every model upgrade, every new feature. If you change the analyst's system prompt, did it get better or worse? Without automated evaluation, you're guessing.

Cost as a design constraint

Tokens cost money. In development it doesn't matter — you're iterating. In production, with 200 queries a day, the difference between using gpt-4.1 and gpt-4.1-mini for the researcher can be $50/day vs $5/day. Token tracking isn't a nice-to-have — it's a business tool.

Module map

#CapsuleWhat you'll learnType
01Introduction (this one)From prototype to production, LangSmith, the full journeyIntro
02Tracing and observabilityLangSmith setup, reading traces, spotting bottlenecks, custom metadataTechnical
03Visual agent debuggingDebugging workflow with the dashboard, comparing traces, errorsTechnical
04Evaluation: datasets and evaluatorsBuilding datasets, specific evaluators, LLM-as-judge, calibrationTechnical
05Token tracking and cost controlUsageMetadataCallbackHandler, cost per operation, rate limitingTechnical
06Production checklistConfiguration management, error monitoring, deployment patternsTechnical
07Monitoring and dashboardsProduction metrics, alerts, quality trendsTechnical
08Project: production Research AssistantResearch Agent v7: tracing + evaluation + cost control + production-readyProject

Learning flow

You start with tracing (capsule 02) — the foundation of all observability: seeing what your agent does, step by step. Then you use those traces for visual debugging (capsule 03) — finding and fixing problems without guessing. With observability in place, you implement evaluation (capsule 04) — measuring quality systematically with datasets and evaluators. You add token tracking (capsule 05) — cost visibility and rate limiting for budget control. Capsule 06 covers the production checklist — everything you need to verify before deploying. Capsule 07 adds monitoring — dashboards and alerts for continuous production. Finally, you put it all together in Research Agent v7 (capsule 08).

The progression is: tracing → debugging → evaluation → cost → checklist → monitoring → project.

Connection with the project

Research Agent v7: production-ready

Your Research Agent gets its final layer:

v1 (Module 6):  Functional but fragile
    ↓
v2 (Module 7):  Robust (retry, branching, error handling)
    ↓
v3 (Module 8):  Persistent (checkpointing, memory, multi-user)
    ↓
v4 (Module 9):  Supervised (approval gates, review, feedback)
    ↓
v5 (Module 10): Multi-agent (researcher + analyst + writer + supervisor)
    ↓
v6 (Module 11): Deep Agent (automatic planning, virtual filesystem)
    ↓
v7 (This module): Production-ready
    │
    │  📊 Tracing: every step visible in LangSmith
    │  🔍 Debugging: find failures with one click
    │  📈 Evaluation: quality measured against specific criteria
    │  💰 Cost control: token tracking and rate limiting
    │  ✅ Production checklist: ready to deploy
    │
    ▼
    🚀 DEPLOYMENT

By the end of this module, you run a full research request and you can: see every step in LangSmith (what it searched, what it analyzed, what it wrote), measure the report's quality with automated evaluators, know exactly what it cost in tokens, and verify that it meets every criterion on the production checklist. That's the difference between "works on my machine" and "ready for real users."

Before and after

To size up what you've achieved, look at how what you can do with your Research Assistant evolves:

WITHOUT this module (Module 11):
  - You run: agent.invoke({"query": "Research AI in finance"})
  - You get: a report
  - Unanswered questions:
      How much did it cost? → No idea
      Which node was the slowest? → No idea
      Are the answers good? → I think so
      Where does it fail? → No idea until a user complains

WITH this module (Module 12):
  - You run: agent.invoke({"query": "Research AI in finance"})
  - You get: a report
  - Answered questions:
      How much did it cost? → $0.12 (breakdown: researcher $0.04, analyst $0.05, writer $0.03)
      Which node was the slowest? → search_web: 2.1s (65% of the total)
      Are the answers good? → Relevance: 0.92, Completeness: 0.85, Accuracy: 0.88
      Where does it fail? → 3 of 200 queries had relevance < 0.5, all about quantum computing

That's the difference between a prototype and a production system.

What this module does NOT cover

  • ❌ LangGraph Platform / Cloud — Serverless deployment with LangGraph Platform is an advanced topic beyond the scope of this guide. We mention it as an option
  • ❌ Infrastructure as Code — Terraform, Kubernetes, CI/CD pipelines. These are DevOps topics that complement this module but aren't AI-specific
  • ❌ A/B testing prompts — Experiments in LangSmith are powerful but go beyond the basic evaluation we cover
  • ❌ Custom dashboards — We use LangSmith's built-in dashboards. Building custom dashboards with Grafana/Datadog is a next step
  • ❌ Security and compliance — PII detection, advanced audit logging, and regulatory compliance are topics that deserve their own module

Technical setup

Prerequisites

  • ✅ Module 11 completed — you have a Research Agent v6 with Deep Agent capabilities
  • ✅ Python 3.11+ installed
  • ✅ At least one API key from a provider (OpenAI recommended)
  • ✅ A LangSmith account — create one at smith.langchain.com

Installation

pip install langsmith langchain-openai langgraph python-dotenv

Check the imports:

import langsmith
from langsmith import Client

print(f"langsmith version: {langsmith.__version__}")
print(f"Client available: {Client is not None}")
# Expected output:
# langsmith version: 0.3.x
# Client available: True

Environment variables

Add the LangSmith variables to your .env:

# .env
OPENAI_API_KEY=sk-...

# LangSmith
LANGSMITH_TRACING=true
LANGSMITH_API_KEY=lsv2_pt_...
LANGSMITH_PROJECT=research-assistant-prod

Three variables:

  • LANGSMITH_TRACING=true — turns on automatic tracing for every LangChain/LangGraph operation
  • LANGSMITH_API_KEY — your LangSmith API key (get it under Settings → API Keys)
  • LANGSMITH_PROJECT — the name of the project where traces get stored (it's created automatically)

Quick check: your first trace

Run this script to confirm tracing works:

from dotenv import load_dotenv
load_dotenv()

from langchain.chat_models import init_chat_model

model = init_chat_model("openai:gpt-4.1-mini")

response = model.invoke("What is observability in AI? Answer in one sentence.")
print(f"Answer: {response.content}")
print(f"\n→ Open LangSmith (smith.langchain.com) and look for this trace in your project.")
print(f"  You should see: the input, the output, tokens used, and latency.")
# Expected output:
# Answer: Observability in AI is the ability to monitor, understand, and diagnose...
#
# → Open LangSmith (smith.langchain.com) and look for this trace in your project.
#   You should see: the input, the output, tokens used, and latency.

If you see the model's answer and you find the trace in LangSmith, you're set. Every LangChain/LangGraph call is traced automatically — without changing a single line of your existing code.

Signs it worked

By the end of this module, you'll know you succeeded if:

  • ✅ You configure tracing with LangSmith and read traces in the dashboard: you identify the slowest node, see the exact prompt that was sent, spot redundant tool calls
  • ✅ You debug visually: you find the exact step where the agent went off track, without adding print statements or re-running anything
  • ✅ You build an evaluation dataset with reference questions and specific criteria (relevance, completeness, accuracy, formatting)
  • ✅ You implement built-in and custom evaluators, including a calibrated LLM-as-judge
  • ✅ You track token usage per operation and per agent, connecting cost to business decisions
  • ✅ Your Research Agent v7 has tracing, evaluation, cost control, and passes the production checklist

Self-assessment test

If you can answer these questions, you're on the right track:

  1. What's the difference between watching the terminal in development and tracing in production?
  2. If your agent produces a bad answer, how do you find the exact step where it went off track using LangSmith?
  3. Why is "is the answer good?" not a valid evaluator? What specific criteria would you use?
  4. If an LLM-as-judge says every answer is "excellent," what do you do?
  5. A research run costs $0.45. How do you decide whether that's acceptable?

Summary

  • The production gap isn't technical — it's about observability. Your agent works, but without tracing you don't know what it does, without evaluation you don't know if it does it well, and without cost tracking you don't know what it costs. This module closes all three gaps
  • LangSmith is the observability platform for LangChain/LangGraph: tracing (see every step), evaluation (measure quality), monitoring (production metrics). It turns on with a single environment variable: LANGSMITH_TRACING=true
  • Observability = confidence. Without it, deploying an agent is an act of faith. With it, you know exactly what it did, what it cost, and where it failed
  • Evaluation is a continuous practice, not a final step. You run it with every prompt change, every model upgrade, every new feature
  • Token cost is operating expense #1 in LLM applications. Token tracking is a business tool, not just a technical one
  • This is Module 12 of 12. You went from model.invoke("hello") to a production multi-agent system with automatic planning, persistent memory, human supervision, and now full observability. That's an AI Engineer

Additional resources

  1. LangSmith Documentation — Complete official LangSmith docs: setup, tracing, evaluation, monitoring
  2. LangSmith — Getting Started — Quickstart guide for setting up your first project
  3. LangSmith Tracing Concepts — Tracing concepts: runs, traces, projects, metadata
  4. LangSmith Evaluation — The evaluation framework: datasets, evaluators, experiments
  5. LangChain — Observability — How to set up observability in LangChain applications
  6. The Production Gap in AI — LangChain Blog — Why observability is the missing piece in AI applications

Module 12 — LangChain & LangGraph: From Chains to Agents

Next capsule: Tracing and Observability — you'll learn how LangSmith automatically records every operation your agent performs, how to read traces to spot bottlenecks and problems, and how to add custom metadata to filter and organize traces in production.

← PreviousEvolving Project: The Deep Agent Version (v6)Next →Tracing and Observability