1. Introduction: Agents Without Tests Are Dangerous Agents

Overview

Module 8 solved a capability problem: your agent stopped working alone and started coordinating a team. With the Supervisor pattern, a central agent decomposes tasks and assigns specialized workers. With Handoffs, agents transfer control peer-to-peer. With Subagents, you delegate sub-tasks with isolated context. With Router, you classify inputs and direct them to the right agent. The Research Agent v5 has 4 agents — Supervisor, Researcher, Analyst, Writer — each with its own StateGraph, its own tools, and its own optimized context. It's a system of coordinated agents that produces research reports of higher quality than a single agent could. But there's a problem nobody has taught you to solve: how do you know it works? Not that it "seems to work" — that it works. That the Supervisor assigns correctly. That the Researcher finds the relevant sources. That the Analyst doesn't invent findings. That the Writer doesn't lose data from the analysis. That when you change the Supervisor's system prompt, you don't break routing to the workers. That when you update the papers MCP server, the Researcher still finds sources. Today, your only way to know is to run the system, read the output, and "feel" whether it's right. That's not engineering. That's faith.

This module closes what may be the biggest gap in the AI agents industry. Most courses, tutorials, and guides stop at "build an agent." Some get as far as "build a multi-agent system." Almost none reach "test your agent." The result is predictable: agents in production that fail silently, that produce inconsistent results between runs, and that nobody knows have regressed — because nobody measures. You change one line of the Supervisor's prompt and report quality drops 20%, but you don't find out until a user complains three weeks later. You update the model version and the Researcher's tool calls change pattern, but you don't detect it because there's no baseline. You add a new MCP server and tool selection degrades, but there's no metric that captures it. This module establishes the discipline of "don't trust agents without evidence." It's not testing as a formality — it's testing as the difference between an agent that works and one you believe works.

The central concept that sets this module apart from any testing tutorial is trajectory evaluation — evaluating not just whether the final result is correct, but whether the path was correct. An agent can arrive at the right answer by luck: it used 8 tool calls when 3 would do, picked suboptimal tools, planned inefficiently, but the LLM compensated with brute reasoning at the end. The result is "fine," but the cost was 3x, the latency was 5x, and next time with a slightly different input it's going to fail because the path was never robust. Trajectory evaluation captures this: did it use the right tools? In the right order? With the right arguments? Was the plan sensible? Did reflection detect the real gaps? This goes far beyond traditional software testing, where the input determines the output. In agents, the same input can produce different paths — and you need to evaluate all of them.

---

Where Are We in the Guide?

Context

This guide has 10 modules organized into 3 phases:

Phase 1: Agent Foundations (Modules 1-3)                ✓ COMPLETED
├── Module 01: Anatomy of an AI Agent                   ✓ COMPLETED
├── Module 02: Tool Use Fundamentals                    ✓ COMPLETED
└── Module 03: Function Calling Patterns                ✓ COMPLETED

Phase 2: Agent Architecture (Modules 4-6)               ✓ COMPLETED
├── Module 04: State Machines for Agents                ✓ COMPLETED
├── Module 05: Multi-Step Reasoning and Planning        ✓ COMPLETED
└── Module 06: Memory Systems for Agents                ✓ COMPLETED

Phase 3: Advanced Integration & Production (Modules 7-10) ← YOU ARE HERE
├── Module 07: MCP and Advanced Tool Integration          ✓ COMPLETED
├── Module 08: Multi-Agent Orchestration                  ✓ COMPLETED
├── Module 09: Testing and Evaluating Agents              ← THIS MODULE
└── Module 10: Agents in Production and Alternatives

Third module of Phase 3 — the formal validation of everything you've built. Phases 1 and 2 built an agent with all its internal capabilities: tool use (M2-M3), flow control (M4), deliberative intelligence (M5), and persistent memory (M6). Phase 3 takes that agent into the real world: external tools at scale (M7), multi-agent coordination (M8), formal testing (M9), and production deployment (M10).

The progression of Phase 3 is deliberate — each module opens a door the previous one makes possible:

1. Module 7 — How it connects to the world → MCP servers, clients, dynamic tools, ecosystem ✓
2. Module 8 — How it works as a team → Supervisor, handoffs, subagents, shared state ✓
3. Module 9 — How you know it works → Unit tests, trajectory evaluation, golden datasets, regression ← HERE
4. Module 10 — How you put it in production → Deployment, observability, costs, alternatives

Where are you coming from?

Module 8 left you with a Research Agent v5 that coordinates 4 specialized agents:

• Supervisor as central coordinator: Decomposes tasks, assigns workers, monitors progress, re-assigns if a worker fails, and aggregates partial results into a coherent output
• 4 coordination patterns: Supervisor (centralized control), Handoffs (peer-to-peer transfer), Subagents (isolated delegation), Router (classification + direction) — each with clear trade-offs
• State designed for coordination: SupervisorState, ResearcherState, AnalystState, WriterState — each agent with its own typed state, no shared mega-state
• Pattern composition: Router → Supervisor → Subagents combined, parallel execution, failure handling

That's a system of individually specialized, coordinated agents. But when you look at how you "validate" it, you see this:

result = await system.ainvoke({
    "task": "Research quantum computing applications in drug discovery"
})

print(result["final_report"])
# "Hmm... looks good. I think. I'm going to deploy."

You run it. You read it. "Looks right." You deploy. And you pray.

Where are you going?

The transition from M8 to M9 is the jump from "a system you built" to "a system you can trust." M8 solved coordination between agents — but with 4 coordinated agents, a bug can live in any individual agent, in the coordination between them, in the shared state, or in the result aggregation. Manual debugging is impossible when there are 14+ LLM calls per run, each one non-deterministic.

After M9, Module 10 takes the tested system and deploys it to production. The transition is direct: "Your system is tested and validated → now ship it to the real world." Without M9, M10 would be deploying hopes. With M9, it's deploying a system with measurable evidence that it works — task completion rate, tool call accuracy, regression baselines, and quality metrics you can monitor in production.

---

The Problem: Non-determinism

Same input, different output

Traditional software testing works because functions are deterministic:

def calculate_tax(income: float, rate: float) -> float:
    return income * rate

assert calculate_tax(100_000, 0.30) == 30_000  # Always. 100% of the time.

Same input → same output. Always. You can write exact assertions. You can do regression testing by comparing output byte by byte. You can trust that if the test passes today, it passes tomorrow.

Agents break this fundamental premise:

agent_result_1 = await research_agent.ainvoke({
    "task": "Research quantum computing applications"
})
# Tool calls: [web_search("quantum computing applications 2025")]
# Plan: 3 sub-tasks, focus on the pharmaceutical industry
# Result: 2,500 words with 8 sources

agent_result_2 = await research_agent.ainvoke({
    "task": "Research quantum computing applications"
})
# Tool calls: [web_search("quantum computing real world uses"),
#              paper_search("quantum computing applications")]
# Plan: 4 sub-tasks, focus on cryptography and finance
# Result: 3,200 words with 12 sources

Exact same input. Different tool calls, different plan, different focus, different result. Both may be "correct" — but they're fundamentally different. How do you write an assert for this?

The three dimensions of non-determinism in agents

1. Model non-determinism.

The LLM has temperature > 0 by design. Each call produces different tokens. Even with temperature=0, differences in batching, quantization, and the model's internal state produce variations. You can't predict exactly what the model is going to say.

2. Plan non-determinism.

The agent decides how to approach the task at runtime. The planning node from M5 decomposes the query into sub-tasks, but the decomposition can be different every time. "Research quantum computing" can decompose into [applications, challenges, future] or into [theory, current_state, industry_impact]. Both decompositions are valid.

3. Tool non-determinism.

External tool results change. web_search("quantum computing") returns different results today than tomorrow — because the web changes. paper_search can return different papers depending on the state of the database. Tools are variable inputs the agent doesn't control.

Why traditional testing isn't enough

Traditional testing:
    Fixed input → Deterministic function → Fixed output
    assert output == expected    ✓

Agent testing:
    Fixed input → Non-deterministic agent → Variable output
    assert output == expected    ✗ (which expected?)

You can't compare outputs byte by byte. You can't predict the exact tool calls. You can't anticipate the plan. But you can evaluate:

• Does the result answer the user's question? (quality)
• Did it use appropriate tools for the task? (tool selection)
• Was the path efficient? (trajectory)
• Was the cost reasonable? (efficiency)
• Does it behave consistently across many runs? (reliability)

This requires a fundamentally different testing approach — and that's what this module teaches.

---

Trajectory Evaluation: The Central Concept

The right result isn't enough

Imagine two runs of the same agent for the same query:

Run A:
  Plan: [search sources → analyze → synthesize]
  Tools: web_search("quantum computing drug discovery") → 3 relevant results
  Analysis: Identifies 2 key applications with evidence
  Result: Accurate 1,500-word report ✓
  Cost: $0.04 | Latency: 6s | Tool calls: 3

Run B:
  Plan: [search general → search specific → re-search → analyze → re-analyze → synthesize]
  Tools: web_search("quantum computing") → generic results
         web_search("quantum applications") → semi-relevant results
         web_search("quantum drug") → no results
         web_search("quantum computing pharmaceutical") → 2 results
         paper_search("quantum drug discovery") → 1 result
  Analysis: Identifies the same 2 applications but with more noise
  Result: Accurate 1,500-word report ✓
  Cost: $0.12 | Latency: 18s | Tool calls: 7

Both runs produced a "correct" result. If you only evaluate the final output, both pass the test. But Run B cost 3x more, took 3x longer, and used tools inefficiently — it searched 5 times when 1-2 well-formed searches would have done.

What trajectory evaluation is

Trajectory evaluation evaluates the agent's entire path, not just the destination:

Trajectory = [
    (step_1, "plan", {sub_tasks: [...], reasoning: "..."}),
    (step_2, "tool_call", {tool: "web_search", args: {...}, result: {...}}),
    (step_3, "tool_call", {tool: "paper_search", args: {...}, result: {...}}),
    (step_4, "analysis", {findings: [...], confidence: 0.85}),
    (step_5, "reflection", {quality: "sufficient", gaps: []}),
    (step_6, "synthesis", {report: "...", sources: 5}),
]

Every step captured. Every decision recorded. Every tool call with its arguments and results. And then you evaluate:

Dimension	Question	How it's measured
Tool selection	Did it use the right tools?	Compare tools used vs tools expected for that query type
Tool arguments	Were the arguments sensible?	Evaluate whether the search terms capture the query's intent
Sequence	Was the order logical?	Plan → search → analyze → synthesize, not search → search → search → search
Efficiency	How many steps did it need?	Compare against a baseline: 3 tool calls or 8?
Reasoning quality	Was the plan sensible?	LLM-as-judge evaluates whether the task decomposition was reasonable
Self-correction	Did reflection detect real gaps?	Verify that the gaps identified are legitimate, not fabricated

Why trajectory evaluation matters for multi-agent

In a multi-agent system (M8), the trajectory has multiple simultaneous dimensions:

Supervisor trajectory:
  step_1: Decompose "quantum computing research" → 3 sub-tasks
  step_2: Assign sub-task 1 to the Researcher
  step_3: Receive the Researcher's result → validate
  step_4: Assign data to the Analyst
  step_5: Receive analysis → assign to the Writer
  step_6: Receive report → validate final quality

Researcher trajectory:
  step_1: Receive sub-task from the Supervisor
  step_2: web_search("quantum computing drug discovery applications")
  step_3: paper_search("quantum pharmaceutical computing 2024")
  step_4: Evaluate relevance of 8 sources → return 5 relevant ones

Analyst trajectory:
  step_1: Receive 5 sources from the Researcher
  step_2: Identify patterns → 3 key applications
  step_3: Compare evidence → confidence ranking

Writer trajectory:
  step_1: Receive analysis with 3 ranked applications
  step_2: Draft report with structure: intro, applications, conclusion
  step_3: Cite 5 sources correctly

A bug can live in the Supervisor's trajectory (assigned badly), the Researcher's (searched badly), the Analyst's (analyzed badly), or the Writer's (lost data). Without per-agent trajectory evaluation, you never know where the problem is.

This concept is the thread running through the entire module. Every capsule reinforces it: unit tests verify isolated components, integration tests verify the full trajectory, LangSmith visualizes it, golden datasets automate it, regression testing monitors it over time.

---

Types of Testing for Agents

The full spectrum

Agent testing isn't one type of test — it's a spectrum that runs from the most granular to the most holistic:

Granularity           Speed            Cost           Confidence
     ↑                   ↑                ↓                ↓
┌─────────────────┐
│   UNIT TESTS    │  Fast, cheap, many.
│   (Tools,       │  Test isolated components.
│    State,       │  Mock the LLM.
│    Edges)       │  Don't test the agent as a system.
├─────────────────┤
│  INTEGRATION    │  Moderate in time and cost.
│   TESTS         │  Test the full agent loop.
│   (Agent loop,  │  Real LLM, real tools.
│    End-to-end)  │  Capture the trajectory.
├─────────────────┤
│  TRAJECTORY     │  Slow, expensive, few.
│  EVALUATION     │  Evaluate the agent's PATH.
│   (Path,        │  LLM-as-judge for quality.
│    Efficiency)  │  Require golden datasets.
├─────────────────┤
│  REGRESSION     │  Periodic (CI/CD).
│   TESTING       │  Detect behavior changes.
│   (Baselines,   │  Compare against a baseline.
│    Golden sets) │  Alert on degradation.
├─────────────────┤
│  BENCHMARKS     │  Monthly or per release.
│  & METRICS      │  Measure aggregate performance.
│   (Task rate,   │  Task completion, cost, latency.
│    Accuracy)    │  Product decisions.
└─────────────────┘
     ↓                   ↓                ↑                ↑
Granularity           Speed            Cost           Confidence

1. Unit tests

They test individual components of the agent without invoking the LLM:

• Individual tools: Does web_search return data in the right format? Does it handle network errors? Does it validate inputs?
• State transitions: Does the quality gate's conditional edge route to "re-plan" when quality_score < 0.7?
• Prompt formatting: Does the Supervisor's system prompt include the right instructions?

Fast, deterministic, cheap. They run on every commit. But they don't test the agent's intelligence — they test the plumbing.

2. Integration tests

They test the complete agent with a real LLM against specific queries:

• Full agent loop: From input to final output, passing through planning, research, analysis, reflection
• Snapshot testing: Capture the trajectory as a snapshot and compare it against previous runs
• Multi-agent coordination: Does the Supervisor assign correctly? Do the handoffs preserve context?

Slower, non-deterministic, more expensive. But they test the system as it actually works.

3. Trajectory evaluation

They evaluate the quality of the path, not just the result:

• Tool call accuracy: Did it use the right tools for the query type?
• Sequence quality: Was the plan logical? Are the steps in a sensible order?
• Efficiency: How many steps did it need vs how many it should need?
• LLM-as-judge: A second LLM evaluates whether the agent's reasoning and decisions were sound

4. Regression testing

They detect when a change breaks existing behavior:

• Golden datasets: 20+ queries with documented expected behavior
• Baseline comparison: Each run is compared against the previous baseline
• CI/CD integration: Automatic alerts when a metric drops below a threshold

5. Benchmarks and metrics

They measure aggregate performance for product decisions:

• Task completion rate: % of queries the agent resolves satisfactorily
• Tool call accuracy: % of tool calls that are appropriate for the task
• Latency: Average response time
• Cost per task: Average cost in tokens/money per run
• Reasoning quality: Reasoning quality score (LLM-as-judge)

This module covers all five types, from unit tests to benchmarks, with practical implementation in every capsule.

---

Prerequisites

From Module 8 (Multi-Agent Orchestration)

This module tests M8's multi-agent system. You need these solid:

• Supervisor pattern: You know how to implement a coordinating agent that decomposes tasks, assigns workers, and aggregates results. M9's tests verify that the supervisor routes correctly
• Coordination patterns: Handoffs, Subagents, Router — you understand the trade-offs and know when to apply each one. The integration tests cover coordination scenarios
• Shared vs Isolated State: You understand what gets shared and what gets isolated between agents. The tests verify that state flows correctly between workers
• Research Agent v5: You have a system of 4 coordinated agents (Supervisor, Researcher, Analyst, Writer) working end-to-end

From Module 7 (MCP and Advanced Tool Integration)

MCP tools need specific testing:

• MCP servers and clients: You know how the agent discovers tools dynamically. The unit tests verify that discovery works correctly
• langchain-mcp-adapters: You integrate MCP tools into your StateGraph. The tests mock MCP servers for isolated testing
• Multi-server setup: Your agent connects to multiple MCP servers. The integration tests verify that the connection is stable

From Module 5 (Multi-Step Reasoning and Planning)

The deliberative intelligence layer is what gets tested most:

• Intelligent planning: Decomposing tasks into sub-tasks with dependencies — trajectory evaluation assesses whether the decomposition was sensible
• Reflection with quality gates: Evaluating the output with a critique — the tests verify that the quality gate works (routes to re-plan when it should)
• Reasoning traces: The thinking process captured as data — the foundation for trajectory evaluation

From Module 4 (State Machines for Agents)

• StateGraph for agents: Functional nodes, explicit edges, controlled cycles — the unit tests verify each edge individually
• Conditional routing: Deterministic decision points — testable by nature
• Extensible typed state: Every state field is verifiable with assertions

Tools for this module

• Python 3.11+
• langchain v1.2+ and langchain-openai
• langgraph v1.0+
• OpenAI API key (GPT-4.1 or GPT-4.1-mini)
• pytest and pytest-asyncio for test execution
• langsmith SDK for tracing and evaluation
• python-dotenv for environment variables

pip install langchain langchain-openai langgraph pytest pytest-asyncio langsmith python-dotenv

New dependencies: pytest and pytest-asyncio to run tests (if you didn't already have them). The langsmith SDK for tracing, evaluation datasets, and evaluators — it's LangChain's observability tool, and it becomes your daily driver for monitoring agents.

---

Objectives of Module 9

By the end of this module you'll be able to:

• ✅ Write unit tests for tools and state transitions: Test each individual tool with mocked inputs, verify outputs, cover edge cases and error handling. Test the StateGraph's conditional edges to verify that routing works correctly
• ✅ Implement integration tests with a real LLM: Run the full agent loop against test queries, capture the trajectory as a snapshot, and verify that the system produces coherent end-to-end results
• ✅ Implement trajectory evaluation: Evaluate the agent's path — tool selection accuracy, sequence quality, efficiency, and reasoning quality — using automated evaluators and LLM-as-judge
• ✅ Use LangSmith as a daily working tool: Configure tracing of full runs, create evaluation datasets, run automated evaluators, interpret results, and use that information to improve your agent
• ✅ Create golden datasets for regression testing: Build a dataset of 20+ queries with expected tool calls, expected reasoning steps, and expected final answers. Automate regression testing to detect degradations
• ✅ Implement regression testing with CI/CD integration: Detect when a change to a prompt, tool, or model breaks existing behavior. Compare against baselines and alert automatically on degradation
• ✅ Define actionable benchmarks and metrics: Measure task completion rate, tool call accuracy, reasoning quality, latency, and cost per task. Communicate performance with numbers, not with "it seems to work"
• ✅ Build the Research Agent v6 with a complete testing suite: Expand the Research Agent with unit tests for every tool and edge, integration tests with trajectories, a golden dataset of 20+ queries, and automated regression testing

---

Module Map

#	Capsule	What you'll learn
02	Unit Testing for Agents	Write unit tests for individual tools: mock inputs, verify outputs, test edge cases, test error handling. Unit tests for state transitions: verify that conditional edges route correctly, that stop conditions work, that state updates as expected. pytest fixtures for agents
03	Integration Testing	Test the full agent loop with a real LLM. Snapshot testing of trajectories: capture the complete sequence of tool calls, decisions, and state changes as a reference. End-to-end tests of the multi-agent system: does the Supervisor coordinate? Do the workers produce? Is the final result coherent?
04	Trajectory Evaluation	The differentiating concept: evaluate the path, not just the destination. Tool call accuracy, sequence quality, efficiency metrics. LLM-as-judge to evaluate reasoning quality. Trajectory comparison between runs. Automated evaluators that scale
05	LangSmith for Agents	LangSmith as a daily driver, not as a demo. Tracing full runs with visualization of every step. Evaluation datasets: create, populate, and run evaluations. Custom evaluators for research quality. Visual debugging of multi-agent trajectories
06	Golden Datasets and Regression Testing	Create golden datasets of 20+ queries: expected tool calls, expected reasoning, expected answers. Regression testing: detect when a change breaks behavior. CI/CD integration: a pipeline that runs tests on every PR. Automatic alerts on degradation
07	Benchmarks and Agent Metrics	Define actionable metrics: task completion rate, tool call accuracy, reasoning quality, latency, cost per task. Performance dashboards. Comparison between agent versions. Decisions based on data, not intuition
08	Project: Testing and Evaluation	Research Agent v6: complete testing suite. Unit tests for tools and edges. Integration tests with trajectory capture. Golden dataset of 20+ queries. Automated regression testing. LangSmith configured. Baseline benchmarks. The agent goes from "it seems to work" to "I have evidence that it works"

Learning flow

The module follows a progression from isolated components → complete system → intelligent evaluation → automation → production metrics.

You start with Unit Testing (capsule 02) because it's the most familiar foundation. If you come from software development, you already know how to write unit tests. What's new is what you test: individual tools, StateGraph conditional edges, and state transitions. Pytest fixtures designed for agents. By the end you'll have 15+ unit tests that run in seconds without an LLM.

Then Integration Testing (capsule 03) raises the level: you run the complete agent with a real LLM. The key is snapshot testing of trajectories — you capture the full sequence of what the agent did (tool calls, decisions, state changes) and use it as a reference. If the next run produces a radically different trajectory, you detect it. It's your first contact with real non-determinism in testing.

Trajectory Evaluation (capsule 04) is the heart of the module. Here you meet the concept that sets this module apart from any tutorial: evaluating the path, not just the result. Tool call accuracy, sequence quality, efficiency, reasoning quality. LLM-as-judge as an evaluator. After this capsule, you'll never again accept "the output looks good" as validation.

Capsule 05 (LangSmith for Agents) connects everything to a real tool. It's not a 5-minute demo — it's configuring LangSmith as your daily dashboard: tracing every run, evaluation datasets you run periodically, custom evaluators that measure what matters to you. LangSmith becomes the "eyes" that see what your agent does.

With Golden Datasets and Regression Testing (capsule 06), you automate. You create a dataset of 20+ queries with expected behavior. You run it automatically. You integrate it into CI/CD. You detect degradations before they reach production. It's the difference between "I saw it once and it looked fine" and "I have a pipeline that verifies it on every change."

Benchmarks and Metrics (capsule 07) closes with the language of production: numbers. Task completion rate 85%. Tool call accuracy 92%. Average latency 4.2s. Average cost $0.08/task. After this capsule you can communicate your agent's performance with data, not with feelings.

Finally, the project (capsule 08) integrates everything into the Research Agent v6: the same multi-agent system from M8, but now with a complete testing suite that lets you trust it.

---

Connection with the Evolving Project

Research Agent v5 (M8) → Research Agent v6 (M9)

The Research Agent from M8 has the entire architecture solved — a system of 4 coordinated agents:

                              ┌─────────────────────┐
                              │     SUPERVISOR      │
                              │  - Decomposes task   │
                              │  - Assigns workers   │
                              │  - Validates result  │
                              └──────────┬──────────┘
                                         │
                    ┌────────────────────┼─────────────────────┐
                    │                    │                     │
                    ▼                    ▼                     ▼
          ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐
          │   RESEARCHER    │  │    ANALYST       │  │     WRITER      │
          │ MCP: web, papers│  │ Tools: analyze   │  │ MCP: filesystem │
          └─────────────────┘  └─────────────────┘  └─────────────────┘

  Without testing:
  - Does the supervisor route well? I don't know, seems like it.
  - Does the researcher find relevant sources? I think so, the report looks good.
  - Does the analyst invent findings? I don't think so... but I have no way to verify.
  - Does changing the supervisor's prompt break something? Only one way to find out: try it and see.

M9 doesn't change the architecture — it adds a verification layer on top of it:

                              ┌─────────────────────┐
                              │     SUPERVISOR      │  ← Unit tests: routing edges
                              │  - Decomposes task   │  ← Integration: assignment quality
                              │  - Assigns workers   │  ← Trajectory: decision evaluation
                              │  - Validates result  │
                              └──────────┬──────────┘
                                         │
                    ┌────────────────────┼─────────────────────┐
                    │                    │                     │
                    ▼                    ▼                     ▼
          ┌─────────────────┐  ┌─────────────────┐  ┌─────────────────┐
          │   RESEARCHER    │  │    ANALYST       │  │     WRITER      │
          │  ← Tool tests   │  │  ← Tool tests   │  │  ← Tool tests  │
          │  ← Search eval  │  │  ← Analysis eval │  │  ← Output eval │
          └─────────────────┘  └─────────────────┘  └─────────────────┘

  ┌──────────────────────────────────────────────────────────────────────┐
  │  TESTING LAYER                                                      │
  │                                                                     │
  │  Unit tests (30+):     Every tool, every edge, every transition     │
  │  Integration tests (5): Full agent loop, trajectory snapshots       │
  │  Golden dataset (20+):  Queries with expected behavior              │
  │  LangSmith:            Tracing + evaluation + debugging             │
  │  Benchmarks:           Task rate, accuracy, latency, cost           │
  │  CI/CD:                Automatic regression testing on every PR     │
  └──────────────────────────────────────────────────────────────────────┘

What changes concretely

1. Every tool has unit tests

class TestWebSearchTool:
    def test_valid_query_returns_results(self):
        result = web_search.invoke({"query": "quantum computing"})
        assert "results" in result
        assert len(result["results"]) > 0

    def test_empty_query_raises_error(self):
        with pytest.raises(ValueError):
            web_search.invoke({"query": ""})

    def test_network_error_handled_gracefully(self, mock_network_error):
        result = web_search.invoke({"query": "test"})
        assert result["error"] is not None
        assert result["results"] == []

No more "the tool works because the agent produced a good result." Every tool is validated independently.

2. Conditional edges have deterministic tests

class TestSupervisorRouting:
    def test_routes_research_query_to_researcher(self):
        state = {"task": "Research quantum computing", "phase": "planning"}
        next_node = supervisor_routing(state)
        assert next_node == "researcher"

    def test_routes_analysis_to_analyst_after_research(self):
        state = {
            "phase": "executing",
            "researcher_result": {"sources": [...]},
            "analyst_result": None
        }
        next_node = supervisor_routing(state)
        assert next_node == "analyst"

    def test_routes_to_validation_when_all_complete(self):
        state = {
            "phase": "executing",
            "researcher_result": {"sources": [...]},
            "analyst_result": {"findings": [...]},
            "writer_result": {"report": "..."}
        }
        next_node = supervisor_routing(state)
        assert next_node == "validate"

The Supervisor's routing is deterministic — it depends on the state, not on the LLM. Perfectly testable with unit tests.

3. Trajectories are captured and evaluated

class TestResearchTrajectory:
    async def test_research_trajectory_is_efficient(self):
        result = await research_agent.ainvoke({
            "task": "Applications of quantum computing in healthcare"
        })

        trajectory = result["trajectory"]

        assert len(trajectory.tool_calls) <= 5, "Too many tool calls"
        assert trajectory.has_tool("web_search"), "Should have used web_search"
        assert trajectory.has_tool("paper_search"), "Should have used paper_search"
        assert trajectory.total_cost < 0.15, "Excessive cost"

        quality = await evaluate_trajectory(trajectory, judge_model="gpt-4.1")
        assert quality.reasoning_score >= 0.7, "Weak reasoning"
        assert quality.tool_selection_score >= 0.8, "Poor tool selection"

You don't just verify the result — you verify how it got to the result.

4. The golden dataset automates regression testing

GOLDEN_DATASET = [
    {
        "query": "Compare machine learning frameworks for production",
        "expected_tools": ["web_search", "paper_search"],
        "expected_sections": ["comparison", "recommendations"],
        "min_sources": 3,
        "max_cost": 0.12,
    },
    {
        "query": "Explain quantum entanglement for beginners",
        "expected_tools": ["web_search"],
        "expected_sections": ["explanation", "examples"],
        "min_sources": 2,
        "max_cost": 0.08,
    },
    # ... 18+ more queries
]

@pytest.mark.parametrize("case", GOLDEN_DATASET)
async def test_golden_case(case):
    result = await research_agent.ainvoke({"task": case["query"]})
    # Verify tools, sections, sources, cost...

20+ queries that run automatically. If you change a prompt and 3 out of 20 fail, you know before the deploy.

5. The full evolution of the Research Agent

v1 (M4): Basic state machine
         → planning → research → synthesis → END
         Tests: none

v2 (M5): + Intelligent planning + Reflection
         → planning_v2 → research → analysis → reflection → [quality gate]
         Tests: none

v3 (M6): + Memory (checkpointing + long-term)
         → Same graph + checkpointer + memory store
         Tests: none

v4 (M7): + MCP dynamic tools
         → Same graph + MCP clients
         Tests: none

v5 (M8): → SYSTEM of 4 coordinated agents
         Supervisor → Researcher + Analyst + Writer
         Tests: none

v6 (M9): → SAME SYSTEM + complete testing suite
         30+ unit tests, 5 integration tests, trajectory evaluation,
         golden dataset (20+ queries), LangSmith, regression CI/CD,
         baseline benchmarks
         Tests: ✅ EVIDENCE THAT IT WORKS

Every previous version added capabilities. v6 adds confidence — the measurable evidence that those capabilities work as expected.

---

What This Module Does NOT Cover

• ❌ Software testing in general — pytest isn't re-taught from scratch, nor TDD, nor coverage reports. That's baseline software engineering knowledge. Here we use pytest as a tool, but the focus is testing agents, not testing in general
• ❌ State machine fundamentals — StateGraph, nodes, and edges aren't re-taught. That's M4. Here you test the edges and transitions, but we assume you know how to build them
• ❌ Planning or reflection from scratch — Intelligent planning and quality gates aren't re-taught. That's M5. Here you evaluate whether the planning was sensible and whether reflection detected real gaps
• ❌ Multi-agent from scratch — The Supervisor, Handoffs, Subagents, and Router patterns aren't re-taught. That's M8. Here you test that the Supervisor routes well and that handoffs preserve context
• ❌ MCP from scratch — How to create servers or clients isn't re-taught. That's M7. The tests verify the MCP integration, but we assume you know how to configure it
• ❌ Testing language models — We don't test whether GPT-4.1 is "good." We test whether your agent (which uses GPT-4.1) behaves correctly. The difference is crucial: you're not evaluating the model, you're evaluating your system
• ❌ Load testing and stress testing — Performance testing under heavy load (1000 concurrent requests) is a production topic. M10 mentions it. M9 focuses on functional and quality testing
• ❌ A/B testing in production — Comparing two agent versions with real split traffic is a deployment pattern (M10), not a testing one. M9 gives you the metrics; M10 teaches you to compare them in production
• ❌ Testing frameworks other than LangGraph/LangSmith — Alternatives exist (Braintrust, Weights & Biases, etc.) with different evaluation tooling. This module uses LangSmith, consistent with the guide's LangChain/LangGraph stack

The boundary is clear: M9 = how you verify your agent works. M4-M8 = how you build it. M10 = how you put it in production.

---

Evidence of Success

By the end of this module, you'll know you succeeded if:

• ✅ You can run pytest and watch 30+ tests pass — unit tests for tools, state transitions, conditional edges, and prompt formatting — in under 10 seconds without needing an LLM
• ✅ You can run integration tests with a real LLM that capture the agent's full trajectory, and those trajectories get stored as snapshots for future comparison
• ✅ You can evaluate a trajectory and say "the agent used 3 tool calls when 2 would do, its sequence was suboptimal, but the reasoning quality was high" — with numbers, not with feelings
• ✅ LangSmith is configured as your dashboard: you see every agent run with full tracing, you have evaluation datasets running, and you use that information to improve prompts and tools
• ✅ You have a golden dataset of 20+ queries with documented expected behavior, and a regression testing pipeline that runs automatically and alerts you if quality degrades
• ✅ You can communicate your agent's performance with concrete metrics: "task completion rate 85%, tool call accuracy 92%, avg latency 4.2s, avg cost $0.08/task"
• ✅ Your Research Agent v6 has the same architecture as v5 but with measurable evidence that it works — not "I think it works" but "I have data showing it works"

Quick self-assessment test

Ask yourself these questions after completing the module:

1. "If I change the Supervisor's system prompt, does my testing pipeline detect whether something broke?" → If yes, your regression testing works
2. "Can I distinguish between an agent that reached the right result via a good path vs one that got there by luck?" → If yes, you understand trajectory evaluation
3. "Can I state exactly what my agent's task completion rate is, with a number?" → If yes, you have real metrics, not vibes
4. "If a teammate asks me 'how do I know your agent won't fail in production?', can I show evidence?" → If yes, you're ready for M10

If you answered yes to all four → ready for Module 10 (Agents in Production and Alternatives). If you answered no to any → reinforce the corresponding capsule before moving on.

---

Summary

• Testing isn't optional — it's the difference between engineering and faith: M4-M8 built a powerful agent system. M9 adds the evidence that it works. Without tests, every deploy is an act of faith. With tests, it's an informed decision
• Non-determinism requires a different approach: You can't do assert output == expected with agents. You need to evaluate quality, tool selection, trajectory efficiency, and consistency across multiple runs
• Trajectory evaluation is the central concept: Evaluating the agent's path — not just whether the result is correct, but whether it used the right tools, in the right order, with the right arguments, efficiently. The right result via the wrong path is a fragile agent
• The full testing spectrum: Unit tests (tools, edges, state), integration tests (agent loop, real LLM), trajectory evaluation (path quality), regression testing (detect degradation), benchmarks (aggregate metrics). Each type plays a different role
• LangSmith as a daily driver: Not as a 5-minute demo. As a daily working tool: tracing every run, evaluation datasets, custom evaluators, visual debugging. The "eyes" that see what your agent does
• Golden datasets automate confidence: 20+ queries with expected behavior. Automatic pipeline. CI/CD integration. Alerts on degradation. The difference between "I saw it once" and "I verify it on every change"
• Metrics as a language: "Task completion rate 85%, tool call accuracy 92%, avg latency 4.2s, cost $0.08/task." After M9, you talk about your agent with numbers, not with "it seems to work fine"
• Research Agent v6: Same architecture as v5. Same 4 agents. But with 30+ unit tests, integration tests, trajectory evaluation, a golden dataset, LangSmith, regression CI/CD, and baseline benchmarks. From "I think it works" to "I have data showing it works"

---

Resources

1. LangSmith Documentation — Official LangSmith documentation. Tracing, datasets, evaluators, and agent testing. The main tool of this module
2. LangGraph Testing Guide — LangGraph's official guide to agent testing. Fixtures, mocking, and test patterns for StateGraph
3. Evaluating LLM Applications — LangChain — LangChain's guide to evaluation: evaluation chains, criteria evaluation, and trajectory evaluation
4. Building Effective Agents — Anthropic — Anthropic's perspective on agent evaluation. Complements it with quality assurance patterns
5. How to Evaluate AI Agents — Hamel Husain — A practical perspective on agent evaluation: golden datasets, LLM-as-judge, and regression testing in production
6. pytest Documentation — Official pytest documentation. Fixtures, parametrize, asyncio — the tools we use to implement the tests

---