4. Trajectory Evaluation

Overview

In the previous capsules you tested your agent's individual pieces: tools with unit tests, the complete loop with integration tests. But there's a question those tests don't answer: did the agent reach the right result by the right path? An agent can produce the perfect answer and still have called unnecessary tools, used the wrong tool that happened to return something useful, or taken 8 steps when 3 would have done. The final result is necessary — but not sufficient.

Trajectory evaluation means evaluating the path, not just the destination. An agent's trajectory is the complete sequence of decisions, tool calls, and reasoning it executed from receiving the task to producing its answer. Evaluating that sequence tells you things the final output will never reveal: whether the agent was efficient, whether it used the right tools, whether its plan made sense. This matters in production because the path directly affects your system's cost, latency, and reliability.

Connection to the module: This capsule is the module's differentiating concept — and probably the whole guide's. Most agent courses evaluate only the final result. Here you're going to build evaluators that inspect every step of the path. In capsule 05 you'll integrate this with LangSmith. In 06, you'll create golden datasets with expected trajectories for regression testing. But it all starts here: understanding what a trajectory is, why it matters, and how to evaluate it.

What a Trajectory Is

The (Thought, Action, Observation) sequence

Every agent step has three components: Thought (the LLM's reasoning), Action (a tool call with a name and arguments), and Observation (the result injected back into the context). The trajectory is the ordered list of these triplets:

Trajectory of a Research Agent
═══════════════════════════════════════════════════════
Step 1:
  Thought:  "I need to search for studies on coffee and sleep."
  Action:   web_search(query="coffee effects on sleep studies 2024")
  Observation: [3 results with URLs and snippets]

Step 2:
  Thought:  "I need data on caffeine and REM cycles."
  Action:   web_search(query="caffeine REM sleep cycle research")
  Observation: [2 results with data on REM cycles]

Step 3:
  Thought:  "I have enough information. I can synthesize."
  Action:   final_answer("Coffee affects sleep in three ways...")

Total: 3 steps, 2 tool calls, 1 final answer

Structured representation and extraction

from dataclasses import dataclass, field
from langchain_core.messages import AIMessage, ToolMessage, HumanMessage

@dataclass
class TrajectoryStep:
    thought: str
    action: str
    action_args: dict
    observation: str
    step_number: int

@dataclass
class Trajectory:
    query: str
    steps: list[TrajectoryStep] = field(default_factory=list)
    final_answer: str = ""
    total_tool_calls: int = 0
    tools_used: list[str] = field(default_factory=list)

def extract_trajectory(messages: list) -> Trajectory:
    """Extracts the structured trajectory from the agent's message list."""
    trajectory = Trajectory(query="")
    step_num = 0
    for msg in messages:
        if isinstance(msg, HumanMessage) and step_num == 0:
            trajectory.query = msg.content
        elif isinstance(msg, AIMessage) and msg.tool_calls:
            step_num += 1
            for tc in msg.tool_calls:
                trajectory.steps.append(TrajectoryStep(
                    msg.content or "", tc["name"], tc["args"], "", step_num))
                trajectory.tools_used.append(tc["name"])
                trajectory.total_tool_calls += 1
        elif isinstance(msg, ToolMessage) and trajectory.steps:
            trajectory.steps[-1].observation = msg.content
        elif isinstance(msg, AIMessage) and not msg.tool_calls:
            trajectory.final_answer = msg.content
    return trajectory

You build the evaluation on top of this structure.

Why the Result Isn't Enough

The mirage of the correct result

"What is the capital of France?" → "Paris." Test passes. But look at the trajectory:

INEFFICIENT trajectory (correct result, bad path)
═══════════════════════════════════════════════════════
Step 1: web_search("capital of France")            → results
Step 2: web_search("France capital city")           → redundant
Step 3: web_search("Paris is the capital of what country") → unnecessary
Step 4: calculator("population of Paris")           → irrelevant
Step 5: final_answer("Paris")
→ 5 steps, 4 tool calls, ~$0.03, ~4.2 seconds

EFFICIENT trajectory (correct result, correct path)
═══════════════════════════════════════════════════════
Step 1: final_answer("Paris")
→ 1 step, 0 tool calls, ~$0.002, ~0.3 seconds

Both produce "Paris." But the first cost 15x more, took 14x longer, and called calculator for no reason.

Three categories of invisible failures

1. Incorrect tool selection — Using web_search for "What's 15% of 230?" when it should have used calculator. Unnecessary latency and a network dependency.

2. Too many steps — It has the URL but first runs a web_search to find the article. Extra tool calls = more cost, more points of failure.

3. Suboptimal arguments — Expected: web_search("transformer architecture NLP papers 2024"). Actual: web_search("transformers"). Results about the movies, not NLP.

The real cost

Trajectory Metrics

To evaluate a trajectory rigorously, you need five metrics:

1. Tool Selection Accuracy

def tool_selection_accuracy(actual_tools: list[str], expected_tools: list[str]) -> float:
    """The proportion of expected tools the agent actually used."""
    if not expected_tools:
        return 1.0 if not actual_tools else 0.0
    expected_set = set(expected_tools)
    actual_set = set(actual_tools)
    return len(expected_set & actual_set) / len(expected_set)

2. Tool Order Correctness

def tool_order_score(actual_tools: list[str], expected_order: list[str]) -> float:
    """Normalized Longest Common Subsequence — evaluates whether the order respects the expected sequence."""
    if not expected_order:
        return 1.0
    def lcs_length(a, b):
        m, n = len(a), len(b)
        dp = [[0] * (n + 1) for _ in range(m + 1)]
        for i in range(1, m + 1):
            for j in range(1, n + 1):
                if a[i-1] == b[j-1]:
                    dp[i][j] = dp[i-1][j-1] + 1
                else:
                    dp[i][j] = max(dp[i-1][j], dp[i][j-1])
        return dp[m][n]
    return lcs_length(actual_tools, expected_order) / len(expected_order)

3. Argument Precision

def argument_precision(actual_args: dict, expected_args: dict) -> float:
    """Evaluates the precision of a tool call's arguments."""
    if not expected_args:
        return 1.0
    scores = []
    for key, expected_value in expected_args.items():
        actual_value = actual_args.get(key)
        if actual_value is None:
            scores.append(0.0)
        elif actual_value == expected_value:
            scores.append(1.0)
        elif isinstance(expected_value, str) and isinstance(actual_value, str):
            expected_words = set(expected_value.lower().split())
            actual_words = set(actual_value.lower().split())
            overlap = len(expected_words & actual_words)
            total = len(expected_words | actual_words)
            scores.append(overlap / total if total > 0 else 0.0)
        else:
            scores.append(0.0)
    return sum(scores) / len(scores) if scores else 1.0

4. Step Efficiency

def step_efficiency(actual_steps: int, expected_steps: int) -> float:
    """1.0 = optimal, < 1.0 = inefficient."""
    if actual_steps == 0:
        return 0.0
    return min(expected_steps / actual_steps, 1.0)

5. Reasoning Coherence (heuristic)

def reasoning_coherence_heuristic(trajectory: Trajectory) -> float:
    """Penalizes steps without reasoning and repeated identical tool calls."""
    penalties = 0
    total_checks = 0
    for i, step in enumerate(trajectory.steps):
        total_checks += 1
        if not step.thought:
            penalties += 1
            continue
        if i > 0:
            prev = trajectory.steps[i - 1]
            if prev.action == step.action and prev.action_args == step.action_args:
                penalties += 1
    return 1.0 - (penalties / total_checks) if total_checks > 0 else 1.0

Combining into a final score

@dataclass
class TrajectoryScore:
    tool_selection: float
    tool_order: float
    argument_precision: float
    step_efficiency: float
    reasoning_coherence: float

    @property
    def weighted_score(self) -> float:
        return (self.tool_selection * 0.30 + self.tool_order * 0.20
                + self.argument_precision * 0.20 + self.step_efficiency * 0.15
                + self.reasoning_coherence * 0.15)

Implementing Trajectory Evaluation

The flow: capture → compare → report

from langchain_openai import ChatOpenAI
from langgraph.prebuilt import create_react_agent
from langchain_core.tools import tool

@tool
def web_search(query: str) -> str:
    """Searches for information on the web."""
    return f"Results for: {query}"

@tool
def calculator(expression: str) -> str:
    """Evaluates a mathematical expression."""
    return str(eval(expression))

model = ChatOpenAI(model="gpt-4o", temperature=0)
agent = create_react_agent(model, tools=[web_search, calculator])

def run_and_capture(query: str) -> Trajectory:
    """Runs the agent and captures its trajectory."""
    result = agent.invoke({"messages": [HumanMessage(content=query)]})
    return extract_trajectory(result["messages"])

Defining expected trajectories

@dataclass
class ExpectedTrajectory:
    expected_tools: list[str]
    expected_order: list[str]
    expected_args: list[dict]
    expected_steps: int
    description: str = ""

test_cases = [
    {"query": "What's 15% of 230?",
     "expected": ExpectedTrajectory(["calculator"], ["calculator"],
         [{"expression": "230 * 0.15"}], 1, "Simple operation, uses calculator")},
    {"query": "Look up Mexico's GDP and calculate 3% growth",
     "expected": ExpectedTrajectory(["web_search", "calculator"], ["web_search", "calculator"],
         [{"query": "GDP Mexico 2024"}, {"expression": "GDP * 0.03"}], 2, "Search + calculation")},
]

Evaluating and reporting

def evaluate_trajectory(trajectory: Trajectory, expected: ExpectedTrajectory) -> TrajectoryScore:
    ts = tool_selection_accuracy(trajectory.tools_used, expected.expected_tools)
    to = tool_order_score(trajectory.tools_used, expected.expected_order)

    arg_scores = []
    for i, exp_args in enumerate(expected.expected_args):
        if i < len(trajectory.steps):
            arg_scores.append(argument_precision(trajectory.steps[i].action_args, exp_args))
        else:
            arg_scores.append(0.0)
    ap = sum(arg_scores) / len(arg_scores) if arg_scores else 1.0
    se = step_efficiency(len(trajectory.steps), expected.expected_steps)
    rc = reasoning_coherence_heuristic(trajectory)

    return TrajectoryScore(ts, to, ap, se, rc)

def run_evaluation_suite(test_cases: list[dict]) -> list[dict]:
    results = []
    for case in test_cases:
        trajectory = run_and_capture(case["query"])
        score = evaluate_trajectory(trajectory, case["expected"])
        results.append({"query": case["query"], "tools_used": trajectory.tools_used,
                        "steps": len(trajectory.steps), "score": score})
        print(f"Query: {case['query']}")
        print(f"  Tools: {trajectory.tools_used} | Steps: {len(trajectory.steps)}")
        print(f"  Score: {score.weighted_score:.2f}")
    avg = sum(r["score"].weighted_score for r in results) / len(results)
    print(f"\nAverage score: {avg:.2f} | Tests: {len(results)}")
    return results

LLM-as-Judge for Trajectories

Programmatic metrics cover the mechanical aspects. But there are things only an LLM evaluates well: was the reasoning coherent? was the strategy smart? did the agent adapt its plan when results surprised it?

Structured scoring rubric

from pydantic import BaseModel, Field

class TrajectoryJudgment(BaseModel):
    reasoning_quality: int = Field(
        ge=1, le=5, description="1=incoherent, 3=acceptable, 5=excellent")
    strategy_quality: int = Field(
        ge=1, le=5, description="1=no strategy, 3=reasonable, 5=optimal")
    adaptability: int = Field(
        ge=1, le=5, description="1=ignores results, 3=adapts partially, 5=optimal")
    information_usage: int = Field(
        ge=1, le=5, description="1=ignores info, 3=uses partially, 5=uses all of it")
    overall_justification: str = Field(description="Brief justification")

    @property
    def average_score(self) -> float:
        return (self.reasoning_quality + self.strategy_quality
                + self.adaptability + self.information_usage) / 4

Implementing the judge

JUDGE_PROMPT = """You are an expert evaluator of AI agents. Evaluate the QUALITY
OF THE TRAJECTORY — the path the agent took, not just the final result.

## Context
- Query: {query}
- Final answer: {final_answer}

## Trajectory
{trajectory_formatted}

## Rubric (1-5 per dimension)
1. reasoning_quality: Is the reasoning at each step coherent?
2. strategy_quality: Was the overall strategy smart?
3. adaptability: Did it adapt its plan based on the results?
4. information_usage: Did it use the information it obtained correctly?

Be rigorous. A 5 means no improvement is possible."""

def format_trajectory_for_judge(trajectory: Trajectory) -> str:
    lines = []
    for step in trajectory.steps:
        lines.append(f"Step {step.step_number}:")
        if step.thought:
            lines.append(f"  Thought: {step.thought}")
        lines.append(f"  Action: {step.action}({step.action_args})")
        lines.append(f"  Result: {step.observation[:200]}")
        lines.append("")
    return "\n".join(lines)

def judge_trajectory(trajectory: Trajectory) -> TrajectoryJudgment:
    judge_model = ChatOpenAI(model="gpt-4o", temperature=0)
    structured_judge = judge_model.with_structured_output(TrajectoryJudgment)
    prompt = JUDGE_PROMPT.format(
        query=trajectory.query, final_answer=trajectory.final_answer,
        trajectory_formatted=format_trajectory_for_judge(trajectory),
    )
    return structured_judge.invoke(prompt)

Combining programmatic scoring and LLM-as-Judge

@dataclass
class FullTrajectoryEvaluation:
    programmatic_score: TrajectoryScore
    llm_judgment: TrajectoryJudgment

    @property
    def combined_score(self) -> float:
        prog = self.programmatic_score.weighted_score
        llm = self.llm_judgment.average_score / 5.0
        return prog * 0.6 + llm * 0.4

The judge's consistency

LLM-as-Judge has natural variability. Mitigate it by running multiple evaluations:

def judge_with_consensus(trajectory: Trajectory, n_judges: int = 3) -> TrajectoryJudgment:
    judgments = [judge_trajectory(trajectory) for _ in range(n_judges)]
    return TrajectoryJudgment(
        reasoning_quality=round(sum(j.reasoning_quality for j in judgments) / n_judges),
        strategy_quality=round(sum(j.strategy_quality for j in judgments) / n_judges),
        adaptability=round(sum(j.adaptability for j in judgments) / n_judges),
        information_usage=round(sum(j.information_usage for j in judgments) / n_judges),
        overall_justification=f"Consensus of {n_judges} evaluations. "
            + judgments[0].overall_justification,
    )

Three judges averaged reduces the variance. The tradeoff: 3x the cost per evaluation.

Custom Evaluators

Generic metrics are a baseline. But your agent has specific behaviors that require custom evaluators. A Research Agent needs to check: did it cite sources? Did it use paper_reader for papers? Did it respect the call budget?

The base structure

from abc import ABC, abstractmethod

class TrajectoryEvaluator(ABC):
    @property
    @abstractmethod
    def name(self) -> str: ...

    @abstractmethod
    def evaluate(self, trajectory: Trajectory) -> float: ...

    @abstractmethod
    def explanation(self, trajectory: Trajectory) -> str: ...

Three concrete evaluators

class SpecializedToolEvaluator(TrajectoryEvaluator):
    """Checks that the agent uses specialized tools when the context requires them."""
    def __init__(self, rules: dict[str, str]):
        self.rules = rules  # keyword → expected_tool

    @property
    def name(self) -> str: return "specialized_tool_usage"

    def evaluate(self, trajectory: Trajectory) -> float:
        violations, checks = 0, 0
        for step in trajectory.steps:
            for keyword, expected_tool in self.rules.items():
                if keyword.lower() in str(step.action_args).lower():
                    checks += 1
                    if step.action != expected_tool: violations += 1
        return 1.0 - (violations / checks) if checks > 0 else 1.0

    def explanation(self, trajectory: Trajectory) -> str:
        issues = [f"Step {s.step_number}: used '{s.action}', should have used '{exp}'"
                  for s in trajectory.steps for kw, exp in self.rules.items()
                  if kw.lower() in str(s.action_args).lower() and s.action != exp]
        return "; ".join(issues) if issues else "OK"


class RedundancyEvaluator(TrajectoryEvaluator):
    """Detects duplicate tool calls."""
    @property
    def name(self) -> str: return "redundancy_check"

    def evaluate(self, trajectory: Trajectory) -> float:
        if len(trajectory.steps) <= 1: return 1.0
        seen, redundant = set(), 0
        for step in trajectory.steps:
            sig = f"{step.action}:{sorted(step.action_args.items())}"
            if sig in seen: redundant += 1
            seen.add(sig)
        return 1.0 - (redundant / len(trajectory.steps))

    def explanation(self, trajectory: Trajectory) -> str:
        seen, dupes = {}, []
        for s in trajectory.steps:
            sig = f"{s.action}({s.action_args})"
            if sig in seen: dupes.append(f"Step {s.step_number} repeats step {seen[sig]}")
            seen[sig] = s.step_number
        return "; ".join(dupes) if dupes else "No redundancy"


class BudgetEvaluator(TrajectoryEvaluator):
    """Checks that the agent doesn't exceed a tool call budget."""
    def __init__(self, max_tool_calls: int = 5):
        self.max_calls = max_tool_calls

    @property
    def name(self) -> str: return "budget_compliance"

    def evaluate(self, trajectory: Trajectory) -> float:
        if trajectory.total_tool_calls <= self.max_calls: return 1.0
        return max(0.0, 1.0 - ((trajectory.total_tool_calls - self.max_calls) * 0.2))

    def explanation(self, trajectory: Trajectory) -> str:
        status = "within budget" if trajectory.total_tool_calls <= self.max_calls else f"exceeded by {trajectory.total_tool_calls - self.max_calls}"
        return f"{trajectory.total_tool_calls}/{self.max_calls} calls — {status}"

Running the evaluators

def run_custom_evaluators(trajectory: Trajectory, evaluators: list[TrajectoryEvaluator]) -> dict:
    results = {}
    for ev in evaluators:
        score = ev.evaluate(trajectory)
        status = "PASS" if score >= 0.7 else "FAIL"
        print(f"  [{status}] {ev.name}: {score:.2f} — {ev.explanation(trajectory)}")
        results[ev.name] = {"score": score, "explanation": ev.explanation(trajectory), "pass": score >= 0.7}
    return results

Connection to the Project

In the module's project (capsule 08), the Research Agent gets evaluated with a complete trajectory evaluation:

┌──────────────────────────────────────────────────────────┐
│              TRAJECTORY EVALUATION SUITE                  │
│                                                          │
│  1. Programmatic metrics                                 │
│     Tool selection · Tool order · Arguments · Efficiency │
│                                                          │
│  2. LLM-as-Judge                                         │
│     Reasoning · Strategy · Adaptability · Info usage     │
│                                                          │
│  3. Custom evaluators                                    │
│     SpecializedTool · Redundancy · Budget · Citations    │
│                                                          │
│  Input: 20 queries from the golden dataset (capsule 06)  │
│  Output: Score per query + averages + regressions        │
└──────────────────────────────────────────────────────────┘

Each golden dataset query will have not just an expected answer, but an expected trajectory: which tools it should use, in what order, with what arguments. That catches subtle regressions: "after changing the system prompt, the agent still produces good answers, but now it uses 3 searches where it used 1 before."

Troubleshooting

Problem 1: Trajectory extraction fails on unexpected messages

Cause: The agent produces message types that extract_trajectory doesn't handle — interleaved SystemMessages, or an AIMessage with tool_calls empty but not null.

Solution: Check msg.tool_calls with if msg.tool_calls (which filters empty lists), not with if hasattr(msg, "tool_calls"). Add a try/except per step so one malformed message doesn't break the whole extraction.

Problem 2: Tool selection metrics give 0 when the tools are equivalent

Cause: The agent used search_web but the expected value was web_search. Same tool, different names — the string match fails.

Solution: Create an equivalence mapping and normalize the names before comparing. Alternatively, use a set of acceptable names instead of one exact name.

Problem 3: LLM-as-Judge gives inconsistent scores across runs

Cause: The LLM's natural variability, especially with temperature > 0.

Solution: Use temperature=0. Implement judge_with_consensus with 3 evaluations. If the standard deviation is > 1 point on any dimension, flag it for manual review.

Problem 4: Step efficiency penalizes agents that explore correctly

Cause: Some queries require legitimate exploration — several searches with progressive refinement.

Solution: Define expected_steps as a range (expected_min, expected_max). If the actual value is within the range, score = 1.0. If it exceeds the maximum, penalize proportionally.

Problem 5: The custom evaluators are too strict

Cause: The 0.7 threshold is arbitrary and may not apply to your domain.

Solution: Start with no thresholds — just collect scores. After 50-100 evaluations, analyze the distribution and define thresholds based on real percentiles. A good starting point: your agent's 25th percentile.

Exercises

Exercise 1: Extract and analyze a trajectory (Easy)

Given this simulated history, use extract_trajectory and report: tool calls, tools used, and steps executed.

messages = [
    HumanMessage(content="How much does a flight to Madrid cost?"),
    AIMessage(content="", tool_calls=[{"name": "web_search", "args": {"query": "flight price Madrid"}, "id": "tc1"}]),
    ToolMessage(content="Flights from $200", tool_call_id="tc1"),
    AIMessage(content="", tool_calls=[{"name": "web_search", "args": {"query": "cheapest flights Madrid 2024"}, "id": "tc2"}]),
    ToolMessage(content="Deals from $150", tool_call_id="tc2"),
    AIMessage(content="Flights to Madrid cost between $150 and $200."),
]

trajectory = extract_trajectory(messages)
# total_tool_calls: 2 | tools_used: ['web_search', 'web_search'] | steps: 2
# Was the second search necessary? That's what trajectory evaluation answers.

Exercise 2: Implement tool selection F1 with a penalty for extras (Medium)

tool_selection_accuracy doesn't penalize extra tools. Implement tool_selection_f1 that penalizes both missing and extra tools using precision and recall.

def tool_selection_f1(actual_tools: list[str], expected_tools: list[str]) -> float:
    if not expected_tools and not actual_tools: return 1.0
    if not expected_tools or not actual_tools: return 0.0
    expected_set, actual_set = set(expected_tools), set(actual_tools)
    tp = len(expected_set & actual_set)
    precision = tp / (tp + len(actual_set - expected_set)) if actual_set else 0.0
    recall = tp / (tp + len(expected_set - actual_set)) if expected_set else 0.0
    if precision + recall == 0: return 0.0
    return 2 * (precision * recall) / (precision + recall)

# Perfect: 1.00 | Extra tool: 0.80 | Missing: 0.67 | Wrong: 0.00

Exercise 3: Evaluate a complete trajectory with every metric (Medium)

Compute the 5 metrics and the final score for this trajectory and its expected reference:

actual = Trajectory(
    query="Look up the price of Bitcoin and calculate how many BTC I can buy with $5000",
    steps=[
        TrajectoryStep("Searching the price", "web_search", {"query": "bitcoin price"}, "$42,000", 1),
        TrajectoryStep("More info", "web_search", {"query": "BTC USD today"}, "$42,150", 2),
        TrajectoryStep("Calculating", "calculator", {"expression": "5000 / 42000"}, "0.119", 3),
    ],
    final_answer="You can buy ~0.119 BTC", total_tool_calls=3,
    tools_used=["web_search", "web_search", "calculator"],
)

expected = ExpectedTrajectory(
    expected_tools=["web_search", "calculator"],
    expected_order=["web_search", "calculator"],
    expected_args=[{"query": "bitcoin price USD"}, {"expression": "5000 / PRICE"}],
    expected_steps=2,
)

score = evaluate_trajectory(actual, expected)
# tool_selection=1.00 (both expected tools present)
# tool_order=1.00 (web_search → calculator is correct)
# argument_precision≈0.60 ("bitcoin price" vs "bitcoin price USD" = partial overlap)
# step_efficiency=0.67 (2 expected / 3 actual)
# reasoning_coherence≈0.67 (the second search is redundant)
# weighted_score≈0.82

Exercise 4: Implement a custom source-citation evaluator (Hard)

Create a SourceCitationEvaluator that checks whether the URLs obtained in the observations appear in the final answer.

import re

class SourceCitationEvaluator(TrajectoryEvaluator):
    @property
    def name(self) -> str: return "source_citation"

    def _extract_urls(self, text: str) -> set[str]:
        return set(re.findall(r'https?://[^\s<>"{}|\\^`\[\]]+', text))

    def evaluate(self, trajectory: Trajectory) -> float:
        sources = set()
        for step in trajectory.steps:
            sources.update(self._extract_urls(step.observation))
        if not sources: return 1.0
        return len(sources & self._extract_urls(trajectory.final_answer)) / len(sources)

    def explanation(self, trajectory: Trajectory) -> str:
        sources = set()
        for step in trajectory.steps:
            sources.update(self._extract_urls(step.observation))
        cited = sources & self._extract_urls(trajectory.final_answer)
        return f"Sources: {len(sources)}, cited: {len(cited)}"

# Test: the agent gets 2 URLs, cites only 1 → score 0.50

Exercise 5: A complete pipeline with a report (Hard)

Build generate_evaluation_report that: (a) runs the agent against the test cases, (b) applies the programmatic metrics + custom evaluators, (c) generates a report with scores per test case, averages, and a list of the cases that failed (score < 0.7).

@dataclass
class EvaluationReport:
    results: list[dict]
    avg_score: float
    failed_cases: list[dict]

    def print_summary(self):
        print(f"\n{'='*60}")
        print(f"TRAJECTORY EVALUATION REPORT")
        print(f"{'='*60}")
        print(f"Total: {len(self.results)} | Passed: {len(self.results) - len(self.failed_cases)} | Failed: {len(self.failed_cases)}")
        print(f"Average score: {self.avg_score:.2f}")
        if self.failed_cases:
            print(f"\nFailed cases:")
            for c in self.failed_cases:
                print(f"  - {c['query']}: {c['combined']:.2f}")
        print(f"{'='*60}")

def generate_evaluation_report(
    test_cases: list[dict], evaluators: list[TrajectoryEvaluator],
) -> EvaluationReport:
    all_results = []
    for case in test_cases:
        traj = run_and_capture(case["query"])
        prog = evaluate_trajectory(traj, case["expected"])
        custom = run_custom_evaluators(traj, evaluators)
        custom_avg = sum(r["score"] for r in custom.values()) / len(custom) if custom else 1.0
        combined = prog.weighted_score * 0.7 + custom_avg * 0.3
        all_results.append({"query": case["query"], "programmatic": prog.weighted_score,
                           "custom_avg": custom_avg, "combined": combined})

    avg = sum(r["combined"] for r in all_results) / len(all_results)
    failed = [r for r in all_results if r["combined"] < 0.7]
    report = EvaluationReport(results=all_results, avg_score=avg, failed_cases=failed)
    report.print_summary()
    return report

# Usage
report = generate_evaluation_report(
    test_cases=test_cases,
    evaluators=[RedundancyEvaluator(), BudgetEvaluator(max_tool_calls=5)],
)

Summary

In this capsule you built the complete trajectory evaluation framework — the differentiating concept that separates superficial evaluation from rigorous agent evaluation:

• A trajectory is the complete sequence of (thought, action, observation) at each of the agent's steps. It isn't just which tools it called — it's what it thought, which arguments it used, and what it got back.
• The final result is necessary but not sufficient. An agent can reach the right answer by an inefficient, expensive, or fragile path. Only by evaluating the trajectory do you catch these problems.
• Five programmatic metrics cover the mechanical aspects: tool selection accuracy, tool order correctness, argument precision, step efficiency, and reasoning coherence.
• LLM-as-Judge complements the metrics by evaluating qualitative aspects with a structured rubric: reasoning quality, strategy quality, adaptability, information usage.
• Custom evaluators capture business rules: did it use the specialized tool? Did it cite sources? Did it respect the tool call budget?
• The complete pipeline — capture, compare, report — is what you'll integrate into CI for automatic regression testing.

Next capsule: LangSmith for Agents — tracing a complete run, visual debugging, and evaluation datasets that automate everything you built here.

Additional Resources

1. LangChain — Agent Trajectory Evaluation — Official documentation on evaluating trajectories with LangChain evaluators
2. LangSmith — Evaluating Agent Trajectories — A guide to evaluating trajectories inside LangSmith with datasets and evaluators
3. LLM-as-a-Judge — Prompting Guide — Techniques for using LLMs as evaluators, including rubrics and consistency
4. ReAct Pattern — Original Paper — The original ReAct paper that defines the (Thought, Action, Observation) cycle structuring trajectories
5. Evaluating LLM-based Applications — DeepLearning.AI — A short course on evaluating LLM applications