3. Integration Testing with a Real LLM

Overview

In the previous capsule you built unit tests: you tested individual tools with controlled inputs, mocked the LLM to verify routing, and validated state transitions without making a single real call. Unit tests are fast, cheap, and deterministic. They're the first line of defense. But there's an uncomfortable truth: an agent where every unit test passes can fail completely in production.

Why? Because unit tests isolate components. You mock the LLM, you mock the tools, you mock the state — and you verify that each piece works on its own. But you never verify that the pieces work together. You never verify that the real LLM picks the right tool with the real prompt. You never verify that a tool's response is parsed correctly on the agent's next turn. Integration tests fill that gap: they run the complete agent, with a real LLM, against real queries, and verify the behavior end-to-end.

Connection to the module: This capsule is the bridge between capsule 02's unit tests and capsule 04's trajectory evaluation. Unit tests validate isolated components. Integration tests validate the complete loop. Trajectory evaluation (the next capsule) goes further: it evaluates not just whether the result is correct, but whether the path was correct. Here you focus on: "does the agent produce an acceptable result when it actually runs?"

Why You Need Tests with a Real LLM

What mocks don't capture

Imagine this scenario: you have an agent with a search_web tool and a calculator tool. Your unit tests pass perfectly:

• search_web returns results when it receives a valid query.
• calculator adds correctly.
• The routing sends search queries to the researcher and calculation queries to the analyst.

All green. You deploy. A user asks: "How many inhabitants does Mexico have and what is that divided by 32 states?" And the agent fails. Why? Because the real LLM first searches for the population, gets "approximately 130 million inhabitants" as text, and then tries to pass that to the calculator — which expects a number, not a string with "approximately" and "million". The integration between one tool's response and the next tool's arguments is invisible to unit tests.

Categories of bugs only integration tests catch

1. Tool selection errors — The LLM picks the wrong tool with the real prompt
2. Argument formatting — The LLM passes arguments in an unexpected format
3. Multi-step coordination — Tool A's result doesn't parse well for tool B
4. Context window overflow — After 3+ tool calls, the context degrades
5. Stop condition failures — The agent doesn't know when to stop iterating
6. Prompt-model interactions — The prompt works with GPT-4o but not with GPT-4o-mini

Without integration tests, you discover these bugs in production. And since agents are non-deterministic, the bug might show up only 1 out of 5 times — making debugging nearly impossible without execution traces. Integration tests cost money and time, but they cost much less than bugs in production.

Implementing Integration Tests

Setting up the testing project

Before writing tests, you need a clear structure:

tests/
├── conftest.py              # Shared fixtures
├── unit/
│   ├── test_tools.py
│   └── test_routing.py
├── integration/
│   ├── test_agent_loop.py   # Tests of the complete loop
│   ├── test_multi_tool.py   # Multi-tool tests
│   └── snapshots/           # Trajectory snapshots
└── pytest.ini

pytest configuration

# pytest.ini
[pytest]
markers =
    unit: Unit tests (no real LLM)
    integration: Integration tests (require a real LLM)
    slow: Tests that take more than 10 seconds
    costly: Tests that cost money (API calls)

testpaths = tests

With these markers you can run only unit tests on each commit and the integration tests only when you need them:

# Unit tests only — fast, free
pytest -m unit

# Integration tests only — slow, costs money
pytest -m integration

# Everything
pytest

Fixtures for integration tests

# tests/conftest.py
import pytest
import os
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage

@pytest.fixture(scope="session")
def llm():
    """A real LLM for integration tests. scope=session to reuse it."""
    return ChatOpenAI(
        model="gpt-4o-mini",
        temperature=0,
        api_key=os.environ["OPENAI_API_KEY"]
    )

@pytest.fixture(scope="session")
def agent_app(llm):
    """Compiles the agent's graph only once per test session."""
    from my_agent.graph import build_agent
    return build_agent(llm)

@pytest.fixture
def run_agent(agent_app):
    """A helper that runs the agent and returns the complete result."""
    def _run(query: str, config: dict = None):
        config = config or {"configurable": {"thread_id": "test"}}
        result = agent_app.invoke(
            {"messages": [HumanMessage(content=query)]},
            config=config
        )
        return result
    return _run

Three key decisions: scope="session" so the agent isn't recreated for every test, temperature=0 to reduce variability, and gpt-4o-mini instead of gpt-4o to make the tests cheaper.

Your first integration test

# tests/integration/test_agent_loop.py
import pytest
from langchain_core.messages import AIMessage, ToolMessage

@pytest.mark.integration
@pytest.mark.slow
def test_agent_produces_final_response(run_agent):
    """The agent must produce a final response (not get stuck in a loop)."""
    result = run_agent("What is the capital of France?")

    messages = result["messages"]
    final_message = messages[-1]

    assert isinstance(final_message, AIMessage), (
        f"The last message must be an AIMessage, got {type(final_message)}"
    )
    assert final_message.content, "The final response must not be empty"
    assert not final_message.tool_calls, "The final response must not have tool_calls"

@pytest.mark.integration
@pytest.mark.slow
def test_agent_uses_tool_when_needed(run_agent):
    """The agent must use tools for queries that require external data."""
    result = run_agent("Look up the current weather in Madrid")

    messages = result["messages"]
    tool_calls = [
        m for m in messages
        if hasattr(m, "tool_calls") and m.tool_calls
    ]
    tool_responses = [
        m for m in messages
        if isinstance(m, ToolMessage)
    ]

    assert len(tool_calls) > 0, "The agent must make at least one tool call"
    assert len(tool_responses) > 0, "There must be at least one tool response"

@pytest.mark.integration
@pytest.mark.slow
def test_agent_handles_unknown_query_gracefully(run_agent):
    """The agent must answer something coherent for ambiguous queries."""
    result = run_agent("xyzzy12345 foobar nonsense")

    messages = result["messages"]
    final_message = messages[-1]

    assert isinstance(final_message, AIMessage)
    assert len(final_message.content) > 10, (
        "It must produce a substantive, non-empty response"
    )

Notice what we're NOT asserting: the exact content of the response. We verify structural properties: that there's a final response, that tools are used when appropriate, that the agent doesn't collapse on weird queries.

Multi-tool tests

@pytest.mark.integration
@pytest.mark.slow
def test_agent_chains_multiple_tools(run_agent):
    """The agent must be able to chain 2+ tools to answer."""
    result = run_agent(
        "Look up the population of Japan and calculate how much that is divided by 47 prefectures"
    )

    messages = result["messages"]
    tool_messages = [m for m in messages if isinstance(m, ToolMessage)]

    assert len(tool_messages) >= 2, (
        f"Expected 2+ tool calls, got {len(tool_messages)}"
    )

    final = messages[-1]
    assert isinstance(final, AIMessage)
    assert final.content

Timeout protection

Agents can get into infinite loops. Protect your tests:

import pytest

@pytest.mark.integration
@pytest.mark.timeout(30)
def test_agent_completes_within_timeout(run_agent):
    """The agent must finish in less than 30 seconds."""
    result = run_agent("Summarize in one sentence what Python is")

    assert result["messages"][-1].content

Install pytest-timeout:

pip install pytest-timeout

If a test goes over 30 seconds, pytest-timeout kills it and marks it as failed. This prevents your CI from hanging indefinitely because of a looping agent.

Asserting on Non-deterministic Output

Run the same agent with the same query 5 times. You'll get 5 different answers. That's normal for LLMs — but it breaks the traditional testing paradigm where assert result == expected always works. You need different strategies.

Strategy 1: Structural assertions

Instead of checking the exact content, check the structure:

def test_structural_assertions(run_agent):
    result = run_agent("What's the temperature in Buenos Aires?")
    messages = result["messages"]

    assert len(messages) >= 3, "Minimum: user msg + tool call + response"

    has_tool_call = any(
        hasattr(m, "tool_calls") and m.tool_calls
        for m in messages
    )
    assert has_tool_call, "It must use the weather tool"

    final = messages[-1]
    assert isinstance(final, AIMessage)
    assert len(final.content) > 20

Strategy 2: Flexible content assertions

Check that the response contains key concepts, not exact text:

def test_flexible_content_assertions(run_agent):
    result = run_agent("What is the capital of Japan?")
    response = result["messages"][-1].content.lower()

    assert any(keyword in response for keyword in ["tokyo", "tōkyō", "tokio"]), (
        f"The response must mention Tokyo. Got: {response[:200]}"
    )

def test_numeric_range_assertion(run_agent):
    result = run_agent("How many continents are there?")
    response = result["messages"][-1].content

    import re
    numbers = re.findall(r'\d+', response)
    assert any(int(n) in [5, 6, 7] for n in numbers), (
        f"It must mention 5, 6, or 7 continents. Got: {response[:200]}"
    )

Strategy 3: Semantic similarity

To verify the response is semantically correct without requiring exact text, use embeddings:

from langchain_openai import OpenAIEmbeddings
from numpy import dot
from numpy.linalg import norm

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

def semantic_similarity(text_a: str, text_b: str) -> float:
    vecs = embeddings.embed_documents([text_a, text_b])
    return dot(vecs[0], vecs[1]) / (norm(vecs[0]) * norm(vecs[1]))

def test_semantic_correctness(run_agent):
    result = run_agent("Explain what machine learning is in one sentence")
    response = result["messages"][-1].content

    expected = (
        "Machine learning is a branch of artificial intelligence "
        "that lets systems learn from data"
    )

    similarity = semantic_similarity(response, expected)
    assert similarity > 0.75, (
        f"Semantic similarity {similarity:.2f} < threshold 0.75. "
        f"Response: {response[:200]}"
    )

Each assertion makes a call to the embeddings API. Use it selectively for queries where the content matters more than the structure.

Strategy 4: LLM-as-judge

Use an LLM to evaluate whether the agent's response is correct:

from langchain_openai import ChatOpenAI

judge = ChatOpenAI(model="gpt-4o-mini", temperature=0)

def llm_judges_response(query: str, response: str, criteria: str) -> bool:
    """Uses an LLM to evaluate whether the response meets the criterion."""
    judgment = judge.invoke([{
        "role": "system",
        "content": (
            "Evaluate whether the response meets the criterion. "
            "Answer ONLY 'PASS' or 'FAIL'."
        )
    }, {
        "role": "user",
        "content": (
            f"Query: {query}\n"
            f"Agent's response: {response}\n"
            f"Criterion: {criteria}"
        )
    }])
    return "PASS" in judgment.content.upper()

@pytest.mark.integration
def test_response_quality_with_judge(run_agent):
    query = "What are the advantages of using Python for data science?"
    result = run_agent(query)
    response = result["messages"][-1].content

    assert llm_judges_response(
        query, response,
        "The response mentions at least 3 concrete advantages of Python "
        "for data science (e.g. libraries, community, ease of use)"
    ), f"The LLM judge failed. Response: {response[:300]}"

LLM-as-judge is powerful but expensive (two LLM calls per test). Reserve this strategy for quality validations you can't express with regex or semantic similarity.

When to use each strategy

The recommendation: every test should have at least one structural assertion (did it finish? did it use tools?) and optionally a content one.

Snapshot Testing of Trajectories

What a trajectory is

A trajectory is the complete sequence of the agent's actions: which tools it called, with what arguments, in what order. It's the "path" to the result.

def extract_trajectory(messages: list) -> list[dict]:
    """Extracts the trajectory from an agent run."""
    trajectory = []
    for msg in messages:
        if hasattr(msg, "tool_calls") and msg.tool_calls:
            for tc in msg.tool_calls:
                trajectory.append({
                    "type": "tool_call",
                    "tool": tc["name"],
                    "args_keys": sorted(tc["args"].keys()),
                })
        elif hasattr(msg, "type") and msg.type == "tool":
            trajectory.append({
                "type": "tool_response",
                "tool": msg.name,
                "has_content": bool(msg.content),
            })
    return trajectory

extract_trajectory captures structure (which tool, which argument keys), not exact values. That makes the comparisons stable against LLM variation.

Snapshot testing with pytest-snapshot

pip install pytest-snapshot

# tests/integration/test_snapshots.py
import json
import pytest

@pytest.mark.integration
def test_weather_trajectory_snapshot(run_agent, snapshot):
    """The trajectory for the weather query must be stable."""
    result = run_agent("What's the weather in Madrid?")
    trajectory = extract_trajectory(result["messages"])

    snapshot.assert_match(
        json.dumps(trajectory, indent=2, ensure_ascii=False),
        "weather_query_trajectory.json"
    )

The first time, pytest-snapshot saves the trajectory as a reference. Subsequent runs compare against that snapshot. If it changes, the test fails. To update snapshots intentionally: pytest --snapshot-update -m integration.

Flexible snapshots for non-determinism

Exact snapshots break on intentional changes (you update the prompt) or non-determinism (the LLM took a different but valid path). To handle non-determinism, use "flexible" snapshots that check properties instead of exact equality:

def assert_trajectory_structure(
    trajectory: list[dict],
    required_tools: list[str],
    min_steps: int = 1,
    max_steps: int = 10
):
    """Checks the trajectory's properties without requiring an exact sequence."""
    tools_used = [
        step["tool"] for step in trajectory
        if step["type"] == "tool_call"
    ]

    assert min_steps <= len(tools_used) <= max_steps, (
        f"Tool calls: {len(tools_used)}, expected between {min_steps} and {max_steps}"
    )

    for required in required_tools:
        assert required in tools_used, (
            f"Tool '{required}' was not used. Tools used: {tools_used}"
        )

@pytest.mark.integration
def test_weather_uses_correct_tools(run_agent):
    result = run_agent("What's the weather in Tokyo?")
    trajectory = extract_trajectory(result["messages"])
    assert_trajectory_structure(
        trajectory,
        required_tools=["get_weather"],
        min_steps=1,
        max_steps=3
    )

Costs and Strategies

The real cost of integration tests

20 integration tests with gpt-4o-mini cost ~$0.05 per run. That's not expensive, but it isn't free either — you need a strategy.

Layered execution strategy

Use a cheap model for CI

Your agent can use gpt-4o in production, but the CI integration tests can use gpt-4o-mini. Integration tests verify structure, not response quality — if the agent with gpt-4o-mini picks the right tools, it will with gpt-4o too.

TEST_MODEL=gpt-4o-mini pytest -m integration   # CI: fast and cheap
TEST_MODEL=gpt-4o pytest -m integration         # Nightly: the production model

Reducing flakiness

Integration tests with a real LLM are inherently flaky. Strategies for reducing the variability:

1. temperature=0 — Reduces (doesn't eliminate) the variability.
2. Broad assertions — "contains X" instead of "is exactly X".
3. Retry in CI — Use pytest-rerunfailures to retry flaky tests.
4. Stable seeds — Simple, direct queries produce more predictable answers.
5. Aggressive timeout — Kill tests that take too long (loops).

pip install pytest-rerunfailures

pytest -m integration --reruns 2 --reruns-delay 5

With --reruns 2, a failing test is retried up to 2 times with 5 seconds between attempts. If it passes on the second attempt, it's marked as "rerun passed" (not as a failure).

Comparison: Unit vs Integration

The rule: many unit tests (50+), few integration tests (10-20). Unit tests are the foundation. Integration tests are the final validation.

Connection to the Project

In the module's project (capsule 08), you're going to build a testing suite for the Research Agent. The integration tests you implement here apply directly:

1. 5 test queries — The Research Agent runs against 5 predefined queries. Each one checks a different aspect: simple search, multi-tool, ambiguous query, a query that should fail gracefully, a complex query that requires multiple steps.

2. Trajectory snapshots — Each query generates a trajectory that's saved as a snapshot. When you change the supervisor's prompt or modify a tool, the snapshots tell you exactly what changed in the agent's behavior.

3. Mixed assertions — Each test combines structural assertions (did it use tools? did it finish?) with flexible content assertions (does the response mention key concepts?).

4. CI configuration — The integration tests run with gpt-4o-mini in CI, and with gpt-4o in the nightly build.

Troubleshooting

"My integration tests pass locally but fail in CI"

Common causes: the API key isn't configured in the CI secrets, rate limiting (use --workers 1), an overly aggressive timeout (CI is slower — raise it by 50%), or a different model version (use snapshots like gpt-4o-2024-08-06).

"A test passes 4 out of 5 times"

The test is flaky. Make the assertion less strict, add --reruns 2, or investigate what's varying (tool selection, content, number of steps) and adjust the assertion. If the variation is in tool selection, the prompt needs to be more explicit.

"The snapshots break after every prompt change"

That's expected. Use assert_trajectory_structure (flexible snapshots) instead of exact equality, or accept that updating snapshots is part of the workflow: change the prompt → run the tests → review the diff → update the snapshots → commit.

"Integration tests take 5+ minutes"

Use gpt-4o-mini (faster), cut down to 10-15 tests for CI, use scope="session" in the fixtures, and parallelize with pytest-xdist: pytest -n 4 -m integration.

"How do I test an agent that depends on external APIs?"

Mock the external APIs (with responses or httpx_mock) but not the LLM. That tests the LLM's decision without depending on external services. Alternatively, use APIs in sandbox/test mode.

Exercises

Exercise 1: A basic integration test (Easy)

Write an integration test that verifies: (a) the agent produces a non-empty final response, (b) the response is more than 50 characters, (c) the agent finishes in less than 20 seconds. Use the query "What is Python and what is it used for?".

import pytest
from langchain_core.messages import AIMessage

@pytest.mark.integration
@pytest.mark.timeout(20)
def test_basic_agent_response(run_agent):
    result = run_agent("What is Python and what is it used for?")
    messages = result["messages"]
    final = messages[-1]

    assert isinstance(final, AIMessage), (
        f"The last message must be an AIMessage, got {type(final).__name__}"
    )
    assert final.content, "The response must not be empty"
    assert len(final.content) > 50, (
        f"The response is too short ({len(final.content)} chars): {final.content}"
    )
    assert not final.tool_calls, (
        "The final response must not have pending tool_calls"
    )

Exercise 2: A tool-selection test (Medium)

Create an integration test with parametrize that verifies the agent picks the right tool: a query for search_web, one for calculator, and one that shouldn't use tools.

import pytest
from langchain_core.messages import ToolMessage

@pytest.mark.integration
@pytest.mark.parametrize("query,expected_tool,should_use_tool", [
    ("Look up the latest news about AI", "search_web", True),
    ("Calculate 145 × 23 + 89", "calculator", True),
    ("Tell me a joke", None, False),
])
def test_tool_selection(run_agent, query, expected_tool, should_use_tool):
    result = run_agent(query)
    messages = result["messages"]

    tool_calls = []
    for m in messages:
        if hasattr(m, "tool_calls") and m.tool_calls:
            tool_calls.extend(m.tool_calls)

    if should_use_tool:
        assert len(tool_calls) > 0, (
            f"Query '{query}' should use tools, but it used none"
        )
        tools_used = [tc["name"] for tc in tool_calls]
        assert expected_tool in tools_used, (
            f"Expected '{expected_tool}' in {tools_used}"
        )
    else:
        assert len(tool_calls) == 0, (
            f"Query '{query}' should not use tools, "
            f"but it used: {[tc['name'] for tc in tool_calls]}"
        )

Exercise 3: Manual snapshot testing (Medium)

Implement snapshot testing without external libraries. Create save_snapshot and load_snapshot functions, and a test that extracts the trajectory, compares it against the saved snapshot, and creates the snapshot if it doesn't exist.

import json
from pathlib import Path
from langchain_core.messages import ToolMessage

SNAPSHOT_DIR = Path(__file__).parent / "snapshots"
SNAPSHOT_DIR.mkdir(exist_ok=True)

def extract_trajectory(messages):
    trajectory = []
    for msg in messages:
        if hasattr(msg, "tool_calls") and msg.tool_calls:
            for tc in msg.tool_calls:
                trajectory.append({"type": "tool_call", "tool": tc["name"],
                                   "args_keys": sorted(tc["args"].keys())})
        elif isinstance(msg, ToolMessage):
            trajectory.append({"type": "tool_response", "tool": msg.name,
                               "has_content": bool(msg.content)})
    return trajectory

def save_snapshot(name, data):
    (SNAPSHOT_DIR / f"{name}.json").write_text(json.dumps(data, indent=2, ensure_ascii=False))

def load_snapshot(name):
    path = SNAPSHOT_DIR / f"{name}.json"
    return json.loads(path.read_text()) if path.exists() else None

@pytest.mark.integration
def test_trajectory_snapshot(run_agent):
    result = run_agent("What is the capital of Germany?")
    trajectory = extract_trajectory(result["messages"])
    existing = load_snapshot("capital_query")
    if existing is None:
        save_snapshot("capital_query", trajectory)
        pytest.skip("Snapshot created for the first time")
    assert trajectory == existing, f"The trajectory changed vs the snapshot"

Exercise 4: Semantic similarity assertions (Hard)

Create assert_semantically_similar(actual, expected, threshold=0.8) that uses embeddings to compare two texts. Write an integration test that checks semantic similarity against an expected response.

import numpy as np
from langchain_openai import OpenAIEmbeddings

embeddings_model = OpenAIEmbeddings(model="text-embedding-3-small")

def assert_semantically_similar(actual: str, expected: str, threshold: float = 0.8):
    vectors = embeddings_model.embed_documents([actual, expected])
    vec_a, vec_b = np.array(vectors[0]), np.array(vectors[1])
    similarity = np.dot(vec_a, vec_b) / (np.linalg.norm(vec_a) * np.linalg.norm(vec_b))
    assert similarity >= threshold, (
        f"Similarity: {similarity:.3f} < {threshold}\n"
        f"Actual: {actual[:200]}...\nExpected: {expected[:200]}..."
    )

@pytest.mark.integration
def test_agent_gives_semantically_correct_answer(run_agent):
    result = run_agent("Briefly explain what a REST API is")
    response = result["messages"][-1].content
    expected = ("A REST API is a programming interface that follows "
                "the REST principles, using HTTP methods to communicate between services")
    assert_semantically_similar(response, expected, threshold=0.75)

Exercise 5: A complete CI suite (Hard)

Create a complete conftest.py that includes: (a) an LLM fixture with a model configurable via the TEST_MODEL env var, (b) an agent fixture with scope="session", (c) an auto-skip of integration tests if there's no API key.

# tests/conftest.py
import os, pytest
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage

@pytest.fixture(scope="session")
def api_key():
    key = os.environ.get("OPENAI_API_KEY")
    if not key: pytest.skip("OPENAI_API_KEY is not configured")
    return key

@pytest.fixture(scope="session")
def llm(api_key):
    model = os.environ.get("TEST_MODEL", "gpt-4o-mini")
    return ChatOpenAI(model=model, temperature=0, api_key=api_key, max_retries=2)

@pytest.fixture(scope="session")
def agent_app(llm):
    from my_agent.graph import build_agent
    return build_agent(llm)

@pytest.fixture
def run_agent(agent_app):
    def _run(query: str):
        return agent_app.invoke({"messages": [HumanMessage(content=query)]},
                                config={"configurable": {"thread_id": "test"}})
    return _run

@pytest.fixture(autouse=True)
def skip_integration_without_key(request):
    markers = {m.name for m in request.node.iter_markers()}
    if markers & {"integration", "costly"} and not os.environ.get("OPENAI_API_KEY"):
        pytest.skip("Requires OPENAI_API_KEY")

TEST_MODEL=gpt-4o-mini pytest -m "unit or integration" --timeout=60 --reruns 2

Summary

In this capsule you implemented integration tests for agents — the validation layer that verifies the system really works, not just in theory:

• Unit tests aren't enough. Mocking the LLM and the tools verifies isolated components, but not the interaction between them. Integration tests run the complete agent with a real LLM.
• Non-deterministic assertions require different strategies: structural (type, length), flexible content (keywords), semantic similarity (embeddings), and LLM-as-judge. Combine structural + one of the others.
• Snapshot testing captures the trajectory (the sequence of tool calls) as a reference. Prompt changes change the trajectory — the snapshots tell you exactly what changed.
• Integration tests cost money. Use gpt-4o-mini for CI, gpt-4o for nightly. Structure your testing in layers: unit (every commit), integration (every PR), the full suite (release).
• Flakiness is inevitable but manageable: temperature=0, broad assertions, retry in CI, simple queries.

Next capsule: Trajectory evaluation — going beyond "is the result correct?" to evaluate "was the path correct?" Did it use the right tools? In the optimal order? With the right arguments?

Additional Resources

1. pytest — Official documentation — A complete reference for fixtures, markers, and parametrize
2. pytest-timeout — A plugin for timing out individual tests
3. pytest-rerunfailures — A plugin for retrying flaky tests automatically
4. LangSmith — Testing & Evaluation — LangChain's platform for evaluating agents with datasets and metrics
5. Testing LLM Applications — DeepLearning.AI — A course on testing strategies for LLM applications