Module 3: Structured Outputs and System Prompts

1. Introduction: Predictable Outputs as a Production Requirement

Next →

Overview

In production, an LLM's output has to be parseable by code. "Free text" doesn't cut it when your application needs to extract data, validate answers or integrate with downstream systems. In this module you'll learn JSON mode, function calling, schemas with Pydantic, system prompt design patterns, guardrails and prompt templates to get structured outputs reliably.

Why it matters: Without structured output, every response is a snowflake: unique, unpredictable, and potentially broken. With structured output, you have a contract: the model returns data your code can consume without fragile parsing or regex. The difference between a prototype and real production is almost always in how you handle the output.

The Free-Text Problem in Production

What breaks

# Scenario: extracting invoice data for an accounting system

# Without structure — what the model might return:
answer_1 = "The invoice totals $1,299 USD, issued on January 15."
answer_2 = "Total: USD 1,299.00\nDate: January 15, 2025"
answer_3 = "1299 dollars, january 2025"
answer_4 = "{'total': '$1,299', 'date': '01/15/2025'}"  # Single quotes

# What your code expects:
# {"total": 1299.0, "currency": "USD", "date": "2025-01-15"}

# None of the answers above is directly usable with json.loads()

Consequences:

  • Fragile regex that breaks on the smallest variation
  • Manual parsing that grows in complexity with every edge case
  • Tests that fail when you update the model
  • Production incidents at 3am

The real cost

# Without structured output: typical parsing code in production
def parse_invoice_total(text: str) -> float:
    # Attempt 1: JSON
    try:
        data = json.loads(text)
        return float(data.get("total", data.get("amount", data.get("value", 0))))
    except json.JSONDecodeError:
        pass
    
    # Attempt 2: regex for amounts
    patterns = [
        r'\$\s*([\d,]+\.?\d*)',
        r'total[:\s]*([\d,]+\.?\d*)',
        r'USD\s*([\d,]+\.?\d*)',
        r'([\d,]+\.?\d*)\s*(?:USD|MXN|EUR|dollars)',
    ]
    for p in patterns:
        m = re.search(p, text, re.IGNORECASE)
        if m:
            return float(m.group(1).replace(",", ""))
    
    # Fallback: grab any large number
    nums = re.findall(r'[\d,]+\.?\d+', text)
    if nums:
        return max(float(n.replace(",", "")) for n in nums)
    
    raise ValueError(f"Could not parse total from: {text[:100]}")

# This code is 25 lines, it's fragile, and it fails on edge cases
# With structured output: 1 line
# total = json.loads(answer)["total"]  # Always works

Structured Output as an API Contract

A structured output defines a contract between the LLM and your code:

from pydantic import BaseModel, Field
from typing import Literal, Optional
from datetime import date

# The contract
class InvoiceData(BaseModel):
    vendor: str
    invoice_number: str
    issue_date: str  # YYYY-MM-DD
    total: float
    currency: Literal["USD", "EUR", "MXN", "GBP"]
    items: list[dict]
    
    class Config:
        # Allows extra fields from the model but ignores them
        extra = "ignore"

# Your system knows exactly what to expect:
# - vendor is always str
# - total is always float
# - currency is always one of 4 possible values
# If the LLM returns something different → ValidationError → handleable, not silent

Analogy: Contract vs Free Text

AspectNo contract (free text)With contract (structured)
ParsingFragile regex, edge casesjson.loads() + Pydantic
ErrorsSilent (wrong value)Explicit (ValidationError)
TestingHard to mockMockable with fixtures
MaintenanceGrows with every variationChange the schema
IntegrationOne parser per systemShared schema

The 5 Structured Output Mechanisms

This module covers the 5 main mechanisms:

1. JSON Mode (capsule 02)

# OpenAI: guarantees valid JSON
response_format={"type": "json_object"}

For: Any JSON output. Simple and effective.

2. JSON Schema / Structured Outputs (capsule 02)

# OpenAI: guarantees a strict schema
response_format={"type": "json_schema", "json_schema": {...}}

For: When you need an exact schema with strict types.

3. Function Calling / Tool Use (capsule 03)

# OpenAI: the model calls a function with typed arguments
tools=[{"type": "function", "function": {"name": "...", "parameters": {...}}}]
# Anthropic: equivalent with tool_choice

For: Integration with real code, agents, guaranteed structured output.

4. Pydantic Integration (capsule 03)

# Define the schema with Pydantic, generate JSON Schema automatically
schema = MyModel.model_json_schema()

For: Output validation, type coercion, automatic documentation.

5. Prompt Engineering for Structure (capsules 04-06)

# System prompt design, guardrails, templates
system = "You are a data extractor. ALWAYS return JSON with schema X."

For: When JSON mode isn't available, or to reinforce structure.

When to Use Each Mechanism

Do you need a 100% guaranteed schema?
├── YES → Function Calling or JSON Schema (response_format with json_schema)
└── NO  → JSON mode (response_format: json_object) is enough

Does the system need to run real code (not just parse)?
├── YES → Function Calling / Tool Use
└── NO  → JSON mode + Pydantic

Multi-provider (OpenAI + Anthropic)?
├── YES → Prompt engineering + robust parsing (you can't depend on proprietary features)
└── NO  → Use the provider's native mechanism

Do you need type validation and coercion?
├── YES → Pydantic (always, whatever the mechanism)
└── NO  → json.loads() may be enough

Module Setup

# requirements.txt for this module
# openai>=1.0.0
# anthropic>=0.25.0
# python-dotenv>=1.0.0
# pydantic>=2.0.0

from openai import OpenAI
import anthropic
from pydantic import BaseModel
import json
import os
from dotenv import load_dotenv

load_dotenv()

oai_client = OpenAI()
ant_client = anthropic.Anthropic()

def verify_setup():
    """Verifies that the setup is correct."""
    # Test OpenAI
    r = oai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Say exactly: JSON_OK"}],
        temperature=0,
        max_tokens=10
    )
    assert "JSON_OK" in r.choices[0].message.content, "OpenAI setup is wrong"
    print("✅ OpenAI: OK")
    
    # Test Anthropic
    r = ant_client.messages.create(
        model="claude-3-5-haiku-20241022",
        messages=[{"role": "user", "content": "Say exactly: JSON_OK"}],
        temperature=0,
        max_tokens=10
    )
    assert "JSON_OK" in r.content[0].text, "Anthropic setup is wrong"
    print("✅ Anthropic: OK")
    
    # Test Pydantic
    class TestModel(BaseModel):
        field: str
    t = TestModel(field="test")
    assert t.field == "test"
    print("✅ Pydantic: OK")

verify_setup()

Module 3 Roadmap

#CapsuleWhat you'll see
01Introduction (this one)Why structured output, the 5 mechanisms, setup
02JSON mode and response formatOpenAI JSON mode, JSON Schema, Anthropic, retry
03Pydantic and function callingPydantic schemas, tools, validation, nested structures
04System prompt design patternsExpert, Analyst, Formatter, Guardian — 4 base patterns
05Guardrails and safetyPrompt injection, validation, content filtering
06Prompt templates and variablesf-strings, Jinja2, modular composition
07Multi-model structured outputAdapter patterns, fallback between providers
08Project: Structured Data ExtractorExtracting invoices, emails, articles with Pydantic schemas

Estimated duration: 1.25-1.5 hrs for the full module.

Connection to the Project

In the Structured Data Extractor (capsule 08) you'll build:

  • Invoice data extraction with the schema InvoiceData(vendor, amount, date, items)
  • Email extraction with the schema EmailData(sender, subject, required_actions)
  • Article extraction with the schema ArticleData(title, summary, key_points, tone)
  • Retry logic for malformed outputs with feedback to the model
  • Automatic fallback between OpenAI and Anthropic
  • Accuracy and cost comparison across mechanisms

Summary

  • Problem: Free text isn't parseable, integrable, or testable in production
  • Solution: Structured output as a contract: defined schema, validated types, explicit errors
  • 5 mechanisms: JSON mode, JSON Schema, Function Calling, Pydantic, System Prompt Engineering
  • When to use each: Guaranteed schema → tools; multi-provider → prompt engineering; always → Pydantic to validate
  • In this module: Each mechanism in depth + an integrative project

From Free Text to a Production System: A Real Transformation

To grasp the full impact, let's look at the same functionality with and without structured output, in a scenario that extracts data from vendor emails:

Version without structured output (starting point)

import re
from openai import OpenAI

client = OpenAI()

def extract_email_data_unstructured(email_text: str) -> dict:
    """
    Version without structured output — fragile, hard to maintain.
    """
    prompt = f"""
Read this email and tell me:
1. The vendor
2. The total amount
3. The due date
4. Whether there's any urgent action

Email:
{email_text}
"""
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0
    )
    
    answer_text = response.choices[0].message.content
    
    # Now you get to parse free text... good luck
    data = {}
    
    # Try to extract the vendor (fragile pattern)
    if "vendor:" in answer_text.lower():
        line = [l for l in answer_text.split("\n") if "vendor" in l.lower()]
        if line:
            data["vendor"] = line[0].split(":")[-1].strip()
    
    # Try to extract the amount (many possible formats)
    amount_patterns = [
        r'\$\s*([\d,]+(?:\.\d{2})?)',
        r'([\d,]+(?:\.\d{2})?)\s*(?:USD|MXN|EUR)',
        r'amount[:\s]+([\d,]+(?:\.\d{2})?)',
        r'total[:\s]+([\d,]+(?:\.\d{2})?)',
    ]
    for pattern in amount_patterns:
        match = re.search(pattern, answer_text, re.IGNORECASE)
        if match:
            data["amount"] = float(match.group(1).replace(",", ""))
            break
    
    # Try to extract the date (totally variable format)
    date_patterns = [
        r'\d{4}-\d{2}-\d{2}',
        r'\d{1,2}/\d{1,2}/\d{4}',
        r'\w+\s+\d{1,2},\s+\d{4}',
    ]
    for pattern in date_patterns:
        match = re.search(pattern, answer_text)
        if match:
            data["due_date"] = match.group(0)
            break
    
    # Urgent: look for keywords
    urgent_keywords = ["urgent", "immediate", "today", "overdue", "deadline"]
    data["has_urgency"] = any(k in answer_text.lower() for k in urgent_keywords)
    
    return data  # May be incomplete, mixed-format, or plain wrong

# PROBLEM: This code is 50 lines, it's fragile, and it fails silently.
# If the model changes its answer format → parsing breaks with no error.

Version with structured output (production)

from pydantic import BaseModel, Field
from typing import Optional, Literal
import json

class EmailData(BaseModel):
    vendor: str
    amount: Optional[float] = None
    currency: Literal["USD", "EUR", "MXN", "GBP", "UNKNOWN"] = "UNKNOWN"
    due_date: Optional[str] = None  # YYYY-MM-DD
    has_urgency: bool = False
    required_actions: list[str] = Field(default_factory=list)
    summary: str

def extract_email_data_structured(email_text: str) -> EmailData:
    """
    Version with structured output — robust, maintainable, testable.
    """
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": """
You are a data extractor for vendor emails.
Extract the requested data and return ONLY valid JSON with the given schema.
If a field isn't available, use null for the optional ones.
Dates always in YYYY-MM-DD format.
"""
            },
            {
                "role": "user",
                "content": f"""
Extract the data from this email as JSON with this exact schema:
{{
  "vendor": "vendor name",
  "amount": null or float,
  "currency": "USD|EUR|MXN|GBP|UNKNOWN",
  "due_date": null or "YYYY-MM-DD",
  "has_urgency": true or false,
  "required_actions": ["action 1", "action 2"],
  "summary": "1-sentence summary"
}}

Email:
{email_text}
"""
            }
        ],
        response_format={"type": "json_object"},  # JSON mode
        temperature=0
    )
    
    data = json.loads(response.choices[0].message.content)
    return EmailData(**data)  # Pydantic validates and coerces types

# ADVANTAGES:
# - 1 line to parse: json.loads() + EmailData(**data)
# - Explicit ValidationError if the schema isn't respected
# - Correct types guaranteed (amount is a float, not a string)
# - Testable with fixtures: EmailData(vendor="X", amount=100.0, ...)

# Test with a real email
sample_email = """
From: invoices@acmecorp.com
Subject: Invoice #INV-2025-0342 - Urgent Due Date

Dear team,

Invoice #INV-2025-0342 for USD 3,450.00 is due on January 15, 2025.
Immediate payment is required to avoid late fees.

Best regards,
ACME Corp
"""

result = extract_email_data_structured(sample_email)
print(f"Vendor: {result.vendor}")
print(f"Amount: {result.amount} {result.currency}")
print(f"Due date: {result.due_date}")
print(f"Urgent: {result.has_urgency}")
print(f"Actions: {result.required_actions}")
# Output:
# Vendor: ACME Corp
# Amount: 3450.0 USD
# Due date: 2025-01-15
# Urgent: True
# Actions: ['Process immediate payment', 'Confirm receipt of invoice']

Key difference: 50 lines of fragile regex → 5 lines of clean code. And if the model produces malformed JSON, you get an explicit ValidationError, not silently wrong data.

Evolution: From Prototype to Production

This diagram shows how output handling evolves:

STAGE 1 — Prototype (days 1-7)
───────────────────────────────
  simple prompt → free text → read it by hand
  ✓ Works for demos and exploration
  ✗ Not scalable, not testable

STAGE 2 — First system (weeks 2-4)
───────────────────────────────────────
  prompt with a format instruction → regex parsing
  ✓ Works for common cases
  ✗ Fails on edge cases, hard to maintain

STAGE 3 — Robust system (month 2+)
────────────────────────────────────
  JSON mode / function calling + Pydantic + retry logic
  ✓ Guaranteed parseable
  ✓ Validated types
  ✓ Explicit errors
  ✓ Testable with fixtures
  ✓ Compatible with multiple providers

STAGE 4 — Full production (month 3+)
────────────────────────────────────
  + Multi-provider fallback
  + Guardrails (content filtering, injection detection)
  + Versioned templates
  + Schema-violation monitoring
  + A/B testing of prompts

This module takes you from Stage 2 to Stage 4.

Why Each Mechanism Exists

Each of the module's 5 mechanisms exists because it solves a specific problem:

MechanismProblem it solvesCapsule
JSON ModeFree text that won't parse02
JSON Schema / Structured OutputsJSON that doesn't follow the expected schema02
Function CallingGuaranteed schema + integration with real code03
PydanticCorrect JSON but wrong types (str vs float)03
System Prompt PatternsInconsistent behavior across requests04
GuardrailsDangerous or out-of-domain outputs05
Prompt TemplatesDuplicated prompts, hard to version06
Multi-providerLock-in to one provider, no fallback07

Further resources

  1. OpenAI Structured Outputs Guide — Comparison between JSON mode and json_schema
  2. Anthropic Structured Output — Tool use for strict schemas in Claude
  3. Pydantic v2 Docs — BaseModel, validators, JSON Schema generation
  4. Function Calling Guide (OpenAI) — Tool use, parallel tool calls, tool choice
  5. JSON Schema Spec — Understand the schemas used in response_format and tools
← Previous8. Project: Few-Shot Classification SystemNext →2. JSON Mode and Response Format