2. JSON Mode and Response Format

Overview

OpenAI offers a JSON mode that forces the model to return valid JSON. Anthropic offers structured output with JSON Schema. In this capsule you'll learn to use both mechanisms, the difference between "valid JSON" and "correct schema", how to handle parsing errors, and robust retry strategies for when the output fails.

Why it matters: In production, json.loads() throwing an exception at 3am is an incident. JSON mode removes the parsing problem in OpenAI, but it doesn't guarantee the schema is correct. You need both: valid JSON + schema validation. This capsule covers how to get both, reliably.

The Problem: Free Text vs Structured JSON

Without any structured output mechanism, the model can give you:

# What you asked for
{"sentiment": "positive", "confidence": 0.95}

# What you might get back
"The sentiment is positive with a confidence of about 95%."
"Sentiment: positive\nConfidence: 0.95"
```json
{"sentiment": "positive", "confidence": 0.95}

{"sentiment": "positive", "confidence": .95} # Invalid JSON: .95 without the 0 {"Sentiment": "Positive", "Confidence": "0.95"} # Keys with different capitalization

**Without JSON mode:** You need regex, fragile parsing, and handling for multiple formats.
**With JSON mode:** The model guarantees syntactically valid JSON. You validate the schema.

---

## OpenAI: JSON Mode with `response_format`

### Basic usage

```python
from openai import OpenAI
import json

client = OpenAI()

def extract_entities(text: str) -> dict:
    """
    Extracts entities using OpenAI's JSON mode.
    Guarantees valid JSON in the answer.
    """
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": """
                Extract named entities from the text.
                Return JSON with this schema:
                {"persons": ["..."], "organizations": ["..."], "locations": ["..."]}
                
                If there are no entities of a given type, use an empty list.
                """
            },
            {"role": "user", "content": text}
        ],
        response_format={"type": "json_object"},  # Turns on JSON mode
        temperature=0
    )
    
    return json.loads(response.choices[0].message.content)

# Test
result = extract_entities("Maria Garcia works at Google in Mountain View.")
print(result)
# {"persons": ["Maria Garcia"], "organizations": ["Google"], "locations": ["Mountain View"]}

Limitations of basic JSON mode

# JSON mode guarantees valid JSON, but it does NOT guarantee the correct schema

# The model could return:
{
    "names": ["Maria Garcia"],     # Different key (names vs persons)
    "companies": ["Google"],       # A synonym, not the key you asked for
    "places": ["Mountain View"]    # Different key
}

# Or add keys you never asked for:
{
    "persons": ["Maria Garcia"],
    "organizations": ["Google"],
    "locations": ["Mountain View"],
    "date": null,               # Extra key, not requested
    "total_entities": 3         # Extra key, not requested
}

For a strict schema: Use Structured Outputs with a JSON schema (see below) or function calling (capsule 03).

OpenAI: Structured Outputs with a Strict Schema

OpenAI offers Structured Outputs that validate against a specific JSON Schema:

from openai import OpenAI
import json

client = OpenAI()

def extract_with_schema(text: str) -> dict:
    """
    Extracts entities with a strictly defined schema.
    """
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": "Extract named entities from the text."
            },
            {"role": "user", "content": text}
        ],
        response_format={
            "type": "json_schema",
            "json_schema": {
                "name": "entity_extraction",
                "strict": True,  # Strict schema
                "schema": {
                    "type": "object",
                    "properties": {
                        "persons": {
                            "type": "array",
                            "items": {"type": "string"},
                            "description": "Proper names of people"
                        },
                        "organizations": {
                            "type": "array",
                            "items": {"type": "string"},
                            "description": "Names of companies, organizations"
                        },
                        "locations": {
                            "type": "array",
                            "items": {"type": "string"},
                            "description": "Cities, countries, places"
                        }
                    },
                    "required": ["persons", "organizations", "locations"],
                    "additionalProperties": False  # No extra keys allowed
                }
            }
        },
        temperature=0
    )
    
    return json.loads(response.choices[0].message.content)

# The output will always have exactly persons, organizations, locations
result = extract_with_schema("Apple was founded in Cupertino by Steve Jobs.")
print(result)
# {"persons": ["Steve Jobs"], "organizations": ["Apple"], "locations": ["Cupertino"]}

Anthropic: Structured Output with JSON Schema

Anthropic offers native structured output from the messages API:

import anthropic

client = anthropic.Anthropic()

def extract_anthropic(text: str) -> dict:
    """
    Extracts entities with a strict schema using Anthropic.
    """
    response = client.messages.create(
        model="claude-3-5-haiku-20241022",
        max_tokens=500,
        messages=[
            {
                "role": "user",
                "content": f"Extract the named entities from: '{text}'"
            }
        ],
        tools=[
            {
                "name": "extract_entities",
                "description": "Extracts named entities from the text",
                "input_schema": {
                    "type": "object",
                    "properties": {
                        "persons": {
                            "type": "array",
                            "items": {"type": "string"},
                            "description": "Names of people"
                        },
                        "organizations": {
                            "type": "array",
                            "items": {"type": "string"}
                        },
                        "locations": {
                            "type": "array",
                            "items": {"type": "string"}
                        }
                    },
                    "required": ["persons", "organizations", "locations"]
                }
            }
        ],
        tool_choice={"type": "auto"}
    )
    
    # Pull the result out of the tool use
    for block in response.content:
        if block.type == "tool_use":
            return block.input
    
    raise ValueError("No tool use response was returned")

result = extract_anthropic("Maria Garcia works at Google in Madrid.")
print(result)

The Portable Alternative: Explicit Instructions + Robust Parsing

For projects that need to run across multiple providers without JSON mode:

from openai import OpenAI
import anthropic
import json
import re
from typing import Literal

oai_client = OpenAI()
ant_client = anthropic.Anthropic()

def clean_json(text: str) -> str:
    """
    Extracts JSON from an answer that may carry extra text.
    """
    text = text.strip()
    
    # Case 1: pure JSON
    if text.startswith('{') or text.startswith('['):
        return text
    
    # Case 2: wrapped in ```json ... ``` or ``` ... ```
    match = re.search(r'```(?:json)?\s*([\s\S]*?)\s*```', text)
    if match:
        return match.group(1).strip()
    
    # Case 3: JSON embedded in text
    match = re.search(r'\{[\s\S]*?\}', text)
    if match:
        return match.group()
    
    raise ValueError(f"No JSON found in: {text[:100]}")

SYSTEM_PORTABLE = """
Analyze the text and return ONLY valid JSON with this exact schema:
{"sentiment": "POSITIVE|NEGATIVE|NEUTRAL", "confidence": 0.0-1.0, "keywords": ["...", "..."]}

Do NOT include any text before or after the JSON.
Correct example: {"sentiment": "POSITIVE", "confidence": 0.92, "keywords": ["excellent", "fast"]}
"""

def analyze_portable(text: str, provider: Literal["openai", "anthropic"] = "openai") -> dict:
    """
    Works with both providers without depending on JSON mode.
    """
    if provider == "openai":
        r = oai_client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": SYSTEM_PORTABLE},
                {"role": "user", "content": text}
            ],
            response_format={"type": "json_object"},  # JSON mode for OpenAI
            temperature=0
        )
        return json.loads(r.choices[0].message.content)
    
    elif provider == "anthropic":
        r = ant_client.messages.create(
            model="claude-3-5-haiku-20241022",
            system=SYSTEM_PORTABLE,
            messages=[{"role": "user", "content": text}],
            temperature=0,
            max_tokens=200
        )
        raw = r.content[0].text
        return json.loads(clean_json(raw))

# Test with both
for provider in ["openai", "anthropic"]:
    result = analyze_portable("The service was fast and the product excellent.", provider)
    print(f"{provider}: {result}")

Error Handling and Retry

Retry pattern with feedback

from openai import OpenAI
import json
import time

client = OpenAI()

def extract_with_retry(
    system_prompt: str,
    text: str,
    max_retries: int = 3,
    delay_base: float = 1.0
) -> dict:
    """
    Extracts JSON with exponential retry when parsing fails.
    
    Strategy:
    - Attempt 1: JSON mode on
    - Attempt 2: add feedback about the error
    - Attempt 3: simplify the prompt
    """
    last_error = None
    
    for attempt in range(max_retries):
        try:
            if attempt == 0:
                # Normal attempt
                messages = [
                    {"role": "system", "content": system_prompt},
                    {"role": "user", "content": text}
                ]
            elif attempt == 1:
                # Add feedback about the previous error
                messages = [
                    {"role": "system", "content": system_prompt},
                    {"role": "user", "content": text},
                    {"role": "assistant", "content": f"[Previous answer that failed: {last_error}]"},
                    {"role": "user", "content": "The previous answer wasn't valid JSON. Return ONLY the JSON, with no extra text."}
                ]
            else:
                # Simplify: ask for the bare minimum
                messages = [
                    {
                        "role": "user",
                        "content": f"""
I need valid JSON. Answer with the JSON only, nothing else.
Schema: {system_prompt[:200]}
Input: {text[:200]}
"""
                    }
                ]
            
            response = client.chat.completions.create(
                model="gpt-4o-mini",
                messages=messages,
                response_format={"type": "json_object"},
                temperature=0,
                max_tokens=500
            )
            
            content = response.choices[0].message.content
            result = json.loads(content)
            
            if attempt > 0:
                print(f"Success on attempt {attempt + 1}")
            
            return result
            
        except json.JSONDecodeError as e:
            last_error = str(e)
            if attempt < max_retries - 1:
                wait = delay_base * (2 ** attempt)  # Exponential backoff
                print(f"Attempt {attempt + 1} failed: {e}. Retrying in {wait}s...")
                time.sleep(wait)
        except Exception as e:
            # For other errors (rate limit, etc.) we also back off
            if attempt < max_retries - 1:
                wait = delay_base * (2 ** attempt)
                time.sleep(wait)
            else:
                raise
    
    raise ValueError(f"Could not get valid JSON in {max_retries} attempts. Last error: {last_error}")

# Usage
SYSTEM = """
Analyze the review and return JSON:
{"sentiment": "POSITIVE|NEGATIVE|NEUTRAL", "aspects": {"food": "positive|negative|null", "service": "positive|negative|null"}}
"""

result = extract_with_retry(SYSTEM, "The food was good but the service was very slow.")
print(result)

Validation with Pydantic

JSON mode guarantees valid JSON but not a correct schema. Pydantic validates the schema:

from pydantic import BaseModel, Field, field_validator
from typing import Optional, Literal
from openai import OpenAI
import json

client = OpenAI()

class SentimentAnalysis(BaseModel):
    sentiment: Literal["POSITIVE", "NEGATIVE", "NEUTRAL"]
    confidence: float = Field(ge=0.0, le=1.0)
    keywords: list[str] = Field(max_length=10)
    
    @field_validator("keywords")
    @classmethod
    def validate_keywords(cls, v: list[str]) -> list[str]:
        return [k.lower().strip() for k in v]  # Normalize

def analyze_with_validation(text: str) -> SentimentAnalysis:
    """
    Analyzes text and validates the output against the Pydantic schema.
    """
    SYSTEM = """
    Analyze the sentiment of the text.
    Return JSON with exactly these fields:
    - sentiment: "POSITIVE", "NEGATIVE" or "NEUTRAL" (exact uppercase)
    - confidence: a number between 0.0 and 1.0
    - keywords: a list of 2-5 words that justify the sentiment
    """
    
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": text}
        ],
        response_format={"type": "json_object"},
        temperature=0
    )
    
    data = json.loads(response.choices[0].message.content)
    
    # Pydantic validates the schema and raises ValidationError if something is off
    return SentimentAnalysis(**data)

# Test
texts = [
    "The product arrived fast and in excellent condition.",
    "Terrible experience. Never buying here again.",
    "It's an ordinary product, neither good nor bad."
]

for text in texts:
    try:
        result = analyze_with_validation(text)
        print(f"Input: {text[:50]}...")
        print(f"  Sentiment: {result.sentiment} ({result.confidence:.0%})")
        print(f"  Keywords: {result.keywords}\n")
    except Exception as e:
        print(f"Error: {e}")

Comparison: JSON Mode vs No JSON Mode

Connection to the Project

In the Structured Data Extractor (capsule 08), you'll use JSON mode + Pydantic to:

• Extract invoice data with the schema InvoiceData(vendor, amount, date, items)
• Handle errors with automatic retry
• Validate the output against the schema before returning
• Fall back to Anthropic if OpenAI fails

Troubleshooting

Problem 1: JSON with a trailing comma or single quotes

Symptom: json.loads() raises JSONDecodeError even with JSON mode on.

Cause: On weaker models, JSON mode sometimes produces almost-valid JSON.

Fix:

import json
import re

def repair_json(text: str) -> dict:
    """Tries to repair JSON with common problems."""
    # Trailing comma: {"a": 1, "b": 2,} → {"a": 1, "b": 2}
    text = re.sub(r',\s*}', '}', text)
    text = re.sub(r',\s*]', ']', text)
    # Single quotes → double (simple fix, doesn't work in every case)
    text = text.replace("'", '"')
    return json.loads(text)

Problem 2: The model invents keys you didn't ask for

Cause: JSON mode only guarantees valid JSON, not a correct schema.

Fix: Use a strict schema with additionalProperties: false, or validate with Pydantic and handle it with extra = "ignore":

class MySchema(BaseModel):
    model_config = {"extra": "ignore"}  # Ignores extra keys
    required_field: str
    optional_field: Optional[str] = None

Problem 3: Truncated output (incomplete JSON)

Symptom: The JSON ends abruptly: {"name": "Juan", "email": "

Cause: max_tokens is too low.

Diagnosis:

print(response.choices[0].finish_reason)  # "length" = truncated

Fix: Raise max_tokens until finish_reason == "stop".

Problem 4: Fields with the wrong types

Symptom: The price arrives as "1299" (string) instead of 1299.0 (float).

Fix: Pydantic coerces automatically in some cases, but it's better to spell it out in the schema and in the prompt:

class ItemData(BaseModel):
    price: float  # Pydantic converts "1299" → 1299.0

# In the prompt: "price: decimal number (no quotes, e.g. 1299.99)"

Exercises

Exercise 1: Implement retry with feedback

Modify the extract_with_retry function so that on the second attempt it includes the faulty output and asks the model to fix it:

def retry_with_correction(system: str, text: str) -> dict:
    """
    Attempt 1: normal
    Attempt 2: show the incorrect output and ask for a correction
    """
    response1 = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": system},
            {"role": "user", "content": text}
        ],
        response_format={"type": "json_object"},
        temperature=0
    )
    raw = response1.choices[0].message.content
    
    try:
        return json.loads(raw)
    except json.JSONDecodeError:
        # Second attempt, with the incorrect output as context
        response2 = client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[
                {"role": "system", "content": system},
                {"role": "user", "content": text},
                {"role": "assistant", "content": raw},
                {"role": "user", "content": f"That JSON isn't valid. The error is: parse error. Return the same content but as valid JSON."}
            ],
            response_format={"type": "json_object"},
            temperature=0
        )
        return json.loads(response2.choices[0].message.content)

Exercise 2: Compare OpenAI JSON mode vs instructions

Run the same extraction task 5 times: (a) without JSON mode (instructions only), (b) with JSON mode. Measure how often the output is valid JSON.

from openai import OpenAI
import json

client = OpenAI()

SYSTEM = 'Extract name and email. Return JSON: {"name": "...", "email": "..."}'
TEXT = "Contact Juan Garcia at juan@company.com"

results = {"without_json_mode": 0, "with_json_mode": 0}

for _ in range(5):
    # Without JSON mode
    r1 = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": TEXT}],
        temperature=0
    )
    try:
        json.loads(r1.choices[0].message.content)
        results["without_json_mode"] += 1
    except json.JSONDecodeError:
        pass
    
    # With JSON mode
    r2 = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "system", "content": SYSTEM}, {"role": "user", "content": TEXT}],
        response_format={"type": "json_object"},
        temperature=0
    )
    try:
        json.loads(r2.choices[0].message.content)
        results["with_json_mode"] += 1
    except json.JSONDecodeError:
        pass

print(f"Without JSON mode: {results['without_json_mode']}/5 valid")
print(f"With JSON mode: {results['with_json_mode']}/5 valid")
# Expected: without=3-5/5, with=5/5

Exercise 3: Pydantic with type coercion

Design a Pydantic schema for an invoice with: vendor (str), amount (float), currency (Literal["USD","EUR","MXN"]), items (a list with name and quantity). Check that it handles amount="1299.99" correctly (string → float).

from pydantic import BaseModel, Field
from typing import Literal, Optional

class InvoiceItem(BaseModel):
    name: str
    quantity: int
    unit_price: Optional[float] = None

class Invoice(BaseModel):
    vendor: str
    total_amount: float  # Pydantic converts "1299.99" → 1299.99
    currency: Literal["USD", "EUR", "MXN"]
    items: list[InvoiceItem] = []

# Coercion test
invoice = Invoice(
    vendor="TechCorp",
    total_amount="1299.99",  # String → float automatically
    currency="USD",
    items=[{"name": "Laptop", "quantity": "1"}]  # "1" → int
)
print(invoice.model_dump())
# {"vendor": "TechCorp", "total_amount": 1299.99, "currency": "USD", "items": [...]}

Exercise 4 (Advanced): Schema with custom validation

Create a Pydantic schema for sentiment analysis where: confidence must be between 0 and 1 (if it comes in >1, divide by 100), and sentiment must be normalized to uppercase.

from pydantic import BaseModel, Field, field_validator
from typing import Literal

class SentimentAnalysis(BaseModel):
    sentiment: Literal["POSITIVE", "NEGATIVE", "NEUTRAL"]
    confidence: float
    
    @field_validator("sentiment", mode="before")
    @classmethod
    def normalize_sentiment(cls, v: str) -> str:
        """Normalize to uppercase and map variants."""
        v_upper = v.upper().strip()
        mapping = {
            "POSITIVO": "POSITIVE",
            "NEGATIVO": "NEGATIVE",
            "NEUTRO": "NEUTRAL",
            "POSITIVE": "POSITIVE",
            "NEGATIVE": "NEGATIVE",
            "NEUTRAL": "NEUTRAL",
        }
        return mapping.get(v_upper, v_upper)
    
    @field_validator("confidence", mode="before")
    @classmethod
    def normalize_confidence(cls, v: float) -> float:
        """If it comes in as a percentage (>1), divide by 100."""
        if v > 1.0:
            return v / 100
        return v

# Test
s1 = SentimentAnalysis(sentiment="positive", confidence=95)
print(s1)  # sentiment=POSITIVE, confidence=0.95

s2 = SentimentAnalysis(sentiment="NEGATIVO", confidence=0.87)
print(s2)  # sentiment=NEGATIVE, confidence=0.87

Summary

• OpenAI JSON mode: response_format={"type": "json_object"} — guarantees syntactically valid JSON
• Structured Outputs with a schema: response_format={"type": "json_schema", ...} — guarantees a specific schema
• Anthropic: Tool use for a strict schema; explicit instructions + robust parsing for free-form JSON
• Retry pattern: Up to 3 attempts with exponential backoff; the second attempt includes feedback about the error
• Pydantic: Validates the schema + coerces types + suggests corrections. Always validate with Pydantic after JSON mode.
• JSON mode doesn't guarantee the schema: It only guarantees that json.loads() won't fail. You always need extra validation.

Further resources

1. OpenAI Structured Outputs Guide — JSON mode, json_schema and a comparison of both
2. OpenAI JSON Mode vs Structured Outputs — When to use each
3. Anthropic Tool Use — How to use tool use for structured output in Claude
4. Pydantic v2 Validators — field_validator, model_validator, type coercion
5. JSON Schema Specification — Understand the schema used in response_format
6. Python json module — Reference for the standard json module