6. Image Analysis Patterns

Description

In the previous capsules you learned to send images to OpenAI, Claude and Gemini, and you compared their strengths. But knowing how to call an API is not the same as knowing what to ask it for. The difference between a useful result and a mediocre one is almost always in the prompt — in how you structure the instruction for the specific task.

This capsule teaches you 7 analysis patterns that work with any provider. Each pattern includes a prompt template, working code, use cases and tips for getting consistent results. Think of them as proven recipes: description, OCR, classification, structured extraction, Q&A, comparison and analysis with additional context.

Why it matters: Without clear patterns, every vision call is an experiment. With patterns, you have predictable results you can test, version and improve. In the end you'll combine several into a multi-step pipeline — the foundation of any production vision system.

Base Function

All the examples use this function. Swap it for whichever provider you prefer:

import base64
import json
from openai import OpenAI

client = OpenAI()

def encode_image(image_path: str) -> str:
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")


def analyze_image(
    image_path: str,
    prompt: str,
    model: str = "gpt-4o",
    temperature: float = 0.3,
    max_tokens: int = 1024,
) -> str:
    b64 = encode_image(image_path)
    response = client.chat.completions.create(
        model=model,
        temperature=temperature,
        max_tokens=max_tokens,
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": prompt},
                {"type": "image_url", "image_url": {
                    "url": f"data:image/jpeg;base64,{b64}",
                    "detail": "high",
                }},
            ],
        }],
    )
    return response.choices[0].message.content

Pattern 1: Description

Goal: Generate an objective textual description of an image.

Template

DESCRIPTION_PROMPT = """Describe this image objectively and in a structured way.

Include:
- Main scene and overall composition
- Visible objects and people (without identifying individuals)
- Dominant colors and lighting conditions
- Visible text (if any)

Format: continuous paragraph.
Length: {length}.
Tone: descriptive, neutral, no subjective interpretations."""

The {length} placeholder controls the length:

Code

def describe_image(image_path: str, detail_level: str = "medium") -> str:
    lengths = {
        "brief": "2 sentences maximum",
        "medium": "between 50 and 100 words",
        "detailed": "between 150 and 250 words",
    }
    length = lengths.get(detail_level, lengths["medium"])
    prompt = DESCRIPTION_PROMPT.format(length=length)
    return analyze_image(image_path, prompt, temperature=0.3)

Tips

• Consistency: Always use the same template to get descriptions that are comparable across images.
• Alt-text: For accessibility, ask for "2 sentences maximum" and add "do not include subjective information or emotions".
• Batch: If you process many images, set temperature=0 to minimize variation.

Pattern 2: OCR (Text Extraction)

Goal: Extract all visible text while preserving the original structure.

Template for plain text

OCR_RAW_PROMPT = """Extract ALL the visible text in this image.

Rules:
- Keep the original reading order (top to bottom, left to right)
- Preserve the structure: paragraphs, lists, headings
- If there are tables, represent them in markdown format
- DO NOT invent or complete text that is not legible
- Illegible text: mark it as [illegible]
- Respond ONLY with the extracted text, no comments"""

Template for structured OCR

OCR_STRUCTURED_PROMPT = """Extract the text from this image and organize it into sections.

Response format:
## Header/Title
[header text]

## Body
[main text]

## Tables
[tables in markdown format]

## Notes/Footer
[secondary text, footnotes, fine print]

If a section does not exist, omit it. DO NOT invent text. Mark doubtful text as [illegible]."""

Code

def extract_text(image_path: str, structured: bool = False, detail: str = "high") -> str:
    prompt = OCR_STRUCTURED_PROMPT if structured else OCR_RAW_PROMPT
    b64 = encode_image(image_path)
    response = client.chat.completions.create(
        model="gpt-4o",
        temperature=0,
        max_tokens=2048,
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": prompt},
                {"type": "image_url", "image_url": {
                    "url": f"data:image/jpeg;base64,{b64}",
                    "detail": detail,
                }},
            ],
        }],
    )
    return response.choices[0].message.content

detail=low vs detail=high

Rule of thumb: Use detail=high if the text uses a font smaller than ~12pt or is handwritten. For screenshots with large text, detail=low saves up to 90% in tokens.

Tips for handwritten text

• Always use detail=high.
• Add to the prompt: "The text may be handwritten. Transcribe it as best as possible."
• Ask for a confidence level: "Mark with [?] the words you are unsure about."

Pattern 3: Classification

Goal: Assign one or more predefined categories to an image.

Single-label template

CLASSIFICATION_PROMPT = """Classify this image into EXACTLY one of the following categories:
{categories}

Rules:
- Respond ONLY with the exact category name
- If no category applies, respond: other
- Do not add explanation or punctuation"""

Multi-label template with confidence

MULTI_CLASSIFICATION_PROMPT = """Classify this image. It may belong to one or more categories.

Possible categories: {categories}

Respond in JSON format:
{{"categories": ["cat1", "cat2"], "confidence": {{"cat1": 0.95, "cat2": 0.72}}}}

Only include categories with confidence > 0.5.
If none apply: {{"categories": ["other"], "confidence": {{"other": 1.0}}}}"""

Code with validation

def classify_image(
    image_path: str,
    categories: list[str],
    multi_label: bool = False,
) -> dict:
    if multi_label:
        prompt = MULTI_CLASSIFICATION_PROMPT.format(categories=", ".join(categories))
    else:
        prompt = CLASSIFICATION_PROMPT.format(categories=", ".join(categories))

    result = analyze_image(image_path, prompt, temperature=0)

    if multi_label:
        try:
            parsed = json.loads(result)
            valid_cats = [c for c in parsed["categories"] if c in categories or c == "other"]
            return {
                "categories": valid_cats,
                "confidence": {k: v for k, v in parsed.get("confidence", {}).items() if k in valid_cats},
            }
        except (json.JSONDecodeError, KeyError):
            return {"categories": ["error_parsing"], "confidence": {}}

    category = result.strip().lower()
    if category not in [c.lower() for c in categories] and category != "other":
        category = "other"
    return {"categories": [category], "confidence": {}}


result = classify_image("photo.jpg", categories=["product", "person", "document", "landscape"])
print(result)

Tips

• temperature=0 always. Classification needs determinism, not creativity.
• Clear, mutually exclusive categories for single-label. If there's ambiguity, use multi-label.
• Maximum ~15 categories in the prompt. With more, the model loses accuracy. For 50+ categories, use hierarchical classification (first the general category, then the subcategory).

Pattern 4: Structured Extraction (JSON)

Goal: Extract specific data from an image and return it as valid JSON.

Template

EXTRACTION_PROMPT = """Extract the following fields from this image and return them as valid JSON.

Fields to extract:
{field_descriptions}

Exact expected format:
{example_json}

Rules:
- Respond ONLY with the JSON, no extra text or markdown
- Use null for fields not visible in the image
- Dates in YYYY-MM-DD format
- Amounts as numbers (no currency symbol)
- Do not invent data that is not visible"""

Schemas for common cases

SCHEMAS = {
    "invoice": {
        "fields": (
            "- date: issue date\n- number: invoice number\n"
            "- issuer: name of the issuing company\n- recipient: client name\n"
            "- subtotal: amount before taxes\n- tax: tax amount\n"
            "- total: total amount\n"
            "- items: list [{description, quantity, unit_price, amount}]"
        ),
        "example": '{"date":"2025-03-15","number":"INV-001234","issuer":"Acme Inc",'
                   '"recipient":"Client LLC","subtotal":1000.00,"tax":160.00,'
                   '"total":1160.00,"items":[{"description":"Consulting",'
                   '"quantity":1,"unit_price":1000.00,"amount":1000.00}]}',
    },
    "receipt": {
        "fields": (
            "- merchant: name of the establishment\n- date: purchase date\n"
            "- items: list [{name, price}]\n- total: total amount\n"
            "- payment_method: cash, card, etc."
        ),
        "example": '{"merchant":"Store XYZ","date":"2025-01-20",'
                   '"items":[{"name":"Coffee","price":4.50}],'
                   '"total":4.50,"payment_method":"card"}',
    },
    "identification": {
        "fields": (
            "- type: document type (ID card, passport, license)\n"
            "- name: full name\n- document_number: number or key\n"
            "- birth_date: date of birth\n"
            "- issue_date: date of issue\n"
            "- expiry_date: expiration date"
        ),
        "example": '{"type":"ID card","name":"John Doe",'
                   '"document_number":"ABCD123456","birth_date":"1990-05-15",'
                   '"issue_date":"2020-01-10","expiry_date":"2030-01-10"}',
    },
}

Code with parsing and error handling

def extract_structured(image_path: str, schema_name: str, max_retries: int = 2) -> dict | None:
    schema = SCHEMAS.get(schema_name)
    if not schema:
        raise ValueError(f"Unknown schema: {schema_name}. Available: {list(SCHEMAS.keys())}")

    prompt = EXTRACTION_PROMPT.format(
        field_descriptions=schema["fields"],
        example_json=schema["example"],
    )

    for attempt in range(max_retries + 1):
        raw = analyze_image(image_path, prompt, temperature=0)
        cleaned = raw.strip()
        if cleaned.startswith("```"):
            cleaned = cleaned.split("\n", 1)[-1].rsplit("```", 1)[0].strip()

        try:
            return json.loads(cleaned)
        except json.JSONDecodeError:
            if attempt < max_retries:
                prompt += "\n\nIMPORTANT: Your previous response was not valid JSON. Respond ONLY with JSON."
                continue
            return None


data = extract_structured("invoice_scan.jpg", "invoice")
if data:
    print(f"Issuer: {data.get('issuer')} — Total: ${data.get('total')}")
else:
    print("Could not extract valid JSON after retries")

The cleaned step handles the most common invalid-JSON case: when the model wraps the response in markdown blocks (```json ... ```). The retry loop gives a second chance with a more emphatic instruction.

Pattern 5: Contextual Q&A

Goal: Answer specific questions based exclusively on what is visible in the image.

Template

QA_PROMPT = """Answer the following question based ONLY on what is visible in the image.

Question: {question}

Rules:
- If the information is NOT visible, respond exactly: "Not visible in the image"
- Do not make assumptions or use external knowledge
- Be concise and direct"""

Code with multiple questions

def ask_about_image(image_path: str, question: str) -> str:
    prompt = QA_PROMPT.format(question=question)
    return analyze_image(image_path, prompt, temperature=0.2)


def ask_multiple(image_path: str, questions: list[str]) -> dict[str, str]:
    combined = (
        "Answer each question based ONLY on what is visible in the image.\n"
        "If the information is not visible, respond 'Not visible in the image'.\n\n"
        "Respond in JSON format: {\"1\": \"answer\", \"2\": \"answer\", ...}\n\n"
    )
    for i, q in enumerate(questions, 1):
        combined += f"{i}. {q}\n"

    raw = analyze_image(image_path, combined, temperature=0.2)
    try:
        answers = json.loads(raw)
        return {questions[int(k) - 1]: v for k, v in answers.items() if int(k) <= len(questions)}
    except (json.JSONDecodeError, ValueError):
        return {questions[0]: raw}


answers = ask_multiple("dashboard.png", [
    "What is the total sales figure?",
    "What period does the report cover?",
    "Which category has the highest revenue?",
])
for q, a in answers.items():
    print(f"Q: {q}\nA: {a}\n")

Tips

• Grounding: The "ONLY on what is visible" instruction reduces hallucinations significantly.
• Batch: Sending several questions in a single call is cheaper, but loses accuracy beyond 5-6 questions.
• Specificity: "What is the number in the top-right corner?" is better than "What is the number?".

Pattern 6: Image Comparison

Goal: Analyze similarities and differences between two or more images.

Template

COMPARISON_PROMPT = """Compare the {n} images provided.

Structure your response:
1. **Similarities:** Common elements across the images
2. **Differences:** Specific changes between each image
3. **Conclusion:** Summary in 1-2 sentences

Be specific: mention locations, colors, text and concrete elements."""

Code

def compare_images(image_paths: list[str], custom_prompt: str | None = None) -> str:
    prompt = custom_prompt or COMPARISON_PROMPT.format(n=len(image_paths))
    content = [{"type": "text", "text": prompt}]
    for path in image_paths:
        b64 = encode_image(path)
        content.append({"type": "image_url", "image_url": {
            "url": f"data:image/jpeg;base64,{b64}", "detail": "high",
        }})

    response = client.chat.completions.create(
        model="gpt-4o", temperature=0.3, max_tokens=1024,
        messages=[{"role": "user", "content": content}],
    )
    return response.choices[0].message.content

Use cases

Pattern 7: Analysis with Additional Context

Goal: Improve accuracy by combining the image with relevant textual information.

Template and code

CONTEXT_ANALYSIS_PROMPT = """Analyze this image considering the following context:

Context: {context}

Task: {task}

Use the context to interpret the image better, but base your observations on what is actually visible."""


def analyze_with_context(image_path: str, context: str, task: str) -> str:
    prompt = CONTEXT_ANALYSIS_PROMPT.format(context=context, task=task)
    return analyze_image(image_path, prompt, temperature=0.3)


result = analyze_with_context(
    image_path="architecture_plan.jpg",
    context="Floor plan of the second floor of a commercial building. "
            "The client requested 3 private offices and a common area.",
    task="Check whether the plan meets the requirements. "
         "Identify each office and the common area.",
)

When to add context

Caution: Context helps the model interpret, but it can also bias it. If you say "this image shows a defect", the model will look for one even if it doesn't exist. Use factual context, not conclusions.

Combining Patterns

The real power emerges when you chain them. Typical pipeline: classify → extract based on classification → validate the result.

def document_pipeline(image_path: str) -> dict:
    classification = classify_image(
        image_path,
        categories=["invoice", "receipt", "identification", "other"],
    )
    doc_type = classification["categories"][0]

    if doc_type == "other":
        return {
            "type": "unrecognized",
            "description": describe_image(image_path, detail_level="detailed"),
            "data": None,
        }

    extracted = extract_structured(image_path, doc_type)

    validation_questions = {
        "invoice": "Does the total match the sum of subtotal and tax?",
        "receipt": "Does the total match the sum of the items?",
        "identification": "Are all the dates legible and in a valid format?",
    }
    validation = ask_about_image(
        image_path,
        validation_questions.get(doc_type, "Is the data consistent?"),
    )

    return {"type": doc_type, "data": extracted, "validation": validation}


result = document_pipeline("document_scan.jpg")
print(f"Type: {result['type']}")
if result["data"]:
    print(f"Data: {json.dumps(result['data'], indent=2, ensure_ascii=False)}")
print(f"Validation: {result.get('validation', 'N/A')}")

This pipeline demonstrates the classify → extract → validate flow that is the foundation of document-processing systems with vision.

Pattern Summary Table

Troubleshooting

Problem 1: OCR "invents" text that doesn't exist

Symptom: The model returns plausible text that is not in the image.

Solution: Add "If a fragment is not legible, write [illegible]. NEVER complete or guess text." Use temperature=0 and detail=high — low resolution causes more hallucinations.

Problem 2: Inconsistent classification

Symptom: The same image returns "product" sometimes and "object" other times.

Solution: temperature=0 mandatory. Check that the categories are mutually exclusive. Add brief definitions: "product: commercial item; object: non-commercial element."

Problem 3: Malformed JSON in extraction

Symptom: json.loads() fails on the response.

Solution: The code already handles ```json ``` blocks. In addition: add "Respond ONLY with JSON, no text or code blocks." Implement retries (Pattern 4). If it persists, use regex to extract the first {...}.

Problem 4: Vague or generic descriptions

Symptom: "The image shows an object on a surface" with no detail.

Solution: Use the template with an explicit {length}. Add categories: "Include: materials, textures, visible brands, condition." Increase max_tokens — the model sometimes truncates.

Exercises

Exercise 1 (Easy): PromptBuilder

Create a PromptBuilder class that generates the correct prompt template for each pattern. It must accept the pattern name and its parameters (categories for classification, schema for extraction, question for Q&A).

Requirements:

• Method build(pattern_name, **kwargs) that returns a str
• Support for the 7 patterns
• Raise ValueError if a required parameter is missing

class PromptBuilder:
    TEMPLATES = {
        "description": DESCRIPTION_PROMPT,
        "ocr": OCR_RAW_PROMPT,
        "ocr_structured": OCR_STRUCTURED_PROMPT,
        "classification": CLASSIFICATION_PROMPT,
        "classification_multi": MULTI_CLASSIFICATION_PROMPT,
        "extraction": EXTRACTION_PROMPT,
        "qa": QA_PROMPT,
        "comparison": COMPARISON_PROMPT,
        "context": CONTEXT_ANALYSIS_PROMPT,
    }
    REQUIRED_PARAMS = {
        "description": ["length"], "ocr": [], "ocr_structured": [],
        "classification": ["categories"], "classification_multi": ["categories"],
        "extraction": ["field_descriptions", "example_json"],
        "qa": ["question"], "comparison": ["n"], "context": ["context", "task"],
    }

    def build(self, pattern_name: str, **kwargs) -> str:
        if pattern_name not in self.TEMPLATES:
            raise ValueError(f"Unknown pattern: {pattern_name}")

        missing = [p for p in self.REQUIRED_PARAMS[pattern_name] if p not in kwargs]
        if missing:
            raise ValueError(f"Missing parameters for '{pattern_name}': {missing}")

        if "categories" in kwargs and isinstance(kwargs["categories"], list):
            kwargs["categories"] = ", ".join(kwargs["categories"])

        return self.TEMPLATES[pattern_name].format(**kwargs)


builder = PromptBuilder()
prompt = builder.build("classification", categories=["cat", "dog", "bird"])
print(prompt)

Exercise 2 (Medium): Invoice Extractor Pipeline

Build extract_invoice(image_path) that combines OCR + structured extraction:

1. Extract the raw text with OCR
2. Use that text as additional context for the invoice JSON extraction

Requirements:

• Returns a dict with the invoice schema fields
• If OCR detects no text, returns {"error": "No text detected"}
• Includes the OCR text under the key "_ocr_raw"

def extract_invoice(image_path: str) -> dict:
    ocr_text = extract_text(image_path, structured=True)
    if not ocr_text or not ocr_text.strip():
        return {"error": "No text detected"}

    enriched_prompt = EXTRACTION_PROMPT.format(
        field_descriptions=SCHEMAS["invoice"]["fields"],
        example_json=SCHEMAS["invoice"]["example"],
    )
    enriched_prompt += (
        f"\n\nContext — text already extracted from the image:\n---\n{ocr_text}\n---\n"
        f"Use both the image and the extracted text for greater accuracy."
    )

    for attempt in range(3):
        raw = analyze_image(image_path, enriched_prompt, temperature=0)
        cleaned = raw.strip()
        if cleaned.startswith("```"):
            cleaned = cleaned.split("\n", 1)[-1].rsplit("```", 1)[0].strip()
        try:
            data = json.loads(cleaned)
            data["_ocr_raw"] = ocr_text
            return data
        except json.JSONDecodeError:
            if attempt < 2:
                enriched_prompt += "\nInvalid JSON. Respond ONLY with JSON."

    return {"error": "Could not parse JSON", "_ocr_raw": ocr_text}


invoice = extract_invoice("invoice_scan.jpg")
if "error" not in invoice:
    print(f"Issuer: {invoice.get('issuer')} — Total: {invoice.get('total')}")

Exercise 3 (Medium): Multi-Pattern Analyzer

Create full_analysis(image_path, categories) that runs 3 patterns in sequence: description → classification → OCR.

Requirements:

• Description at "medium" level, single-label classification, raw OCR
• Return {"description": str, "classification": dict, "ocr_text": str}
• If a step fails, include the error in the corresponding field without stopping the pipeline

def full_analysis(image_path: str, categories: list[str]) -> dict:
    result = {"description": None, "classification": None, "ocr_text": None}

    for key, fn in [
        ("description", lambda: describe_image(image_path, detail_level="medium")),
        ("classification", lambda: classify_image(image_path, categories)),
        ("ocr_text", lambda: extract_text(image_path, structured=False)),
    ]:
        try:
            result[key] = fn()
        except Exception as e:
            result[key] = f"Error: {e}" if key != "classification" else {"error": str(e)}

    return result


analysis = full_analysis(
    "document.jpg",
    categories=["invoice", "receipt", "contract", "identification"],
)
print(f"Type: {analysis['classification']}")
print(f"Description: {str(analysis['description'])[:100]}...")

Summary

• Every vision task has an optimal pattern with its own template, temperature and output format.
• temperature=0 for deterministic tasks (classification, OCR, extraction). 0.2-0.3 for descriptive ones.
• The patterns are provider-agnostic: they work with OpenAI, Claude and Gemini by changing only the base function.
• Combining patterns into pipelines (classify → extract → validate) is how real systems are built.
• Always validate the output: categories within the allowed list, parseable JSON, OCR with no invented text.

Additional Resources

1. OpenAI Vision Best Practices
2. Prompt Engineering for Vision Models
3. Anthropic Vision Documentation
4. Google Gemini Vision Guide