4. Gemini Vision (Google)

Description

Google Gemini represents a different approach to visual analysis with LLMs. While GPT-4 Vision (capsule 02) and Claude 3 Vision (capsule 03) offer solid vision capabilities, Gemini stands out for three concrete advantages: a 1 million token context window (vs ~128K for the others), native video support as input, and Gemini Flash, an optimized model that cuts costs by up to ~33x compared to GPT-4o.

Google's AI API is structurally different. Instead of OpenAI/Anthropic's messages → content pattern, Gemini uses GenerativeModel + generate_content() with a flat list of parts. Images are sent as raw bytes, not as Base64 strings.

These differences aren't just cosmetic — they define when and why to choose Gemini over the alternatives.

Models with Vision

Google offers three main models with vision capability:

They all share the 1M-token context window. The difference is in reasoning quality and cost.

Which one to choose?

• gemini-2.0-flash: Your default. A balance of quality, speed and cost. It supports the most recent features.
• gemini-1.5-flash: When you process thousands of images and every cent counts. Slightly lower quality but extremely cheap.
• gemini-1.5-pro: When you need the best possible reasoning: complex documents, multi-page analysis, tasks that require deep understanding.

For comparison: GPT-4o costs ~$2.50/1M input tokens. Gemini 1.5 Flash costs ~$0.075. That's ~33x cheaper.

Estimated cost for 1,000 images (~500 tokens/image):
  GPT-4o:           ~$1.25
  Gemini 1.5 Flash: ~$0.04  ← ~33x cheaper

Setup and API

Installation

pip install google-generativeai

Configuration

import os
import google.generativeai as genai

genai.configure(api_key=os.environ["GOOGLE_API_KEY"])

You get the API key at Google AI Studio. Unlike OpenAI and Anthropic, Google offers a free tier with generous limits for experimentation.

To check that it works:

model = genai.GenerativeModel("gemini-2.0-flash")
response = model.generate_content("Hello, does this work?")
print(response.text)

Request Structure — Differences

The fundamental difference between Gemini and OpenAI/Anthropic is in how you build the request.

The Gemini pattern

import google.generativeai as genai
from pathlib import Path

genai.configure(api_key=os.environ["GOOGLE_API_KEY"])

model = genai.GenerativeModel("gemini-2.0-flash")

image_part = {
    "mime_type": "image/jpeg",
    "data": Path("photo.jpg").read_bytes()
}

response = model.generate_content(["Describe this image.", image_part])
print(response.text)

Notice the key differences:

1. There's no messages: Instead of an array of messages with roles, Gemini takes a flat list of "parts" (text, images, etc.)
2. Raw bytes, not Base64: The data field expects bytes directly. You don't need base64.b64encode().
3. Explicit mime_type: Similar to Anthropic, but without the source wrapper.
4. The model as an object: You create a GenerativeModel once and call generate_content() multiple times.

Side-by-side comparison

Why bytes instead of Base64?

Base64 increases the size by ~33%. For a 3MB image, the Base64 string weighs ~4MB. Gemini avoids that conversion by taking bytes directly, which reduces the request payload. This matters especially when you send many images or large documents.

Input: Bytes vs URL vs File Upload

Gemini offers three ways to send images, each optimized for a different scenario.

Method 1: Direct bytes (local files)

from pathlib import Path

image_part = {
    "mime_type": "image/jpeg",
    "data": Path("photo.jpg").read_bytes()
}
response = model.generate_content(["Describe this image.", image_part])

When to use it: Local files under ~20MB.

Method 2: Public URL

image_part = genai.types.Part.from_uri(
    "https://example.com/image.png",
    mime_type="image/png"
)
response = model.generate_content(["Describe this image.", image_part])

When to use it: Images already hosted on the web. It avoids download + send.

Method 3: File API (large files)

For large files (>20MB) or videos, Gemini offers a File API that uploads the file first and then references it:

uploaded = genai.upload_file("long_document.pdf", mime_type="application/pdf")

response = model.generate_content([
    "Summarize the key points of this document.",
    uploaded
])
print(response.text)

When to use it: Large PDFs, videos, or when you're going to reuse the same file across multiple requests.

Multiple Images

This is where the 1M-token context window shines. Sending multiple images with Gemini is trivial — you just add more parts to the list:

from pathlib import Path

model = genai.GenerativeModel("gemini-2.0-flash")

parts = ["Compare these three images. What do they have in common and how do they differ?"]

image_paths = ["photo1.jpg", "photo2.jpg", "photo3.jpg"]
for path in image_paths:
    parts.append({
        "mime_type": "image/jpeg",
        "data": Path(path).read_bytes()
    })

response = model.generate_content(parts)
print(response.text)

With 1M tokens of context, you can send dozens of images in a single request without worrying about the limit. For OpenAI (128K) or Anthropic (200K), sending more than 5-10 high-resolution images already starts to be risky.

A practical example — analyzing a full catalog:

from pathlib import Path

def analyze_catalog(image_dir: str, prompt: str) -> str:
    model = genai.GenerativeModel("gemini-1.5-flash")

    parts = [prompt]
    for img in sorted(Path(image_dir).glob("*.jpg")):
        parts.append({
            "mime_type": "image/jpeg",
            "data": img.read_bytes()
        })

    response = model.generate_content(parts)
    return response.text

result = analyze_catalog(
    "product_catalog/",
    "Classify each product by category and estimate its price range."
)

Example 1: Image description

import os
import google.generativeai as genai
from pathlib import Path

genai.configure(api_key=os.environ["GOOGLE_API_KEY"])

def describe_image(image_path: str, detail_level: str = "medium") -> str:
    model = genai.GenerativeModel("gemini-2.0-flash")

    prompts = {
        "brief": "Describe this image in one sentence.",
        "medium": "Describe this image in 2-3 sentences, covering the main elements.",
        "detailed": "Describe this image in detail: objects, colors, composition, context and mood."
    }

    image_part = {
        "mime_type": "image/jpeg",
        "data": Path(image_path).read_bytes()
    }

    response = model.generate_content([prompts[detail_level], image_part])
    return response.text

print(describe_image("landscape.jpg", "detailed"))

Example 2: OCR with Gemini

import os
import google.generativeai as genai
from pathlib import Path

genai.configure(api_key=os.environ["GOOGLE_API_KEY"])

def extract_text(image_path: str, preserve_layout: bool = False) -> str:
    model = genai.GenerativeModel("gemini-2.0-flash")

    if preserve_layout:
        prompt = (
            "Extract ALL the visible text in this image. "
            "Keep the original structure and formatting as faithfully as possible. "
            "If there are tables, represent them in markdown format."
        )
    else:
        prompt = (
            "Extract all the visible text in this image. "
            "Return only the text, with no additional explanations."
        )

    image_part = {
        "mime_type": "image/png",
        "data": Path(image_path).read_bytes()
    }

    response = model.generate_content([prompt, image_part])
    return response.text

text = extract_text("receipt.png", preserve_layout=True)
print(text)

Example 3: Long document analysis

This is one of Gemini's unique strengths. With 1M tokens of context, you can send a 20+ page PDF and analyze it whole — something GPT-4o (128K) and Claude (200K) can't do with documents that long.

import os
import google.generativeai as genai

genai.configure(api_key=os.environ["GOOGLE_API_KEY"])

def analyze_long_document(pdf_path: str, question: str) -> str:
    uploaded_file = genai.upload_file(pdf_path, mime_type="application/pdf")

    model = genai.GenerativeModel("gemini-1.5-pro")

    response = model.generate_content([
        f"""Analyze this complete document and answer the following question:

{question}

Base your answer solely on the content of the document.
If the information isn't in the document, say so explicitly.""",
        uploaded_file
    ])
    return response.text

result = analyze_long_document(
    "annual_report_2024.pdf",
    "What were the three main risks identified and what mitigations were proposed?"
)
print(result)

Comparison of document capacity:

50-page document (~75,000 tokens):
  GPT-4o (128K):     ✅ It fits, but little room is left for the answer
  Claude 3.5 (200K): ✅ It fits with margin
  Gemini 1.5 Pro (1M): ✅ Uses only ~7.5% of the context

200-page document (~300,000 tokens):
  GPT-4o (128K):     ❌ Doesn't fit
  Claude 3.5 (200K): ❌ Doesn't fit (or it's very tight)
  Gemini 1.5 Pro (1M): ✅ Uses only ~30% of the context

Example 4: Video Analysis

Gemini is the only one of the three providers that supports video as native input. You can send a video and ask questions about its visual and audio content.

import os
import time
import google.generativeai as genai

genai.configure(api_key=os.environ["GOOGLE_API_KEY"])

def analyze_video(video_path: str, prompt: str) -> str:
    video_file = genai.upload_file(video_path)

    while video_file.state.name == "PROCESSING":
        time.sleep(5)
        video_file = genai.get_file(video_file.name)

    if video_file.state.name == "FAILED":
        raise ValueError(f"Video processing failed: {video_file.state.name}")

    model = genai.GenerativeModel("gemini-2.0-flash")
    response = model.generate_content([prompt, video_file])
    return response.text

result = analyze_video(
    "product_demo.mp4",
    "Describe step by step what happens in this video. "
    "Identify the main actions and any visible text."
)
print(result)

The File API processes the video asynchronously. The while loop waits until it's ready. Supported formats: MP4, MOV, AVI, MKV, WEBM, among others.

This opens up use cases that don't exist in OpenAI or Anthropic: QA of training videos, analysis of product demos, monitoring recordings. With the other providers, the alternative is extracting frames as individual images — Gemini handles it natively.

Safety Settings

Gemini includes safety filters that can block responses without a clear warning. This is a common pitfall — your code runs but response.text raises an exception.

The problem

response = model.generate_content(["Analyze this medical image.", image_part])
print(response.text)  # ValueError: response has no candidates

Gemini may decide the image or the prompt violates its policies and block the whole response.

The solution

Configure safety_settings to adjust the thresholds:

from google.generativeai.types import HarmCategory, HarmBlockThreshold

safety_config = {
    HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_NONE,
    HarmCategory.HARM_CATEGORY_HATE_SPEECH: HarmBlockThreshold.BLOCK_NONE,
    HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: HarmBlockThreshold.BLOCK_NONE,
    HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE,
}

model = genai.GenerativeModel("gemini-2.0-flash", safety_settings=safety_config)
response = model.generate_content(["Analyze this medical image.", image_part])

Always check response.candidates before accessing response.text — if it was blocked, candidates will be empty or have finish_reason == "SAFETY".

Tokens and Costs

Reading token usage

response = model.generate_content(["Describe this image.", image_part])

print(f"Input tokens:  {response.usage_metadata.prompt_token_count}")
print(f"Output tokens: {response.usage_metadata.candidates_token_count}")
print(f"Total tokens:  {response.usage_metadata.total_token_count}")

Cost comparison between providers

For high-volume vision tasks (e.g., processing catalogs, mass OCR, image classification), Gemini Flash can cut costs drastically.

Estimate costs before running

def estimate_cost_flash(image_paths: list[str], avg_output_tokens: int = 200) -> float:
    estimated_tokens_per_image = 500
    total_input = len(image_paths) * estimated_tokens_per_image
    total_output = len(image_paths) * avg_output_tokens

    cost_input = (total_input / 1_000_000) * 0.075
    cost_output = (total_output / 1_000_000) * 0.30

    return cost_input + cost_output

cost = estimate_cost_flash(["img1.jpg"] * 10_000)
print(f"Estimated cost for 10,000 images: ${cost:.4f}")

Troubleshooting

1. response has no candidates / Blocked response

Cause: The safety filters blocked the response.

response = model.generate_content([prompt, image_part])

if not response.candidates:
    print("Blocked. Reason:", response.prompt_feedback)
else:
    finish = response.candidates[0].finish_reason.name
    if finish == "SAFETY":
        print("Blocked by safety. Ratings:")
        for r in response.candidates[0].safety_ratings:
            print(f"  {r.category.name}: {r.probability.name}")
    else:
        print(response.text)

Solution: Configure safety_settings as shown in the previous section.

2. google.api_core.exceptions.InvalidArgument: API key not valid

Cause: An incorrect or unconfigured API key.

import os
key = os.environ.get("GOOGLE_API_KEY", "NOT_SET")
print(f"Key present: {'Yes' if key != 'NOT_SET' else 'No'}")
print(f"Key starts with: {key[:8]}..." if key != "NOT_SET" else "")

Solution: Check in Google AI Studio that the key is active and that the project has the API enabled.

3. File upload failed / Timeout on large files

Cause: A file that's too large or a slow connection.

uploaded = genai.upload_file("large_video.mp4")

import time
timeout = 300
start = time.time()
while uploaded.state.name == "PROCESSING":
    if time.time() - start > timeout:
        raise TimeoutError(f"Upload processing exceeded {timeout}s")
    time.sleep(10)
    uploaded = genai.get_file(uploaded.name)

Solution: For large videos, wait patiently. Processing can take minutes.

4. models/gemini-xxx is not found

Cause: An incorrect model name or a deprecated model.

for m in genai.list_models():
    if "vision" in m.name or "gemini" in m.name:
        print(f"{m.name} — supports generateContent: {m.supported_generation_methods}")

Solution: List the available models and check the exact name. Google renames models frequently.

5. Resource exhausted / Rate limiting

Cause: Too many requests per minute.

import time

def generate_with_retry(model, parts, max_retries: int = 3):
    for attempt in range(max_retries):
        try:
            return model.generate_content(parts)
        except Exception as e:
            if "Resource exhausted" in str(e) and attempt < max_retries - 1:
                wait = 2 ** attempt * 10
                print(f"Rate limited. Waiting {wait}s...")
                time.sleep(wait)
            else:
                raise

Exercises

Exercise 1 (Easy): Configurable analysis function

Write a gemini_analyze function that takes an image path, a prompt, and optionally a model (default gemini-2.0-flash). It must return the response text. Automatically detect the mime_type based on the file extension.

import os
import google.generativeai as genai
from pathlib import Path

genai.configure(api_key=os.environ["GOOGLE_API_KEY"])

MIME_TYPES = {
    ".jpg": "image/jpeg",
    ".jpeg": "image/jpeg",
    ".png": "image/png",
    ".gif": "image/gif",
    ".webp": "image/webp",
}

def gemini_analyze(
    image_path: str,
    prompt: str,
    model_name: str = "gemini-2.0-flash"
) -> str:
    path = Path(image_path)
    mime_type = MIME_TYPES.get(path.suffix.lower(), "image/jpeg")

    model = genai.GenerativeModel(model_name)
    image_part = {
        "mime_type": mime_type,
        "data": path.read_bytes()
    }

    response = model.generate_content([prompt, image_part])
    return response.text

result = gemini_analyze("photo.png", "What do you see in this image?")
print(result)

Exercise 2 (Medium): Batch processor with Gemini Flash

Create a batch_analyze function that takes a list of image paths and a prompt, processes them all with gemini-1.5-flash (the cheapest), and returns a dictionary {filename: response}. Include: mime type detection, per-image error handling (if one fails, continue with the rest), and a final report with the total tokens used.

import os
import google.generativeai as genai
from pathlib import Path

genai.configure(api_key=os.environ["GOOGLE_API_KEY"])

MIME_TYPES = {
    ".jpg": "image/jpeg",
    ".jpeg": "image/jpeg",
    ".png": "image/png",
    ".gif": "image/gif",
    ".webp": "image/webp",
}

def batch_analyze(
    image_paths: list[str],
    prompt: str
) -> dict:
    model = genai.GenerativeModel("gemini-1.5-flash")
    results = {}
    total_input_tokens = 0
    total_output_tokens = 0
    errors = 0

    for img_path in image_paths:
        path = Path(img_path)
        filename = path.name

        try:
            mime_type = MIME_TYPES.get(path.suffix.lower(), "image/jpeg")
            image_part = {
                "mime_type": mime_type,
                "data": path.read_bytes()
            }

            response = model.generate_content([prompt, image_part])
            results[filename] = response.text

            total_input_tokens += response.usage_metadata.prompt_token_count
            total_output_tokens += response.usage_metadata.candidates_token_count

        except Exception as e:
            results[filename] = f"ERROR: {e}"
            errors += 1

    cost_input = (total_input_tokens / 1_000_000) * 0.075
    cost_output = (total_output_tokens / 1_000_000) * 0.30

    print(f"\n--- Batch Report ---")
    print(f"Images processed: {len(image_paths) - errors}/{len(image_paths)}")
    print(f"Errors: {errors}")
    print(f"Input tokens: {total_input_tokens:,}")
    print(f"Output tokens: {total_output_tokens:,}")
    print(f"Estimated cost: ${cost_input + cost_output:.6f}")

    return results

images = ["img1.jpg", "img2.png", "img3.jpg"]
results = batch_analyze(images, "Describe this image in one sentence.")

for name, text in results.items():
    print(f"\n{name}: {text}")

Exercise 3 (Medium): Multi-provider comparator

Create a compare_providers function that takes an image path and a prompt, sends the same image to OpenAI (gpt-4o-mini), Anthropic (claude-sonnet-4-20250514) and Google (gemini-2.0-flash), and returns a dictionary with each one's response and latency. Use time.time() to measure the latency.

import os
import time
import base64
import google.generativeai as genai
from pathlib import Path
from openai import OpenAI
import anthropic

genai.configure(api_key=os.environ["GOOGLE_API_KEY"])

def compare_providers(image_path: str, prompt: str) -> dict:
    path = Path(image_path)
    raw_bytes = path.read_bytes()
    b64_string = base64.b64encode(raw_bytes).decode()
    results = {}

    # --- OpenAI ---
    start = time.time()
    try:
        client_oai = OpenAI()
        resp_oai = client_oai.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{
                "role": "user",
                "content": [
                    {"type": "text", "text": prompt},
                    {"type": "image_url", "image_url": {
                        "url": f"data:image/jpeg;base64,{b64_string}"
                    }}
                ]
            }],
            max_tokens=500
        )
        results["openai"] = {
            "response": resp_oai.choices[0].message.content,
            "latency_s": round(time.time() - start, 2)
        }
    except Exception as e:
        results["openai"] = {"response": f"ERROR: {e}", "latency_s": round(time.time() - start, 2)}

    # --- Anthropic ---
    start = time.time()
    try:
        client_ant = anthropic.Anthropic()
        resp_ant = client_ant.messages.create(
            model="claude-sonnet-4-20250514",
            max_tokens=500,
            messages=[{
                "role": "user",
                "content": [
                    {"type": "image", "source": {
                        "type": "base64",
                        "media_type": "image/jpeg",
                        "data": b64_string
                    }},
                    {"type": "text", "text": prompt}
                ]
            }]
        )
        results["anthropic"] = {
            "response": resp_ant.content[0].text,
            "latency_s": round(time.time() - start, 2)
        }
    except Exception as e:
        results["anthropic"] = {"response": f"ERROR: {e}", "latency_s": round(time.time() - start, 2)}

    # --- Gemini ---
    start = time.time()
    try:
        model_gem = genai.GenerativeModel("gemini-2.0-flash")
        resp_gem = model_gem.generate_content([
            prompt,
            {"mime_type": "image/jpeg", "data": raw_bytes}
        ])
        results["gemini"] = {
            "response": resp_gem.text,
            "latency_s": round(time.time() - start, 2)
        }
    except Exception as e:
        results["gemini"] = {"response": f"ERROR: {e}", "latency_s": round(time.time() - start, 2)}

    print("\n=== Multi-Provider Comparison ===\n")
    for provider, data in results.items():
        print(f"--- {provider.upper()} ({data['latency_s']}s) ---")
        print(data["response"][:300])
        print()

    return results

compare_providers("test.jpg", "Describe this image in 2 sentences.")

Summary

• API: genai.configure() + GenerativeModel() + generate_content([parts])
• Image as bytes: Dict {"mime_type": ..., "data": bytes} — no Base64
• Three input methods: direct bytes, URL with Part.from_uri(), File API for large files
• 1M-token context window: Enables long documents and multiple images where other models can't
• Native video: The only provider with direct support for video as input
• Gemini Flash: ~33x cheaper than GPT-4o for high-volume vision tasks
• Safety settings: Configure them explicitly to avoid unexpected blocks

Additional resources

1. Google AI Python SDK
2. Gemini Vision Guide
3. Gemini File API
4. Models and Pricing
5. Safety Settings