Module 4: Image Generation
1. Introduction: Image Generation
Description
This is the first capsule of Module 4 of the Multimodal AI Guide. Here you'll understand the current state of AI image generation, what models exist, how they work at a high level, and what you can build with them. If the earlier modules (vision, documents) focused on understanding visual content, this module focuses on creating visual content from scratch.
Why it matters: Image generation is the most visible "output" modality of multimodal AI. In 2023-2024, DALL-E 3, Midjourney and Stable Diffusion transformed entire industries: marketing, e-commerce, publishing, gaming, and prototyping. However, most developers have only tried these tools casually. Mastering the generation APIs — understanding parameters, costs, limitations and how to combine generation with analysis — sets you apart from the rest.
This module is not a "how to generate pretty images" tutorial. It's a technical guide for integrating image generation into software systems: APIs, parameters, fallbacks between providers, pipelines that combine generation with vision, and architecture decisions (DALL-E or Stable Diffusion? When to use each?).
The module's final project — an Image Generator with fallback — demonstrates resilience against provider failures: if DALL-E fails, the system automatically generates with Stable Diffusion, without the user noticing.
Where Are We in the Guide?
Context
This guide has 8 modules organized into 3 phases:
Phase 1: Multimodal Foundations (Modules 1-3)
├── Module 1: Introduction to Multimodal AI ✓ completed
├── Module 2: Vision + LLMs ✓ completed
└── Module 3: Document Understanding ✓ completed
Phase 2: Generation and Audio (Modules 4-5)
├── Module 4: Image Generation ← YOU ARE HERE
└── Module 5: Audio Processing
Phase 3: RAG, Use Cases and Project (Modules 6-8)
├── Module 6: Multimodal RAG
├── Module 7: Use Cases
└── Module 8: Final Project — Multimodal Document Analyzer
Total estimated duration: 6-7 hours (self-paced).
Where are we headed?
In modules 1-3 you learned to analyze content: send images to GPT-4 Vision, extract data from PDFs, classify documents. Now we reverse the direction: instead of image → text, we do text → image.
The progression within this module is deliberate:
- First you understand the landscape (this capsule) — models, providers, use cases
- Then you master DALL-E 3 (capsule 2) — the most accessible and highest-quality option
- Then Stable Diffusion (capsule 3) — more control, cheaper, open source
- You compare both (capsule 4) — data-driven architecture decisions
- You design prompts for images (capsule 5) — visual prompt engineering
- You edit and create variations (capsule 6) — inpainting, outpainting, variations
- You integrate into pipelines (capsule 7) — generation + vision in one flow
- You build the project (capsule 8) — a complete Image Generator with fallback
The Era of Image Generation
A brief history
AI image generation has a short but intense history:
2021 — DALL-E (original)
OpenAI presents a model that generates images from text.
Impressive but limited. Access for researchers only.
2022 — Stable Diffusion + Midjourney + DALL-E 2
Stable Diffusion is released as open source → community explosion.
Midjourney gains popularity for artistic quality.
DALL-E 2 launches a public API.
2023 — DALL-E 3 + SDXL
DALL-E 3 integrated into ChatGPT. Dramatically improved prompt understanding.
SDXL (Stable Diffusion XL) raises the quality of the open source ecosystem.
2024 — SD3 + Flux + massive competition
Stable Diffusion 3 with a DiT (Diffusion Transformer) architecture.
Flux emerges as a high-quality alternative.
Image generation becomes a commodity: more options, lower costs.
Current state of the market
The image generation market has split into two large camps:
Closed models (API):
- DALL-E 3 (OpenAI) — native integration with GPT, consistent quality
- Imagen (Google) — access via Vertex AI, less popular than DALL-E
- Ideogram — strong at typography within images
Open models (open source):
- Stable Diffusion XL, SD3 — the open source standard
- Flux — high quality, fast growth
- Playground v2 — oriented toward photorealistic quality
For this module we focus on DALL-E 3 and Stable Diffusion because:
- They represent the two paradigms (closed vs open)
- They have accessible APIs (OpenAI and Replicate respectively)
- They cover 90% of production use cases
- The techniques you learn are transferable to any other model
What Image Generation Is
Definition
Image generation is the process of creating new images from natural-language text descriptions (prompts). Diffusion models — the dominant architecture — work by progressively "cleaning up" noise until they form an image coherent with the prompt.
You don't need to understand the math of diffusion models to use the APIs. But you do need to understand the parameters that control generation, because they determine the quality, style, cost and time of your results.
How it works (high level)
Prompt (text) → Encoder → Latent space → Diffusion process → Decoder → Image (pixels)
- The prompt is converted into a numeric vector (embedding)
- The model starts from random noise in a compressed (latent) space
- At each diffusion step, the model reduces the noise guided by the prompt's embedding
- More steps = more detail, but more time and cost
- The resulting latent is decoded into an image in pixels
Main providers
| Provider | Model | API type | Approx. cost | Main features |
|---|---|---|---|---|
| OpenAI | DALL-E 3 | Official REST API | $0.04-0.08/img | High quality, superior semantic understanding, rewrites prompts |
| Replicate | SDXL, SD3, Flux | REST API | $0.002-0.02/img | Multiple models, pay per second of GPU |
| Stability AI | SD3, SDXL | Official REST API | $0.01-0.03/img | Direct access to the developer of Stable Diffusion |
| Midjourney | Midjourney v6 | Discord only (no REST API) | $10-60/mo subscription | Exceptional artistic quality, but no programmatic API |
| Imagen 2 | Vertex AI | Variable | Integration with Google Cloud | |
| Hugging Face | Multiple | Inference API | Free (limited) | Ideal for experimenting, requires GPU for production |
Production Use Cases
1. Marketing and advertising
Generate variations of ad creatives for A/B testing. A single prompt can produce dozens of images to test in campaigns. The cost of a graphic designer for 50 variations is prohibitive; with DALL-E 3 it costs less than $4.
campaign_prompts = [
"Minimalist tech product on a white background, soft studio lighting",
"Tech product on a wooden desk, natural window light",
"Tech product floating against a blue-to-purple gradient background",
]2. E-commerce
Generate product images from different angles, backgrounds or usage contexts. It eliminates the need for photo shoots for each color variation or scenario.
backgrounds = ["white studio background", "modern kitchen", "minimalist desk", "outdoor nature"]
for background in backgrounds:
prompt = f"Stainless steel water bottle on {background}, professional product photography"3. Prototyping and design
Generate interface mockups, visual concepts for presentations, storyboards for video. It lets you iterate fast before involving a designer for the final result.
4. Editorial content
Illustrations for articles, thumbnails for videos, images for social media. Publications that used to require generic stock photos now have unique illustrations aligned with the content.
5. Multimodal pipelines
The most advanced use case: combining generation with analysis. Real examples:
- Document → Diagram: Analyze a PDF with GPT-4 Vision → extract structure → generate a diagram with DALL-E
- Description → Image → Analysis: Generate an image → analyze it with vision → iterate on the prompt
- Visual chatbot: User describes what they want → system generates → user gives feedback → system adjusts
def pipeline_document_to_diagram(document_path: str) -> str:
text = analyze_document(document_path)
image_prompt = f"Technical diagram showing: {text[:500]}"
image_url = generate_image(image_prompt)
return image_url6. Gaming and entertainment
Asset generation: textures, backgrounds, conceptual sprites, concept art. Especially useful in early development phases where many fast iterations are needed.
Module 4 Roadmap
| # | Capsule | What you'll learn | Duration |
|---|---|---|---|
| 01 | Introduction (this one) | Landscape, models, use cases, setup | 8 min |
| 02 | DALL-E 3 (OpenAI) | Full API, parameters, formats, costs | 8 min |
| 03 | Stable Diffusion | Replicate API, SDXL, advanced parameters | 8 min |
| 04 | DALL-E vs SD Comparison | Benchmark, costs, architecture decisions | 7 min |
| 05 | Prompts for images | Visual prompt engineering, negative prompts | 7 min |
| 06 | Editing and variations | Inpainting, outpainting, variations | 6 min |
| 07 | Pipeline integration | Generation + vision, composite flows | 6 min |
| 08 | Project: Image Generator | DALL-E + SD fallback, complete system | 10 min |
Estimated total module duration: ~60 min
Module Objectives
By completing this module you'll be able to:
| Objective | Capsule where it's covered |
|---|---|
| Use the DALL-E 3 API with all its parameters (size, quality, style) | 02 |
| Integrate Stable Diffusion via Replicate with fine parameter control | 03 |
| Compare DALL-E vs SD and choose the right one for the context | 04 |
| Design effective prompts for image generation | 05 |
| Apply editing (inpainting) and generate variations | 06 |
| Build pipelines that combine generation with analysis | 07 |
| Implement an Image Generator with fallback between providers | 08 |
Specific technical skills
- Calls to the OpenAI Images API (
client.images.generate) - Calls to Replicate for Stable Diffusion
- Handling URL and base64 responses
- Downloading and storing generated images
- Negative prompts for Stable Diffusion
- Sampling parameters (steps, guidance_scale, scheduler)
- Error handling for content policy violations
- Fallback pattern across multiple providers
Prerequisites
What you need from earlier modules
This module assumes you completed modules 1-3 or have equivalent experience:
| Prerequisite | Where you learned it |
|---|---|
Configuring OPENAI_API_KEY and using the OpenAI() client | Module 1 |
| Understanding the difference between analysis and generation models | Module 1 |
| Working with images in base64 and URLs | Module 2 |
| Handling API errors (rate limits, content policy) | Module 2 |
| Building reusable functions that call APIs | Module 3 |
Quick check
If you can run this code without issues, you're ready:
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": "Respond only 'OK'"}],
max_tokens=5
)
print(response.choices[0].message.content)If you get OK, your API key works and you can continue.
Technical Setup
Dependencies
pip install openai>=1.0.0 replicate requests Pillow| Package | What we use it for |
|---|---|
openai | DALL-E 3 API |
replicate | Stable Diffusion API via Replicate |
requests | Downloading images from URLs |
Pillow | Basic image manipulation (resize, display) |
API Keys
export OPENAI_API_KEY="sk-..."
export REPLICATE_API_TOKEN="r8_..."OPENAI_API_KEY is required for capsules 2, 4-8. REPLICATE_API_TOKEN is required for capsules 3, 4, 7-8. You can create a free account at replicate.com that includes initial credits.
Setup verification
import openai
import replicate
print(f"openai version: {openai.__version__}")
print(f"replicate imported successfully")
client = openai.OpenAI()
response = client.images.generate(
model="dall-e-3",
prompt="A red square on a white background",
size="1024x1024",
quality="standard",
n=1
)
print(f"DALL-E 3 works. URL: {response.data[0].url[:80]}...")If you get a URL, your setup is complete for the whole module.
Recommended file structure
module-04-workspace/
├── generated/ ← Generated images are saved here
├── 01_intro.py
├── 02_dalle3.py
├── 03_stable_diffusion.py
├── 04_comparison.py
├── 05_prompts.py
├── 06_editing.py
├── 07_pipelines.py
└── 08_project.py
What This Module Does NOT Cover
It's equally important to know what we will not cover to avoid false expectations:
| Topic | Why we don't cover it | Where to learn it |
|---|---|---|
| Fine-tuning image models | Requires GPUs, datasets, and is a specialized field | Stability AI documentation, ML courses |
| Training from scratch | Out of scope for AI Engineers (vs ML Engineers) | Original papers, deep learning courses |
| Midjourney (Discord interface) | No REST API; not integrable in code | Midjourney documentation |
| Video generation | Still immature for production; will change fast | Future module or guide update |
| Advanced ControlNet | Requires local GPU infrastructure | Stable Diffusion documentation |
| Legal copyright aspects | Depends on jurisdiction and changes constantly | Consult up-to-date legal sources |
Mental Model: Analysis vs Generation
In modules 1-3 you worked the image → text flow (analysis). Now we work text → image (generation). This change of direction has practical implications:
| Aspect | Analysis (M1-3) | Generation (M4) |
|---|---|---|
| Input | Existing image | Text (prompt) |
| Output | Text (description, data) | New image |
| Determinism | High (same image = same analysis) | Low (same prompt ≠ same image) |
| Control | Specific questions | Prompt engineering + parameters |
| Cost | Based on input/output tokens | Based on resolution and quality |
| Typical errors | Hallucinations in text | Visual artifacts, malformed hands |
| Evaluation | Compare against ground truth | Subjective + defined criteria |
Non-determinism is the most important difference. Every time you generate an image with the same prompt, you get a different result. This is a feature (variety) and a bug (inconsistency) at the same time. You'll learn to manage it with seeds and parameters.
Self-Assessment
Before moving on to capsule 2, check that you can answer these questions:
Conceptual:
- What are the two main paradigms of image generation (closed vs open)?
- Why is image generation non-deterministic?
- How does using DALL-E 3 vs Stable Diffusion differ from a developer's perspective?
Practical:
4. Do you have your OPENAI_API_KEY configured and working?
5. Do you have REPLICATE_API_TOKEN configured (or do you know how to get it)?
6. Can you name three production use cases for image generation?
Expected answers
- Closed: DALL-E 3 (OpenAI) — proprietary API, high quality, less control. Open: Stable Diffusion — open source, more control (negative prompts, parameters), runnable locally.
- Because the diffusion process starts from different random noise on each run. The seed controls that noise; same seed = same image.
- DALL-E 3: a single API (
openai), few parameters, consistent quality, more expensive. SD: multiple APIs (Replicate, Stability, local), many parameters (steps, cfg_scale, scheduler, negative_prompt), cheaper, requires more tuning. - Run the verification code from the Setup section.
- Sign up at replicate.com → Settings → API tokens.
- Marketing (creative variations), e-commerce (product photos), multimodal pipelines (document → diagram).
Evidence of Success
By the end of this module, you'll know you succeeded if:
- ✅ You can generate an image with DALL-E 3 from a prompt and receive the result's URL or Base64
- ✅ You can generate an image with Stable Diffusion via Replicate, configuring parameters like steps, cfg_scale and negative prompt
- ✅ You know the differences in cost, control and quality between DALL-E 3 and Stable Diffusion
- ✅ You know how to design effective prompts for image generation (structure, descriptors, styles)
- ✅ You can implement a fallback that tries DALL-E 3 first and falls back to Stable Diffusion if it fails
- ✅ Your Image Generator with Fallback works: it generates images, verifies quality, and tracks costs
Summary
- Image generation is the modality that creates visual content from text — the complement of the analysis you mastered in M1-M3.
- The two main paradigms are: closed (DALL-E 3 — proprietary API, high quality, less control) and open (Stable Diffusion — open source, more parameters, cheaper).
- Real use cases include: marketing (creative variations), e-commerce (product photos), multimodal pipelines (document → diagram).
- This module covers: the DALL-E 3 API, Stable Diffusion via Replicate, visual prompt engineering, editing/variations, and generation pipelines.
- The module's project is an Image Generator with Fallback that tries multiple providers, verifies quality and tracks costs.
Module Terminology
| Term | Meaning |
|---|---|
| Prompt | Text that describes the image to generate |
| Negative prompt | Text that describes what should NOT appear (Stable Diffusion only) |
| Seed | Number that controls the initial noise; same seed = same result |
| Steps | Number of diffusion steps; more steps = more detail |
| CFG Scale / Guidance | How strictly the model follows the prompt (higher = more literal) |
| Inpainting | Editing a specific area of an existing image |
| Outpainting | Expanding an image beyond its original borders |
| Variations | Generating images similar to a reference image |
| Revised prompt | The prompt rewritten internally by DALL-E 3 |
| Scheduler | The sampling algorithm in Stable Diffusion (K_EULER, DPM, etc.) |
Additional Resources
- OpenAI Images API — Official DALL-E 3 documentation
- Replicate — Stable Diffusion and other models via API
- Stability AI — Creators of Stable Diffusion
- Prompt Engineering for Image Generation (OpenAI Cookbook) — Advanced examples
- Hugging Face Diffusers — Library for running diffusion models locally