3. Stable Diffusion

Description

Stable Diffusion is the most important open source image generation model in the ecosystem. Unlike DALL-E 3 (closed, single API), Stable Diffusion can be run locally, via third-party APIs (Replicate, Stability AI), or on GPU services like Hugging Face. It offers more control than DALL-E 3: negative prompts, sampling parameters (steps, guidance scale, scheduler), seeds for reproducibility, and a massive community of fine-tuned models.

Why it matters: Stable Diffusion is the preferred option when you need fine control, low costs at volume, or advanced capabilities like inpainting with ControlNet. For an AI Engineer, mastering DALL-E 3 and Stable Diffusion gives you the flexibility to choose based on context: DALL-E for simple quality, SD for control and cost.

In this capsule we use Replicate as the main API because it offers access to multiple SD versions (SDXL, SD3) without needing your own GPU, with a pay-per-second-of-compute model.

Accessing Stable Diffusion

Access options

Why Replicate in this guide

Replicate offers the best balance for learning: you don't need a GPU, the cost is low, the API is simple, and you can switch between models (SDXL, SD3, Flux) by changing a string. In production, the decision may be different (your own hosting if the volume justifies it).

Setup

Installation

pip install replicate requests Pillow

API Key

export REPLICATE_API_TOKEN="r8_..."

Get your token at replicate.com → Settings → API tokens. New accounts include free credits.

Verification

import replicate

output = replicate.run(
    "stability-ai/sdxl:39ed52f2a40e4be0e682a3e7d0645ef75e93dd3bd23a9c5fe73e589d1a3adc3b",
    input={
        "prompt": "A red square on a white background, simple geometric shape",
        "width": 512,
        "height": 512,
        "num_inference_steps": 10
    }
)
print(f"Stable Diffusion works. Output: {output}")

Available Models on Replicate

Main versions

Which to choose

• SDXL: The standard. Good quality/cost/speed balance. Start here.
• SD3: When you need the maximum quality of SD and can wait longer.
• Flux Schnell: When speed is a priority (previews, prototypes).
• Flux Dev: Direct competitor to DALL-E 3 in quality.

Base Function with SDXL

import replicate
import requests
from pathlib import Path

SDXL_MODEL = "stability-ai/sdxl:39ed52f2a40e4be0e682a3e7d0645ef75e93dd3bd23a9c5fe73e589d1a3adc3b"

def generate_sd(
    prompt: str,
    negative_prompt: str = "blurry, low quality, distorted, deformed, ugly, bad anatomy",
    width: int = 1024,
    height: int = 1024,
    steps: int = 25,
    guidance_scale: float = 7.5,
    seed: int | None = None,
    save_path: str | None = None,
) -> dict:
    input_params = {
        "prompt": prompt,
        "negative_prompt": negative_prompt,
        "width": width,
        "height": height,
        "num_inference_steps": steps,
        "guidance_scale": guidance_scale,
    }

    if seed is not None:
        input_params["seed"] = seed

    output = replicate.run(SDXL_MODEL, input=input_params)
    image_url = output[0] if isinstance(output, list) else str(output)

    result = {
        "url": image_url,
        "prompt": prompt,
        "negative_prompt": negative_prompt,
        "width": width,
        "height": height,
        "steps": steps,
        "guidance_scale": guidance_scale,
        "seed": seed,
    }

    if save_path:
        img_data = requests.get(image_url, timeout=60).content
        Path(save_path).parent.mkdir(parents=True, exist_ok=True)
        Path(save_path).write_bytes(img_data)
        result["saved_to"] = save_path

    return result

Example usage:

result = generate_sd(
    prompt="A futuristic city at sunset with flying cars, digital art style, highly detailed",
    negative_prompt="blurry, low quality, watermark, text overlay",
    width=1024,
    height=768,
    steps=30,
    guidance_scale=8.0,
    save_path="generated/sd_city.png"
)

print(f"URL: {result['url'][:80]}...")
print(f"Saved to: {result.get('saved_to')}")

Parameters in Detail

prompt

The text that describes the image. Unlike DALL-E 3, Stable Diffusion doesn't rewrite your prompt. What you send is exactly what the model uses. This means the quality of the prompt is your direct responsibility.

Tips for good prompts in SD:

basic_prompt = "a cat"

improved_prompt = (
    "A majestic orange tabby cat sitting on a velvet cushion, "
    "soft studio lighting, shallow depth of field, "
    "professional pet photography, 4k, highly detailed"
)

Prompts in SD benefit from including: artistic style, lighting, desired quality, and level of detail.

negative_prompt

What you do not want in the image. It's one of the most powerful controls in SD and doesn't exist in DALL-E 3.

negatives_general = (
    "blurry, low quality, distorted, deformed, ugly, "
    "bad anatomy, bad proportions, watermark, text, "
    "signature, logo, cropped, out of frame"
)

negatives_for_people = (
    "blurry, low quality, deformed, bad anatomy, "
    "extra fingers, mutated hands, extra limbs, "
    "disfigured, cross-eyed, ugly face"
)

negatives_for_landscapes = (
    "blurry, low quality, oversaturated, underexposed, "
    "watermark, text, people, humans, buildings"
)

result = generate_sd(
    prompt="Serene mountain lake at golden hour, cinematic photography, 8k",
    negative_prompt=negatives_for_landscapes,
    save_path="generated/sd_lake.png"
)

num_inference_steps (steps)

How many diffusion steps to run. More steps = more detail but more time and cost.

def compare_steps(prompt: str, step_values: list[int]) -> list[dict]:
    results = []
    for steps in step_values:
        result = generate_sd(
            prompt=prompt,
            steps=steps,
            seed=42,
            save_path=f"generated/steps_{steps}.png"
        )
        results.append({"steps": steps, **result})
        print(f"Steps={steps}: generated")
    return results

compare_steps(
    "Medieval castle on a cliff, dramatic clouds, fantasy art",
    [10, 20, 30, 50]
)

Note: We use seed=42 so the only variable is the steps, not the initial noise.

guidance_scale (CFG Scale)

Controls how strictly the model follows your prompt. A high value generates images more faithful to the prompt but potentially less "creative" or natural.

def compare_guidance(prompt: str, scales: list[float]) -> list[dict]:
    results = []
    for cfg in scales:
        result = generate_sd(
            prompt=prompt,
            guidance_scale=cfg,
            seed=42,
            save_path=f"generated/cfg_{cfg:.1f}.png"
        )
        results.append({"guidance_scale": cfg, **result})
        print(f"CFG={cfg}: generated")
    return results

compare_guidance(
    "A cyberpunk samurai in neon-lit Tokyo streets, rain, reflections",
    [3.0, 7.0, 10.0, 15.0]
)

seed

The seed controls the initial noise of the diffusion process. Same seed + same parameters = same image. Essential for reproducibility.

result_a = generate_sd("Mountain landscape", seed=12345, save_path="generated/seed_a.png")
result_b = generate_sd("Mountain landscape", seed=12345, save_path="generated/seed_b.png")

result_c = generate_sd("Mountain landscape", seed=99999, save_path="generated/seed_c.png")

width and height

Supported resolutions depend on the model. SDXL works best with resolutions that add up to ~1 megapixel:

def generate_widescreen(prompt: str) -> dict:
    return generate_sd(
        prompt=prompt,
        width=1344,
        height=768,
        save_path="generated/sd_widescreen.png"
    )

generate_widescreen("Panoramic view of a vast alien desert with two suns setting")

scheduler

The sampling algorithm. Different schedulers produce slightly different results:

def compare_schedulers(prompt: str, schedulers: list[str]) -> list[dict]:
    results = []
    for sched in schedulers:
        output = replicate.run(
            SDXL_MODEL,
            input={
                "prompt": prompt,
                "negative_prompt": "blurry, low quality",
                "scheduler": sched,
                "seed": 42,
                "num_inference_steps": 25,
            }
        )
        url = output[0] if isinstance(output, list) else str(output)
        results.append({"scheduler": sched, "url": url})
        print(f"{sched}: generated")
    return results

compare_schedulers(
    "Watercolor painting of a Japanese garden in autumn",
    ["K_EULER", "DPMSolverMultistep", "HeunDiscrete"]
)

Using Other Models on Replicate

Flux Schnell (ultra-fast generation)

FLUX_SCHNELL = "black-forest-labs/flux-schnell"

def generate_flux_fast(prompt: str, save_path: str | None = None) -> dict:
    output = replicate.run(
        FLUX_SCHNELL,
        input={
            "prompt": prompt,
            "num_outputs": 1,
            "aspect_ratio": "1:1",
            "output_format": "png",
        }
    )
    image_url = output[0] if isinstance(output, list) else str(output)

    result = {"url": image_url, "model": "flux-schnell", "prompt": prompt}

    if save_path:
        img_data = requests.get(image_url, timeout=60).content
        Path(save_path).parent.mkdir(parents=True, exist_ok=True)
        Path(save_path).write_bytes(img_data)
        result["saved_to"] = save_path

    return result

result = generate_flux_fast(
    "A minimalist logo for a tech startup, clean lines, blue gradient",
    save_path="generated/flux_logo.png"
)

Stability AI API (alternative to Replicate)

import requests as req

STABILITY_API_KEY = "sk-..."
STABILITY_URL = "https://api.stability.ai/v2beta/stable-image/generate/sd3"

def generate_stability_sd3(prompt: str, save_path: str) -> str:
    response = req.post(
        STABILITY_URL,
        headers={
            "Authorization": f"Bearer {STABILITY_API_KEY}",
            "Accept": "image/*",
        },
        files={"none": ""},
        data={
            "prompt": prompt,
            "negative_prompt": "blurry, low quality",
            "output_format": "png",
            "aspect_ratio": "1:1",
        },
        timeout=60,
    )
    response.raise_for_status()
    Path(save_path).parent.mkdir(parents=True, exist_ok=True)
    Path(save_path).write_bytes(response.content)
    return save_path

Comparison with DALL-E 3

Summary: DALL-E 3 is "easy mode" — few parameters, consistent results. SD is "manual mode" — more parameters, more control, but requires more knowledge.

Troubleshooting

Exercises

Exercise 1: Effective negative prompt

Create a function that builds negative prompts based on the image type (person, landscape, product, art). Test with the same positive prompt and different negative prompts.

NEGATIVE_PRESETS = {
    "person": (
        "blurry, low quality, deformed, bad anatomy, extra fingers, "
        "mutated hands, extra limbs, disfigured, cross-eyed, ugly face, "
        "bad proportions, watermark, text"
    ),
    "landscape": (
        "blurry, low quality, oversaturated, underexposed, "
        "watermark, text, people, humans, buildings, ugly, "
        "cropped, out of frame"
    ),
    "product": (
        "blurry, low quality, distorted, shadows, "
        "cluttered background, watermark, text, logo, "
        "bad lighting, grainy"
    ),
    "art": (
        "blurry, low quality, ugly, amateur, "
        "watermark, text, signature, frame border, "
        "photorealistic, photograph"
    ),
}

def get_negative_prompt(image_type: str) -> str:
    return NEGATIVE_PRESETS.get(image_type, NEGATIVE_PRESETS["art"])

prompt = "Portrait of a young woman in a garden, golden hour light"

for img_type in ["person", "landscape", "art"]:
    neg = get_negative_prompt(img_type)
    result = generate_sd(
        prompt=prompt,
        negative_prompt=neg,
        seed=42,
        save_path=f"generated/negative_{img_type}.png"
    )
    print(f"Type={img_type}: generated")

Exercise 2: Seed exploration

Generate 5 images with the same prompt but different seeds. Then pick the best one and regenerate at higher quality (more steps).

import random

def explore_seeds(prompt: str, count: int = 5) -> list[dict]:
    results = []
    seeds = [random.randint(0, 2**32 - 1) for _ in range(count)]

    for i, seed in enumerate(seeds):
        result = generate_sd(
            prompt=prompt,
            seed=seed,
            steps=15,
            save_path=f"generated/seed_explore_{i}.png"
        )
        results.append({"index": i, "seed": seed, **result})
        print(f"Seed {seed}: generated/seed_explore_{i}.png")

    return results

results = explore_seeds(
    "A cozy bookshop interior with warm lighting, fantasy illustration style"
)

best_seed = results[0]["seed"]
print(f"\nRegenerating seed {best_seed} at high quality...")

final = generate_sd(
    prompt="A cozy bookshop interior with warm lighting, fantasy illustration style",
    seed=best_seed,
    steps=40,
    guidance_scale=8.0,
    save_path="generated/seed_final_hq.png"
)
print(f"HQ result: {final.get('saved_to')}")

Exercise 3: Multi-model function

Create a function generate_image_replicate that accepts a model parameter and routes to the right model (SDXL, Flux Schnell, Flux Dev).

MODEL_REGISTRY = {
    "sdxl": {
        "id": "stability-ai/sdxl:39ed52f2a40e4be0e682a3e7d0645ef75e93dd3bd23a9c5fe73e589d1a3adc3b",
        "input_map": lambda p, neg: {
            "prompt": p,
            "negative_prompt": neg,
            "width": 1024,
            "height": 1024,
            "num_inference_steps": 25,
        },
    },
    "flux-schnell": {
        "id": "black-forest-labs/flux-schnell",
        "input_map": lambda p, neg: {
            "prompt": p,
            "num_outputs": 1,
            "aspect_ratio": "1:1",
            "output_format": "png",
        },
    },
    "flux-dev": {
        "id": "black-forest-labs/flux-dev",
        "input_map": lambda p, neg: {
            "prompt": p,
            "num_outputs": 1,
            "aspect_ratio": "1:1",
            "output_format": "png",
            "guidance": 3.5,
        },
    },
}


def generate_image_replicate(
    prompt: str,
    model: str = "sdxl",
    negative_prompt: str = "blurry, low quality",
    save_path: str | None = None,
) -> dict:
    if model not in MODEL_REGISTRY:
        raise ValueError(f"Model '{model}' not supported. Options: {list(MODEL_REGISTRY.keys())}")

    config = MODEL_REGISTRY[model]
    inputs = config["input_map"](prompt, negative_prompt)
    output = replicate.run(config["id"], input=inputs)
    url = output[0] if isinstance(output, list) else str(output)

    result = {"url": url, "model": model, "prompt": prompt}

    if save_path:
        img_data = requests.get(url, timeout=60).content
        Path(save_path).parent.mkdir(parents=True, exist_ok=True)
        Path(save_path).write_bytes(img_data)
        result["saved_to"] = save_path

    return result

for model_name in ["sdxl", "flux-schnell"]:
    result = generate_image_replicate(
        "Minimalist tech logo, clean design",
        model=model_name,
        save_path=f"generated/multi_{model_name}.png"
    )
    print(f"{model_name}: {result.get('saved_to')}")

Exercise 4: Batch generation with variable parameters

Generate a series of images varying a single parameter at a time (steps, guidance, seed) while keeping the others fixed. Save a JSON report with the results.

import json
from datetime import datetime

def parameter_sweep(
    prompt: str,
    param_name: str,
    param_values: list,
    base_params: dict | None = None,
    output_dir: str = "generated/sweep"
) -> list[dict]:
    base = base_params or {
        "negative_prompt": "blurry, low quality, distorted",
        "width": 1024,
        "height": 1024,
        "steps": 25,
        "guidance_scale": 7.5,
        "seed": 42,
    }

    Path(output_dir).mkdir(parents=True, exist_ok=True)
    results = []

    for val in param_values:
        params = {**base, param_name: val}
        filename = f"{output_dir}/{param_name}_{val}.png"

        result = generate_sd(prompt=prompt, save_path=filename, **params)
        result["varied_param"] = param_name
        result["varied_value"] = val
        results.append(result)
        print(f"{param_name}={val}: {filename}")

    report = {
        "timestamp": datetime.now().isoformat(),
        "prompt": prompt,
        "varied_parameter": param_name,
        "values": param_values,
        "base_params": base,
        "results": [
            {"value": r["varied_value"], "url": r["url"][:80], "saved": r.get("saved_to")}
            for r in results
        ],
    }
    report_path = f"{output_dir}/{param_name}_report.json"
    Path(report_path).write_text(json.dumps(report, indent=2, ensure_ascii=False))
    print(f"\nReport: {report_path}")

    return results

parameter_sweep(
    "A steampunk airship flying over Victorian London, detailed illustration",
    param_name="guidance_scale",
    param_values=[3.0, 5.0, 7.5, 10.0, 12.0]
)

Additional Resources

1. Replicate — Platform for running ML models via API
2. Replicate - SDXL — SDXL model documentation on Replicate
3. Stability AI — Creators of Stable Diffusion
4. Stability AI API Docs — Stability API reference
5. Hugging Face Diffusers — For running diffusion models locally