Module 4: Background Tasks

Introduction to Module 4: Background Tasks

Next →

Overview

Your Task Manager API from Module 3 responds professionally: differentiated schemas, CSV export with streaming, custom headers, complete OpenAPI documentation. But there's one pattern you still don't handle: operations that take time and shouldn't block the response to the client.

Picture this: a user creates a task with POST /tasks. Your API creates the task, saves it, and returns the JSON to the client. But you also want to send a notification email to the team. Sending the email takes 2-3 seconds. Does the client have to wait those extra 2-3 seconds? No. The response to the client should be immediate — "task created, here's the data." The email goes out afterward, in the background. The client never even notices there's an extra process running.

This module teaches you to run operations after returning the response. FastAPI has BackgroundTasks built in for lightweight work. For heavier work, or work that needs retries, scheduling, and monitoring, there's Celery and RQ. You'll learn both levels: what FastAPI gives you natively, and when to scale up to an external task queue.

Where are we in the guide?

You're in Module 4 of 6 of the FastAPI Advanced Features guide:

Module 1: Dependency Injection ✅ (completed)
    → Depends(), sub-dependencies, yield patterns

Module 2: APIRouter and Middleware ✅ (completed)
    → Modular routers, custom middleware, events

Module 3: Advanced Response Models ✅ (completed)
    → Multiple schemas, streaming, custom responses

Module 4: Background Tasks ← YOU ARE HERE
    → BackgroundTasks, patterns, intro to Celery/RQ

Module 5: WebSockets and File Uploads
Module 6: Project — Complete Task Manager API

Cumulative progression

Module 1: DI for DRY code
    ↓
Module 2: + Modular structure with routers and middleware
    ↓
Module 3: + Professional responses and streaming
    ↓
Module 4: + Background processing  ← HERE
    ↓
Module 5: + Real-time communication + files
    ↓
Module 6: Integration project

Module 3 gave you control over what you return. Module 4 gives you control over what happens after you return.

What you already know vs what's new

What you already know

From Modules 1-3 you bring:

  • ✅ Dependency injection with Depends()
  • ✅ APIRouter with a modular structure
  • ✅ Middleware for logging and timing
  • ✅ response_model with differentiated schemas
  • ✅ StreamingResponse for large data
  • ✅ Custom headers and responses
  • ✅ OpenAPI documentation with the responses parameter

What's new in this module

  • 🆕 FastAPI's BackgroundTasks — running functions after the response
  • 🆕 Adding multiple background tasks
  • 🆕 Passing arguments to background tasks
  • 🆕 Patterns: fire-and-forget, task chaining, status tracking
  • 🆕 Error handling in the background (try/except + logging)
  • 🆕 When BackgroundTasks isn't enough
  • 🆕 An introduction to Celery with Redis as the broker
  • 🆕 An introduction to RQ (Redis Queue)
  • 🆕 A comparison: BackgroundTasks vs Celery vs RQ

From "synchronous" to "respond fast, process later"

AspectBefore (your current API)After (this module)
POST /tasksCreates task → respondsCreates task → responds → sends email in the background
PATCH /tasks/{id} (status=completed)Updates → respondsUpdates → responds → log + cleanup in the background
DELETE /tasks/{id}Deletes → respondsDeletes → responds → audit log in the background
Slow operationsBlock the clientRun after the response
Errors in secondary operationsFail the main operationGet logged without affecting the client

Why background tasks?

Scenario 1: Notifications

WITHOUT background tasks:
  POST /tasks → create task → send email (3s) → respond to the client
  Total time: 3.2 seconds
  The client waits 3 extra seconds for something that doesn't affect them

WITH background tasks:
  POST /tasks → create task → respond to the client → send email in the background
  Response time: 0.2 seconds
  The email goes out afterward without the client ever noticing

Scenario 2: Data processing

WITHOUT background tasks:
  POST /reports → generate CSV (10s) → respond to the client
  The client waits 10 seconds staring at a spinner

WITH background tasks:
  POST /reports → respond 202 Accepted → generate CSV in the background
  The client gets immediate confirmation
  They can check GET /reports/{id}/status to see the progress

Scenario 3: Cleanup and maintenance

WITHOUT background tasks:
  DELETE /tasks/{id} → delete → clear cache → update index → respond
  Every extra operation adds latency

WITH background tasks:
  DELETE /tasks/{id} → delete → respond → clear cache in the background → update index in the background
  Immediate response, cleanup doesn't block

The golden rule

If an operation doesn't affect the result the client receives, do it in the background.

The client calling POST /tasks needs to know whether the task got created. They don't need to wait for the email to be sent. The client calling DELETE /tasks/{id} needs to know whether it got deleted. They don't need to wait for the audit log to be written.

Sync vs async in FastAPI

Before we get into background tasks, let's clear up the difference between sync and async in FastAPI:

# Sync — runs in a thread pool
@app.get("/sync")
def sync_endpoint():
    time.sleep(1)  # Blocks the thread, but not the event loop
    return {"mode": "sync"}

# Async — runs in the event loop
@app.get("/async")
async def async_endpoint():
    await asyncio.sleep(1)  # Blocks nothing
    return {"mode": "async"}

BackgroundTasks works with both:

  • Sync functions as a background task → they run in a thread pool
  • Async functions as a background task → they run in the event loop

You don't need to make your background tasks async for them to work. FastAPI handles both cases.

Module objective

By the end of this module you'll be able to:

  • ✅ Use FastAPI's BackgroundTasks for post-response operations
  • ✅ Add multiple background tasks to a single endpoint
  • ✅ Pass arguments to background tasks
  • ✅ Implement patterns: fire-and-forget, task chaining, status tracking
  • ✅ Handle errors in background tasks (try/except + logging)
  • ✅ Understand the limitations of BackgroundTasks (in-process, no retries, no persistence)
  • ✅ Know Celery with Redis as the broker (basic setup, the anatomy of a task)
  • ✅ Know RQ as a lightweight alternative to Celery
  • ✅ Decide when to use BackgroundTasks vs Celery vs RQ
  • ✅ Integrate background tasks into your Task Manager API

Prerequisites

For this module you need:

  • Module 3 completed — a Task Manager API with professional responses
  • Basic Python logging — we'll use logging to see the output of background tasks
  • uvicorn running with --reload

Quick check

cd fastapi-advanced
source venv/bin/activate
uvicorn app.main:app --reload

Check that your Module 3 API works: GET /tasks, POST /tasks, GET /tasks/export/csv.

What if you don't meet one of the prerequisites?

Missing prerequisiteWhat to do
You didn't do Module 3You can still follow this module — the examples are self-contained. But the final project assumes the M3 structure
You don't know loggingThe capsules explain what you need. import logging; logger = logging.getLogger(__name__) is all it takes

Module roadmap

CapsuleTopicWhat you'll learn
01Introduction (this capsule)Context, why background tasks, sync vs async
02BackgroundTasks in FastAPIBasic usage, multiple tasks, arguments, common patterns
03Patterns and error handlingChaining, dependencies, error handling, status tracking
04Celery and RQ — an introductionWhen to scale, Redis as the broker, a comparison, a basic example
05Project: Background ProcessingEmail notifications, cleanup of old tasks, status tracking

Learning flow

First you'll learn the basic mechanism of BackgroundTasks in FastAPI: how to add functions that run after the response, pass arguments, and add multiple tasks (Capsule 02). Then you'll go deeper into advanced patterns: chaining tasks, handling errors without affecting the client, and tracking the state of background tasks with an in-memory pattern (Capsule 03). Next you'll meet the alternatives for when BackgroundTasks isn't enough: Celery with Redis for heavy tasks with retries and scheduling, and RQ as a lighter alternative (Capsule 04). At the end, you'll pull it all together in your Task Manager API: simulated notifications when tasks are created/completed, periodic cleanup of completed tasks, and status tracking (Capsule 05).

The progression is: basic mechanism → advanced patterns → scale when you need to → integrate.

Capsule by capsule

Capsule 02 — BackgroundTasks in FastAPI: The mechanism is simple: you inject BackgroundTasks as a parameter, you call background_tasks.add_task(func, arg1, arg2), and FastAPI runs func(arg1, arg2) after sending the response. You'll implement email simulation, logging, and cleanup with this pattern.

Capsule 03 — Patterns and error handling: Background tasks can fail — and when they do, the client already got their response. So how do you find out? Logging. How do you retry? Manually (BackgroundTasks has no retries). How do you track state? With an in-memory dict that stores the progress. These patterns are the difference between background tasks that work and background tasks you can actually debug.

Capsule 04 — Celery and RQ: BackgroundTasks has limitations: tasks are lost if the server restarts, there's no automatic retry, there's no scheduling, and they run in the same process. Celery solves all of that with separate workers and Redis as the broker. RQ is a simpler alternative. This capsule is conceptual/introductory — not a complete guide to Celery.

Capsule 05 — Project: Your Task Manager API gains background processing: creating a task "sends" an email (a log), completing a task gets recorded in an audit log, tasks completed more than 30 days ago can be cleaned up automatically, and a /tasks/background/status endpoint lets you check the state of background operations.

The mental model: the request lifecycle with background tasks

Without background tasks:
┌─────────┐     ┌─────────────┐     ┌──────────┐
│ Request │ ──▶ │  Endpoint   │ ──▶ │ Response │
│         │     │ (all of it) │     │          │
└─────────┘     └─────────────┘     └──────────┘

With background tasks:
┌─────────┐     ┌─────────────┐     ┌──────────┐     ┌──────────────┐
│ Request │ ──▶ │  Endpoint   │ ──▶ │ Response │ ──▶ │  Background  │
│         │     │ (fast part) │     │ (to the client) │ (slow part) │
└─────────┘     └─────────────┘     └──────────┘     └──────────────┘
                                         ↑
                                    The client already has
                                    their response here

The endpoint does the essential work (create the task, validate the data, prepare the response), sends the response to the client, and then runs the secondary operations. The client never waits on the secondary operations.

Connection with the project

Your Task Manager API from Module 3 is going to evolve like this:

Before (Module 3)

POST /tasks → creates task → returns TaskPublic (done)
PATCH /tasks/{id} → updates → returns TaskPublic (done)
DELETE /tasks/{id} → deletes → returns confirmation (done)
No post-response processing
No operation logging
No notifications

After (Module 4)

POST /tasks → creates task → returns TaskPublic → email notification in the background
PATCH /tasks/{id} (→completed) → updates → returns → audit log in the background
DELETE /tasks/{id} → deletes → returns → cleanup in the background
GET /tasks/background/status → the state of background operations
Structured logging of every background task
Error handling that doesn't affect the client

How to use this guide in Module 4

A practical approach

Every capsule includes complete code you should run. The cycle is:

  1. Read the explanation — understand the concept and the pattern
  2. Write the code — background tasks require watching the server console
  3. Watch the console — background task logs show up in the uvicorn terminal
  4. Check the timing — the response arrives before the background task finishes
  5. Cause errors on purpose — what happens if the background task fails? Does the client find out?

Estimated time

  • Introduction capsule (this one): 15-20 minutes
  • Technical capsules (02-04): 30-45 minutes each
  • Project (05): 45-60 minutes
  • Total for Module 4: 2-2.5 hours

Signs of success

By the end of this module (all 5 capsules), you'll know you succeeded if:

  • ✅ You can add background tasks to any endpoint with BackgroundTasks
  • ✅ You understand that the response reaches the client before the background tasks finish
  • ✅ You know how to handle errors in the background without affecting the response to the client
  • ✅ You can implement a status tracking pattern for background tasks
  • ✅ You understand when BackgroundTasks is enough and when you need Celery/RQ
  • ✅ You know Celery's basic architecture (worker + broker + task)
  • ✅ You can choose between BackgroundTasks, Celery, and RQ for a given case
  • ✅ Your Task Manager API has notifications and cleanup in the background

Summary

  • This is Module 4 of 6, focused on background processing
  • The key principle: if an operation doesn't affect the response to the client, do it in the background
  • FastAPI's BackgroundTasks runs functions after sending the response
  • It works with sync functions (thread pool) and async ones (event loop)
  • The limitations of BackgroundTasks: in-process, no retries, no persistence, lost if the server restarts
  • Celery and RQ are the alternatives for heavy tasks with retries, scheduling, and monitoring
  • The basic pattern: endpoint → response → background processing → logging
  • Error handling in the background: try/except + logging — the client never sees the errors

Additional resources

  1. FastAPI - Background Tasks — The official BackgroundTasks tutorial
  2. Starlette - Background Tasks — Starlette's documentation (the foundation of BackgroundTasks in FastAPI)
  3. Python logging — Python's logging module (used to observe background tasks)
  4. Celery Documentation — A distributed task queue for Python
  5. RQ (Redis Queue) — A lightweight alternative to Celery
  6. Real Python - Async IO — A complete guide to async/await in Python

Next capsule: BackgroundTasks in FastAPI — You'll learn to inject BackgroundTasks, add functions that run after the response, and the most common patterns.

← PreviousProject: Professional Responses for the Task Manager APINext →BackgroundTasks in FastAPI