Module 4: The N+1 problem with SQLAlchemy

Eager loading anti-patterns: when the cure is worse

Capsule overview

After capsules 04-06 you have a strong idea in your head: "lazy loading is bad, eager is good". That's a useful simplification while you learn, but false when it becomes dogma.

Poorly applied eager loading has its own set of problems:

  • Over-fetching: you load megabytes of relationships the client never reads.
  • Unnecessary eager: you enable selectinload on a query that returns 1 entity, spending an extra query for nothing.
  • Indiscriminate selectinload('*'): "load everything" sounds convenient and ends up loading polymorphic relationships, giant joins, and sensitive data the endpoint shouldn't even touch.
  • Eager loading on small queries: for 5 rows, lazy is faster than eager (the extra round-trip of the IN list).

This capsule teaches you to:

  • Detect over-fetching by counting real bytes/rows vs useful ones.
  • Apply raiseload in production so any accidental lazy fails visibly instead of blowing up silently.
  • Use defer and load_only to load only the necessary columns.
  • Avoid selectinload('*') and other dangerous wildcards.
  • Recognize the balance: when lazy really is the right choice.

By the end, you'll be able to defend both "add selectinload" and "remove selectinload" in a code review, depending on the case.


Mental model: the pendulum of balance

Imagine a pendulum with two extremes:

Extreme A (pure lazy): everything is lazy default. The endpoint fires N+1, high latency, unnecessary queries on demand.

Extreme B (total eager): you load all the relationships of everything, always. A "list 50 authors" query loads their books, their reviews, their tags, their awards, their collaborators — 80,000 rows to return 50 names.

The center: you load eagerly what the specific endpoint needs, and nothing more.

Capsule 06 pushed you from extreme A to the center. This capsule prevents you from overshooting to extreme B.

The single criterion

For each relationship in your query, ask yourself: does the endpoint's response USE this data?

  • Yes → load eagerly.
  • No → don't load. If the lazy fires accidentally, let it fail (with raiseload).

If your query loads Author.books "just in case" and the endpoint only returns author.name, you're making a wasted query and byte transfer per request.


Anti-pattern 1: over-fetching in list endpoints

The mistake

@app.get("/authors")
async def list_authors(session: AsyncSession = Depends(get_db)):
    stmt = (
        select(Author)
        .options(
            selectinload(Author.books),       # ← loaded but not used
            selectinload(Author.awards),      # ← loaded but not used
            selectinload(Author.collaborators), # ← loaded but not used
        )
        .limit(50)
    )
    authors = (await session.scalars(stmt)).all()
    return [{"id": a.id, "name": a.name} for a in authors]

What does it do? It loads 50 authors + their books (~1000 books) + their awards (~200) + their collaborators (~500). Total: ~1750 rows transferred.

What does it return? 50 dicts with id and name. Only 100 useful values. Useful/total efficiency: ~5%.

Detection

Enable echo=True and make the request. You'll see:

SELECT authors.id, authors.name FROM authors LIMIT 50
SELECT books.id, ..., books.author_id FROM books WHERE books.author_id IN (id1, ..., id50)
SELECT awards.id, ..., awards.author_id FROM awards WHERE awards.author_id IN (id1, ..., id50)
SELECT collaborators.id, ..., collaborators.author_id FROM collaborators WHERE collaborators.author_id IN (id1, ..., id50)

Four queries. The first is the only necessary one. The other three are waste.

Solution

Remove the selectinload for relationships the endpoint doesn't use:

@app.get("/authors")
async def list_authors(session: AsyncSession = Depends(get_db)):
    stmt = select(Author).limit(50)
    authors = (await session.scalars(stmt)).all()
    return [{"id": a.id, "name": a.name} for a in authors]

A single query. 50 rows. 100% efficiency.

When this anti-pattern appears

  • Copy-paste code: someone copied an endpoint that returned full author details, then edited it to return only name but forgot to remove the selectinload.
  • Poorly applied preventive defense: "let's add eager loading just in case, it doesn't break anything."
  • A refactor that left zombie code: the endpoint's logic was changed but the loading options weren't adjusted.

How to prevent it in code review

Key question: "which fields of relationship X does this endpoint's response use?". If the dev can't answer, the selectinload is unnecessary.


Anti-pattern 2: selectinload('*') and wildcards

The mistake

SQLAlchemy allows loading "all relationships" with a wildcard:

from sqlalchemy.orm import selectinload

stmt = select(Author).options(selectinload('*'))

What does it load? All the relationships defined on Author: books, awards, collaborators, profile, reviews_received, mentions, etc. If your model has 10 relationships, that's 10 additional queries.

Why it's dangerous

1. Uncontrollable cardinality. You have no visibility into how many children you load. A large 1:N relationship (like Author.book_pages with 50,000 pages per author) gets loaded along with all the others.

2. Sensitive data. If you have an Author.private_notes relationship you normally don't want to expose, selectinload('*') loads it. If your serializer has a bug and exposes them, that's a leak.

3. Fragile refactor. If someone adds a new relationship to the model (for example Author.audit_log with thousands of entries), your endpoint that used selectinload('*') now loads the audit log on every request, without anyone changing that code.

4. Unstable plan. PostgreSQL runs N additional queries whose cardinality you can't predict. Your p99 will fluctuate.

Solution

List the relationships you need explicitly:

stmt = select(Author).options(
    selectinload(Author.books),
    selectinload(Author.profile),
)

Verbose but explicit. If someone adds Author.audit_log tomorrow, it doesn't affect this endpoint.

Dangerous variant: joinedload('*')

Just as bad, worse because of the cartesian explosion:

stmt = select(Author).options(joinedload('*'))

If you have 5 1:N relationships with 10 elements each, that's 1 × 10 × 10 × 10 × 10 × 10 = 100,000 duplicated rows for 1 author. Catastrophic.

Simple rule

Never use selectinload('*') or joinedload('*') in production code. Always list explicit relationships. Verbose > ambiguous.


Anti-pattern 3: eager loading where lazy was optimal

The case: 1 entity, a rarely accessed relationship

@app.get("/authors/{author_id}")
async def get_author(author_id: int, session: AsyncSession = Depends(get_db)):
    stmt = (
        select(Author)
        .options(selectinload(Author.books))  # ← do I really need it?
        .where(Author.id == author_id)
    )
    author = await session.scalar(stmt)
    if not author:
        raise HTTPException(404)
    return {"name": author.name, "country": author.country}

The endpoint DOES NOT USE author.books. The selectinload fires a useless query:

-- Query 1 (necessary)
SELECT * FROM authors WHERE id = 42

-- Query 2 (WASTE)
SELECT * FROM books WHERE author_id IN (42)

If tolkien has 50 books, you transferred 50 book rows you never used.

Why it hurts

  • 1 extra round-trip to the database.
  • Memory spent building 50 Book objects.
  • If the relationship is a large 1:N (1000+ rows), measurable latency.

Solution

Remove the selectinload. If the endpoint only returns Author fields, don't load books.

stmt = select(Author).where(Author.id == author_id)

A single query, 1 row, no waste.

When eager IS worth it in 1-entity endpoints

  • The response USES the relationship. Example: /authors/{id}/with-books that returns {"name": ..., "books": [...]}. Here selectinload(Author.books) is correct.
  • To avoid a later lazy load with MissingGreenlet. If at some point in the endpoint you have to access author.books (for example in validation), eager avoids the error.

Rule: eager if the endpoint uses the relationship. Lazy (no options) if not.


Anti-pattern 4: loading columns you don't use

selectinload and joinedload load all the columns of the related entity by default. If you only need book.title but the Book model has 20 columns (including a PDF BLOB), you bring everything.

Solution 1: load_only

from sqlalchemy.orm import selectinload, load_only

stmt = (
    select(Author)
    .options(
        selectinload(Author.books).load_only(Book.title, Book.id),
    )
    .where(Author.id == 42)
)

Result: query 2 brings only id and title of books, not all the columns. Each row smaller, less memory, faster.

Solution 2: defer

defer is the opposite: it loads all columns EXCEPT the specified ones (deferred). Useful when there are 1-2 heavy columns (like blobs).

from sqlalchemy.orm import defer

stmt = (
    select(Book)
    .options(defer(Book.pdf_content))  # ← don't bring pdf_content
    .where(Book.id == 42)
)

If you later need book.pdf_content, SQLAlchemy loads it lazy (in sync) or raises an error in async. For async, configure raiseload:

.options(defer(Book.pdf_content, raiseload=True))

When it matters

  • Tables with large BLOB/TEXT columns (PDFs, images, huge JSON).
  • Listing endpoints where you only want summary fields.
  • Public APIs where every byte transferred costs (mobile latency, bandwidth costs).

When it doesn't matter

  • Tables with all small columns (id, name, dates).
  • Internal low-traffic endpoints.

If your Book has id, title, author_id (all small), load_only doesn't help you. If it also has full_text with 50KB per row, it does.


raiseload: production discipline

raiseload is the option that says: "if anyone tries to lazy-load this relationship, raise an exception".

Configuration

Per-query:

from sqlalchemy.orm import raiseload

stmt = (
    select(Author)
    .options(
        selectinload(Author.books),
        raiseload(Author.awards),  # ← if anyone accesses author.awards, error
    )
    .where(Author.id == 42)
)

As a safety wildcard (with care):

stmt = (
    select(Author)
    .options(
        selectinload(Author.books),
        raiseload('*'),  # ← any OTHER relationship raises if accessed lazy
    )
)

raiseload('*') affects only the relationships not listed explicitly in the options. The ones you load eagerly keep working.

Why it's valuable discipline

Without raiseload: an accidental lazy access in async raises MissingGreenlet with a cryptic stack trace. In sync, it fires an unexpected lazy query — a silent N+1.

With raiseload: an accidental lazy access raises a visible and descriptive error:

sqlalchemy.exc.InvalidRequestError: 'Author.awards' is not available due to lazy='raise'

The error tells you exactly which relationship you tried to access. You go to the code and decide: either load it eagerly, or don't access it.

Model-level configuration

To force raiseload always on a sensitive relationship:

class Author(Base):
    __tablename__ = "authors"

    id: Mapped[int] = mapped_column(primary_key=True)
    name: Mapped[str]

    # Relationships with raise default
    private_notes: Mapped[list["Note"]] = relationship(lazy="raise")
    audit_log: Mapped[list["AuditEntry"]] = relationship(lazy="raise")

    # Normal relationships (lazy default)
    books: Mapped[list["Book"]] = relationship()

Any access to author.private_notes or author.audit_log without explicit eager loading raises an error. You force the developer to be explicit.

When to apply raiseload

  • Production always: enable raiseload('*') on critical queries. If something slips through as lazy, it fails in CI before production.
  • Sensitive relationships at the model level: private data, audit logs, anything that should NOT be accessed without explicit intent.
  • In tests: combined with nplusone, a double safety net.

When NOT to apply

  • Exploratory dev: you're iterating, you don't want every lazy access to blow up.
  • Trivial relationships: if the model has 2 small relationships and you always load them together, raiseload adds noise without value.

Anti-pattern 5: nested eager loading without thinking

The case

stmt = select(Author).options(
    selectinload(Author.books)
        .selectinload(Book.reviews)
            .selectinload(Review.author)        # ← review.author is a User
                .selectinload(User.profile)
                    .selectinload(Profile.tags)
)

Five levels of eager loading. For 1 author, it could load thousands of rows in cascade.

Why it's a problem

  • Each selectinload is an additional SQL query.
  • For a "give me an author" request, you end up with 6 queries.
  • If the cardinalities are large, you transfer enormous amounts of data to reconstruct an entire graph.

Diagnostic question

Does the endpoint USE all those fields in every request? If the response is:

{
  "name": "tolkien",
  "books": [
    {"title": "...", "review_count": 100}
  ]
}

— you only need Author.books with len(book.reviews). That's 2 queries (author + books, then you compute review_count with a func.count or by accessing the preloaded list).

Solution

Design the response model first. Then design the query that loads exactly what the response needs. If the response doesn't use review.author, don't load it.

stmt = select(Author).options(
    selectinload(Author.books).selectinload(Book.reviews)
)
# ↑ 3 queries. Enough for review_count = len(book.reviews).

If you need book.reviews_aggregated_score, better a func.avg(Review.rating) with group_by(Book.id) than loading all the whole reviews.

Practical rule

Never nest selectinload more than 2-3 levels. If you need more, consider:

  • A specific endpoint for that view (not generic).
  • SQL aggregations (COUNT, AVG, SUM) instead of loading entire graphs.
  • Materialized views or caching for read-heavy data (other guides in the path).

Why this matters in real work

1. Over-fetching is the inverse N+1.

If you obsess only over N+1, you end up loading everything in excess. The correct metric is useful bytes vs transferred bytes. An endpoint with 1 query that brings 10MB to return 1KB is just as bad as one with 50 queries.

2. raiseload is what separates senior code from junior.

Junior: "I added eager loading, problem solved". Senior: "I added explicit eager loading + raiseload('*') so any future lazy access fails in CI". The second is production discipline.

3. load_only is the tool nobody knows and every serious team uses.

If your API has listing endpoints with limited fields (the common case), load_only reduces transferred bytes significantly. The difference between bringing 5 columns or 25 columns on each row adds up over thousands of requests.

4. Productive code review.

When a colleague submits a PR with selectinload('*') or eager loading without justification, you can ask for an explanation with concrete data: "which fields of awards does the response use? If you don't use any, remove that eager loading."


Traps and common mistakes

Mistake 1 (conceptual): assuming "more eager = better"

Symptom: "I added selectinload to all the model's relationships, now everything will be loaded."

Why it's wrong: you're loading data the endpoint never uses. Memory + transferred bytes + additional queries for nothing.

How to distinguish: the principle is "load what the endpoint needs". If your response returns only name, don't load relationships. If it returns books, load books (not awards or collaborators).

Mistake 2 (practical): selectinload('*') "because it's convenient"

Symptom: "When I don't know which relationships I'll need, I put selectinload('*')."

Why it's dangerous: it loads polymorphic, sensitive, or enormous-cardinality relationships without you noticing. Fragile refactor when new relationships are added.

How to fix it: always be explicit. If you really "don't know what you need", the problem is the endpoint's design, not the loading. Design specific endpoints instead of "give me everything".

Mistake 3 (conceptual): raiseload only in tests

Symptom: "raiseload is in my tests but not in production."

Why it's suboptimal: raiseload in tests prevents bugs in new code. raiseload in production prevents bugs in code that slipped through (example: a fix's rollback). Enabling it in production is the final net.

How to apply it correctly: raiseload('*') on critical queries in production too. If something accidentally stays lazy, it raises a visible error (which your monitoring catches) instead of silently running 50 extra queries.

Mistake 4 (practical): load_only with a relationship without thinking

Symptom: "load_only(Book.title) and then I need book.author_id and it crashes."

Why it happens: load_only loads ONLY the listed columns. Any other requires a lazy load (which fails in async, adds a query in sync).

How to fix it: list all the columns you'll use. Including FK keys needed for future joins. SQLAlchemy is strict.

.options(selectinload(Author.books).load_only(Book.id, Book.title, Book.author_id))

Mistake 5 (conceptual): thinking defer always improves performance

Symptom: "I added defer to all the large columns, but my endpoint is slower."

Why it happens: defer doesn't save IO in the main query — those columns are omitted from the SELECT but PostgreSQL still reads the full row from disk. You only save bytes transferred to the Python client.

When defer helps: TOAST columns (PostgreSQL stores BLOBs separately). For small columns, defer gives no measurable benefit.

How to distinguish: measure with EXPLAIN (ANALYZE, BUFFERS) before and after. If Buffers doesn't change, defer doesn't help on that column.

Mistake 6 (structural): massive eager loading in a generic endpoint

Symptom: "My /entities/{id} endpoint loads all the possible relationships because it's generic."

Why it's a problem: a generic endpoint that serves multiple cases ends up loading too much for some and not enough for others. Inefficient for everyone.

How to fix it: specific endpoints for each view. /authors/{id}/summary (only name), /authors/{id}/full (with books), /authors/{id}/with-stats (with aggregated counts). Each one loads what's necessary.


Exercises

Exercise 1: detect over-fetching

Look at this endpoint and answer:

  1. How many queries does it fire?
  2. How many rows does it transfer (estimate)?
  3. How many are useful (used in the response)?
@app.get("/authors-list")
async def authors_list(session: AsyncSession = Depends(get_db)):
    stmt = (
        select(Author)
        .options(
            selectinload(Author.books).selectinload(Book.reviews),
            selectinload(Author.awards),
            joinedload(Author.profile),
        )
        .limit(100)
    )
    authors = (await session.scalars(stmt)).unique().all()
    return [{"id": a.id, "name": a.name} for a in authors]

Assume: 100 authors, 30 books average per author, 10 reviews per book, 5 awards per author.

See solution

Analysis:

1. Queries fired: 4.

  • Query 1: SELECT authors + JOIN profiles (joinedload) → 100 rows (1:1).
  • Query 2: SELECT books WHERE author_id IN (...) → 100 × 30 = 3,000 rows.
  • Query 3: SELECT reviews WHERE book_id IN (...) → 3,000 × 10 = 30,000 rows.
  • Query 4: SELECT awards WHERE author_id IN (...) → 100 × 5 = 500 rows.

2. Rows transferred: ~33,600.

100 authors + 100 profiles + 3,000 books + 30,000 reviews + 500 awards = 33,600.

3. Useful rows: 100.

The response only returns id and name of the 100 authors. 0.3% efficiency.

Correct solution:

@app.get("/authors-list")
async def authors_list(session: AsyncSession = Depends(get_db)):
    stmt = select(Author).limit(100)
    authors = (await session.scalars(stmt)).all()
    return [{"id": a.id, "name": a.name} for a in authors]

1 query, 100 rows, 100% efficiency.

If you suspect the endpoint will use profile in the future:

.options(joinedload(Author.profile))  # 1:1, optimal

But do NOT add selectinload(Author.books) or selectinload(Author.awards) "just in case". Wait until the endpoint actually uses them.

Exercise 2: apply raiseload('*') correctly

Refactor this query so that any lazy access to relationships not listed fails visibly:

stmt = (
    select(Author)
    .options(selectinload(Author.books))
    .where(Author.id == 42)
)
See solution
from sqlalchemy.orm import selectinload, raiseload

stmt = (
    select(Author)
    .options(
        selectinload(Author.books),
        raiseload('*'),  # any OTHER relationship raises an error if accessed
    )
    .where(Author.id == 42)
)

Behavior:

  • author.books works (loaded eagerly).
  • author.awards, author.profile, author.collaborators, etc., raise InvalidRequestError if accessed.

Test that validates:

@pytest.mark.asyncio
async def test_only_books_loaded(session):
    author = await session.scalar(stmt)

    # This passes:
    assert len(author.books) >= 0

    # This fails with a clear error:
    with pytest.raises(InvalidRequestError, match="raiseload"):
        _ = author.awards

Why it's valuable discipline:

If in the future someone edits the endpoint and adds author.awards to the response without adding selectinload(Author.awards), the test fails with a clear message: "you're accessing awards without loading it eager." It forces explicit loading.

Exercise 3: use load_only to reduce transfer

Your Book model has 15 columns, including pdf_content (BLOB) and summary (TEXT 5KB). The endpoint only returns id, title, and published_year. Refactor the query to minimize transferred bytes.

stmt = (
    select(Book)
    .where(Book.author_id == 42)
)
books = (await session.scalars(stmt)).all()
return [{"id": b.id, "title": b.title, "year": b.published_year} for b in books]
See solution
from sqlalchemy.orm import load_only

stmt = (
    select(Book)
    .options(load_only(Book.id, Book.title, Book.published_year))
    .where(Book.author_id == 42)
)
books = (await session.scalars(stmt)).all()
return [{"id": b.id, "title": b.title, "year": b.published_year} for b in books]

Generated SQL:

Before:

SELECT books.id, books.title, books.author_id, books.published_year, books.pdf_content,
       books.summary, books.isbn, ... (all the columns)
FROM books WHERE author_id = 42

After:

SELECT books.id, books.title, books.published_year
FROM books WHERE author_id = 42

Benefit:

  • If pdf_content is 500KB and there are 20 books → you save 10MB of transfer.
  • If summary is 5KB → you save an additional 100KB.
  • For listing endpoints on mobile APIs, this is the difference between a 1s and a 10s load.

Careful:

If you later try to access book.pdf_content:

  • In sync: it fires a lazy load → additional query.
  • In async: it raises MissingGreenlet (because it's a lazy load).

To force strict behavior:

.options(load_only(Book.id, Book.title, Book.published_year, raiseload=True))

(raiseload=True makes the unlisted columns raise an error if accessed, instead of lazy loading.)

Exercise 4: identify and remove selectinload('*')

You're reviewing a PR with this:

@app.get("/users/{user_id}/dashboard")
async def dashboard(user_id: int, session: AsyncSession = Depends(get_db)):
    stmt = (
        select(User)
        .options(selectinload('*'))
        .where(User.id == user_id)
    )
    user = await session.scalar(stmt)
    return {
        "name": user.name,
        "order_count": len(user.orders),
        "favorite_count": len(user.favorites),
    }

What do you suggest? Assume User has relationships: orders, favorites, addresses, payment_methods, notifications (with thousands of rows), audit_log (with tens of thousands), chat_messages (with hundreds of thousands).

See solution

Suggested comment:

Hey, two important issues with selectinload('*'):

1. It loads dangerous relationships:

  • notifications (~10,000 rows per user)
  • audit_log (~50,000 rows)
  • chat_messages (~500,000 rows)

That's ~600k rows per request to a dashboard endpoint. The p95 will be miserable.

2. Fragile refactor: If someone adds a new relationship to User tomorrow, this endpoint loads it without anyone noticing.

Suggestion: be explicit about only what's necessary:

stmt = (
    select(User)
    .options(
        selectinload(User.orders),
        selectinload(User.favorites),
        raiseload('*'),  # any other lazy relationship → visible error
    )
    .where(User.id == user_id)
)

Benefits:

  • Only 3 queries (user + orders + favorites).
  • raiseload('*') prevents someone from adding user.audit_log to the response without adding the selectinload.
  • Predictable cardinality.

Additional improvement with counts:

If the endpoint only needs len(user.orders) and len(user.favorites) (not the lists themselves), consider using SQL aggregations instead of loading all the rows:

from sqlalchemy import func

stmt = (
    select(
        User,
        func.count(Order.id).label("order_count"),
        func.count(Favorite.id).label("favorite_count"),
    )
    .outerjoin(Order)
    .outerjoin(Favorite)
    .where(User.id == user_id)
    .group_by(User.id)
)

This brings 1 row with the counts directly. Without transferring the complete relationships. If the counts are the only thing you need, this version is 10-100x faster.

Why this comment is senior:

  • Identifies the concrete risk of selectinload('*').
  • Shows the real cost with numbers (estimated rows).
  • Gives the exact code for the minimal fix (raiseload('*') + explicit).
  • Suggests an additional improvement (aggregated counts) that demonstrates deep understanding of the case.

Exercise 5: refactor a "generic endpoint" into specific endpoints

You have this endpoint that serves 3 different views:

@app.get("/authors/{author_id}")
async def get_author(
    author_id: int,
    detail_level: str = "summary",  # "summary", "with-books", "full"
    session: AsyncSession = Depends(get_db),
):
    options = []
    if detail_level in ("with-books", "full"):
        options.append(selectinload(Author.books))
    if detail_level == "full":
        options.extend([
            selectinload(Author.awards),
            selectinload(Author.profile),
        ])

    stmt = select(Author).options(*options).where(Author.id == author_id)
    author = await session.scalar(stmt)
    if not author:
        raise HTTPException(404)

    if detail_level == "summary":
        return {"name": author.name}
    elif detail_level == "with-books":
        return {"name": author.name, "books": [{"title": b.title} for b in author.books]}
    else:  # full
        return {
            "name": author.name,
            "books": [{"title": b.title} for b in author.books],
            "awards": [a.name for a in author.awards],
            "bio": author.profile.bio if author.profile else None,
        }

Refactor it into 3 separate endpoints. Justify pros and cons.

See solution
# Endpoint 1: summary
@app.get("/authors/{author_id}/summary")
async def author_summary(author_id: int, session: AsyncSession = Depends(get_db)):
    stmt = select(Author).where(Author.id == author_id)
    author = await session.scalar(stmt)
    if not author:
        raise HTTPException(404)
    return {"name": author.name}


# Endpoint 2: with books
@app.get("/authors/{author_id}/with-books")
async def author_with_books(author_id: int, session: AsyncSession = Depends(get_db)):
    stmt = (
        select(Author)
        .options(selectinload(Author.books))
        .where(Author.id == author_id)
    )
    author = await session.scalar(stmt)
    if not author:
        raise HTTPException(404)
    return {
        "name": author.name,
        "books": [{"title": b.title} for b in author.books],
    }


# Endpoint 3: full
@app.get("/authors/{author_id}/full")
async def author_full(author_id: int, session: AsyncSession = Depends(get_db)):
    stmt = (
        select(Author)
        .options(
            selectinload(Author.books),
            selectinload(Author.awards),
            joinedload(Author.profile),
        )
        .where(Author.id == author_id)
    )
    author = await session.scalar(stmt)
    if not author:
        raise HTTPException(404)
    return {
        "name": author.name,
        "books": [{"title": b.title} for b in author.books],
        "awards": [a.name for a in author.awards],
        "bio": author.profile.bio if author.profile else None,
    }

Pros of the refactor:

  1. Explicit URLs: clients know what they'll receive before calling.
  2. Independently cacheable: you can cache /summary aggressively and /full with a low TTL.
  3. Optimal loading per endpoint: each one loads exactly what it returns.
  4. Clear tests: one test per endpoint, specific validation.
  5. Granular permissions: you can have different auth for /summary (public) and /full (authenticated).
  6. Auto-generated documentation (FastAPI): OpenAPI shows 3 different endpoints with different response models. Clearer for consumers.

Cons:

  1. More routes: 3 endpoints vs 1. More code.
  2. The client changes the URL instead of a parameter: if the client wanted to change dynamically, it now changes the URL. A small additional cost.

Decision: the pros outweigh the cons almost always. The "generic endpoint with detail_level" is typically an over-engineering anti-pattern disguised as flexibility.

Exception: if you have 50 different views (not 3), consider GraphQL instead of 50 REST endpoints. But that's another debate.

Exercise 6: measure the impact of raiseload('*') in CI

Configure raiseload('*') on a main query of your app and run the test suite. Document:

  1. How many tests failed.
  2. How many were accidental lazy loads (real bugs) vs intentional (need eager).
  3. Which you'd fix with eager loading vs which with whitelist or refactor.
See solution

There's no single solution. Analysis structure:

## raiseload('*') audit: [endpoint or module]

**Setup:**
- Query under analysis: select(Author).options(selectinload(Author.books), raiseload('*'))
- Tests exercising the endpoint: N

**Results:**

| Test | Status | Diagnosis | Action |
|------|--------|-------------|--------|
| test_get_author | ✅ passes | N/A | None |
| test_author_with_awards | ❌ fails on author.awards | Accidental lazy — the response uses awards | Add selectinload(Author.awards) |
| test_author_serializer | ❌ fails on author.profile | Serializer touches profile silently | Add joinedload(Author.profile) |
| test_legacy_compat | ❌ fails on author.collaborators | Legacy code no longer used | Refactor: remove access to collaborators |

**Conclusions:**
- 2 real bugs detected (accidental lazy loads the endpoint uses).
- 1 zombie code identified (refactor pending).
- 0 false positives.

**Actions taken:**
1. Added selectinload(Author.awards) to the query.
2. Added joinedload(Author.profile) to the query.
3. Removed legacy code from test_legacy_compat.
4. raiseload('*') merged to master.

**Result:**
- Endpoint ends with 3 fixed queries (author + books + awards). Profile via JOIN.
- Any future lazy access to unlisted relationships → CI breaks.
- Zero possible silent N+1s.

If your audit finds no accidental lazy, either your endpoint is already perfect (congratulations), or your tests don't exercise the endpoint in depth (more likely). Add tests that touch every field of the response and run again.


Summary and next step

In this capsule you learned:

  • Poorly applied eager loading has its own cost: over-fetching, unnecessary queries, fragile refactors with selectinload('*').
  • raiseload('*') is the discipline that fails any accidental lazy access with a visible error.
  • load_only and defer control which columns to bring, not just which relationships.
  • Typical anti-patterns: eager loading "just in case", indiscriminate selectinload('*'), eager loading where lazy was optimal, deep nesting without need.
  • Specific endpoint > generic: better 3 endpoints (summary, with-books, full) than one endpoint with a detail_level parameter.
  • The correct metric: useful bytes vs transferred bytes. Rows in the response vs rows loaded.

Before moving on you should be able to:

  • Detect over-fetching by enabling echo=True and comparing rows loaded vs rows returned in the response.
  • Apply raiseload('*') on critical queries to force discipline.
  • Use load_only for listing queries where you only need some columns.
  • Refactor "generic" endpoints into specific endpoints with optimal eager loading per case.
  • Argue in code review both for and against eager loading depending on the case.

Next capsule — Project: eliminating N+1 from the bookstore. Now you apply everything you've learned to the path's canonical endpoint: /books-with-author?author_name=tolkien from module 1. You'll measure the baseline (current queries, p95 latency), apply the correct combination of selectinload + raiseload, validate with nplusone that no N+1s remain, and report the improvement with concrete numbers. The result is a portfolio-worthy "before/after" commit that serves you in senior interviews.


Resources

  1. SQLAlchemy 2.0 — raiseload — the official reference. Section on how to prevent accidental lazy loads.
  2. SQLAlchemy 2.0 — load_only — column control.
  3. SQLAlchemy 2.0 — defer — for heavy columns like BLOBs.
  4. SQLAlchemy 2.0 — Wildcard Loading Strategies — explains selectinload('*') and why you must use it with care.
  5. PostgreSQL Documentation — TOAST — to understand why defer helps with large columns (TOAST overhead).
  6. Mike Bayer — "Performance" (official FAQ) — the official section on performance, includes a discussion of over-fetching and recommended patterns.
  7. Asif Muhammad — "Optimizing FastAPI database queries" — a practical case applied to the path's stack with a discussion of the lazy/eager balance.

Module 4 — Database Performance & Query Tuning Guide