Module 2: Doing Soft Deletes Right
Implementing `deleted_at` in PostgreSQL
Capsule overview
You already decided soft delete is the right answer for your table. Now it's time to land it in pure PostgreSQL: which column to add, with what type, what index has to back it, how to migrate from DELETE to UPDATE, and how to measure the impact before and after. This capsule is 100% PostgreSQL — no SQLAlchemy yet. The ORM integration arrives in 04.
The goal is that you end up with a pattern you understand at the engine level: why TIMESTAMPTZ NULL and not a boolean, why a partial index is 50x faster than a normal index over the same column, and what exactly happens with MVCC when you run UPDATE SET deleted_at = NOW(). This level of detail is what separates someone who "knows SQL" from someone who knows how to design for production.
You're going to set up a table with 1M rows, apply the pattern, measure the degradation of the anti-pattern (no partial index) versus the correct implementation, and leave everything documented so you can reproduce it. It's the exercise that trains you to make schema changes with evidence, not with faith.
The column: why deleted_at TIMESTAMPTZ NULL
The canonical pattern is:
ALTER TABLE tasks ADD COLUMN deleted_at TIMESTAMPTZ NULL;
Three implicit decisions worth understanding:
1. TIMESTAMPTZ (not BOOLEAN, not DATE, not TIMESTAMP)
Why not is_deleted BOOLEAN:
- You lose information: when was it deleted? That's useful for reports, audits, "deletions in the last 30 days," or cleanup jobs.
- It costs the same on disk (a boolean takes 1 byte, a timestamp 8 bytes — but row alignment usually evens the cost out).
- It forces you to add a
deleted_atcolumn anyway when you discover you need the date. Start straight away with the option that covers both cases.
Why not TIMESTAMP (without time zone):
- In multi-region SaaS, timestamps without TZ generate subtle bugs. A row marked as deleted at 23:50 in Madrid shows up as deleted at 17:50 in Mexico City if you interpret the
TIMESTAMPwith the client's TZ. PostgreSQL recommendsTIMESTAMPTZfor any timestamp with real-world meaning. TIMESTAMPTZalways stores UTC internally and converts on display according to the client's TZ. Predictable behavior.
Why not DATE:
- You lose precision. If two users delete within the same second, you can't tell who went first or reconstruct the ordering.
Conclusion: TIMESTAMPTZ NULL is the default for any soft delete or audit timestamp column.
2. NULL (not DEFAULT NOW(), not NOT NULL)
The default has to be NULL because NULL is the representation of "not deleted." A concrete value means "deleted at this moment." The deleted_at IS NULL convention becomes idiomatic and every query follows it.
If you set DEFAULT NOW(), every new row would be born "deleted," which makes no sense. If you set NOT NULL with a default of '1970-01-01' (a sentinel), every query would have to compare against the sentinel instead of against NULL, and you'd lose the clear semantics of "absence of value = active."
3. No foreign key to a "deletion reasons" table
Don't add deleted_reason TEXT or deleted_by_user_id BIGINT to the same table. That data belongs in an audit log (module 3), not in the operational table. Mixing audit with current state pollutes the model and grows without control.
Exception: if the domain requires showing the user "this task was deleted for X reason" as part of the recovery UI, then yes — but document that it's a conscious exception.
The partial index: why it's part of the pattern, not optional
When you add deleted_at, all of your app's queries change from:
SELECT id, title FROM tasks WHERE author_id = 42 ORDER BY created_at DESC LIMIT 50;
To:
SELECT id, title FROM tasks
WHERE author_id = 42 AND deleted_at IS NULL
ORDER BY created_at DESC LIMIT 50;
If your existing index is CREATE INDEX idx_tasks_author_created ON tasks (author_id, created_at DESC), that index includes rows with a non-null deleted_at. PostgreSQL walks them and discards them in the post-scan filter. When the delete ratio is high (>50%), you walk twice the rows to return half.
The partial index eliminates that work:
CREATE INDEX idx_tasks_author_created_active
ON tasks (author_id, created_at DESC)
WHERE deleted_at IS NULL;
Now the index contains only active rows. PostgreSQL can use it when the query includes WHERE deleted_at IS NULL (the planner detects the implication). The walk is exactly the same as before you introduced soft delete.
A deeper treatment of how PostgreSQL decides to use a partial index is in guide #12, module 3 (Advanced indexing: partial, covering, expression indexes). In this capsule we apply it without re-explaining it at the planner level.
Why this isn't an optional optimization
Without a partial index, soft delete has a cost that grows with the delete ratio. You'll see it measured below. The operational rule is: when you add deleted_at, you add the partial index at the same time. It's not a follow-up. It's the other half of the pattern.
Worked case: setup, before, after
We're going to set up a table with 1M rows, 60% soft-deleted, measure common queries with a normal index vs a partial index, and document the speedup.
Setup
-- setup_soft_delete_demo.sql
DROP TABLE IF EXISTS tasks;
CREATE TABLE tasks (
id BIGSERIAL PRIMARY KEY,
author_id BIGINT NOT NULL,
title TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
deleted_at TIMESTAMPTZ NULL
);
-- Seed 1M tasks spread across 1000 authors
INSERT INTO tasks (author_id, title, created_at)
SELECT
(random() * 999 + 1)::bigint,
'task_' || g,
now() - (random() * interval '365 days')
FROM generate_series(1, 1000000) g;
-- Mark 60% as soft-deleted (with deleted_at spread over the last year)
UPDATE tasks
SET deleted_at = created_at + (random() * interval '180 days')
WHERE id IN (
SELECT id FROM tasks ORDER BY random() LIMIT 600000
);
-- Force fresh statistics
ANALYZE tasks;
createdb softdelete_demo
psql -d softdelete_demo -f setup_soft_delete_demo.sql
# CREATE TABLE
# INSERT 0 1000000
# UPDATE 600000
# ANALYZE
# Took ~25s on my MacBook Pro M2
Verify the ratio:
SELECT
COUNT(*) FILTER (WHERE deleted_at IS NULL) AS active,
COUNT(*) FILTER (WHERE deleted_at IS NOT NULL) AS deleted,
ROUND(
100.0 * COUNT(*) FILTER (WHERE deleted_at IS NOT NULL) / COUNT(*),
1
) AS deleted_pct
FROM tasks;
-- Expected output:
-- active | deleted | deleted_pct
-- --------+---------+-------------
-- 400000 | 600000 | 60.0
Anti-pattern: normal index on (author_id, created_at)
CREATE INDEX idx_tasks_author_created ON tasks (author_id, created_at DESC);
ANALYZE tasks;
Typical query for the GET /tasks?author=42 endpoint:
EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT id, title FROM tasks
WHERE author_id = 42 AND deleted_at IS NULL
ORDER BY created_at DESC
LIMIT 50;
Expected output (abridged, real numbers from the example):
Limit (cost=0.42..183.56 rows=50 width=20) (actual time=0.045..28.230 rows=50 loops=1)
Buffers: shared hit=2412
-> Index Scan using idx_tasks_author_created on tasks
(cost=0.42..1832.18 rows=400 width=20)
(actual time=0.043..28.215 rows=50 loops=1)
Index Cond: (author_id = 42)
Filter: (deleted_at IS NULL)
Rows Removed by Filter: 1483
Buffers: shared hit=2412
Planning Time: 0.108 ms
Execution Time: 28.290 ms
What's critical:
Filter: (deleted_at IS NULL)— PostgreSQL applied the filter after the Index Scan.Rows Removed by Filter: 1483— it walked 1483 deleted rows to return 50 live ones.Buffers: shared hit=2412— it read a lot of unnecessary pages.Execution Time: 28.290 ms— the cost of the extra work.
Correct pattern: partial index
DROP INDEX idx_tasks_author_created;
CREATE INDEX idx_tasks_author_created_active
ON tasks (author_id, created_at DESC)
WHERE deleted_at IS NULL;
ANALYZE tasks;
Same query:
EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT id, title FROM tasks
WHERE author_id = 42 AND deleted_at IS NULL
ORDER BY created_at DESC
LIMIT 50;
Expected output:
Limit (cost=0.42..68.42 rows=50 width=20) (actual time=0.025..0.412 rows=50 loops=1)
Buffers: shared hit=54
-> Index Scan using idx_tasks_author_created_active on tasks
(cost=0.42..548.32 rows=400 width=20)
(actual time=0.024..0.402 rows=50 loops=1)
Index Cond: (author_id = 42)
Buffers: shared hit=54
Planning Time: 0.115 ms
Execution Time: 0.475 ms
What's critical:
Index Scan using idx_tasks_author_created_active— it used the partial index.- No
Filter:line — thedeleted_at IS NULLfilter is implicit in the index; there's no post-scan work. - No
Rows Removed by Filterline — every row it walked counts. Buffers: shared hit=54— it read 45x fewer pages.Execution Time: 0.475 ms— 60x faster than the anti-pattern.
Comparison table
| Metric | Anti-pattern (normal index) | Pattern (partial index) | Improvement |
|---|---|---|---|
| Execution Time | 28.29 ms | 0.48 ms | 59x |
| Buffers read | 2,412 | 54 | 45x |
| Rows walked | 1,533 | 50 | 31x |
| Index size | ~32 MB (1M entries) | ~13 MB (400k entries) | 2.5x smaller |
The exact figures vary by hardware. The shape of the change (order of magnitude in latency + buffers) is invariant.
Verify the index size
SELECT
indexname,
pg_size_pretty(pg_relation_size(indexname::regclass)) AS size
FROM pg_indexes
WHERE tablename = 'tasks';
Typical output:
indexname | size
-------------------------------------+---------
tasks_pkey | 21 MB
idx_tasks_author_created_active | 13 MB
The partial index is smaller because it only indexes the 400k active rows, not the 1M total. That also means: faster lookups (fewer pages to walk) and less space in shared_buffers taken up by the index (which leaves room for other indexes and hot data).
The delete operation: UPDATE, not DELETE
The change in the app is trivial:
-- Hard delete (what you used to do)
DELETE FROM tasks WHERE id = $1;
-- Soft delete (what you do now)
UPDATE tasks SET deleted_at = NOW() WHERE id = $1 AND deleted_at IS NULL;
Important notes:
1. AND deleted_at IS NULL in the UPDATE's WHERE.
Without that filter, if the row was already deleted, UPDATE would modify its deleted_at, refreshing it to "now." That's semantically wrong: the row was deleted at its original moment, not now. It also generates unnecessary updates on already-deleted rows. The filter is defensive and cheap.
2. Recovery with UPDATE SET deleted_at = NULL:
UPDATE tasks SET deleted_at = NULL WHERE id = $1 AND deleted_at IS NOT NULL;
Same principle: the IS NOT NULL filter is defensive. If the row is already active, there's nothing to do.
3. What happens with MVCC?
PostgreSQL creates a new version of the row with the updated deleted_at. The old version (with deleted_at = NULL) is marked as "not visible to new transactions." Until VACUUM runs, that old version stays on disk. This means UPDATE is more expensive in bloat terms than DELETE (which also leaves a "dead" version until VACUUM, but at least only one). On tables with many updates (not just soft deletes), this accumulates. Module 5 (zero-downtime migrations) and module 7 (bulk operations) discuss strategies for keeping bloat low.
Soft delete with an additional WHERE: the real case
The app's queries are rarely WHERE id = $1. They're usually:
-- "Delete every overdue task in a project"
UPDATE tasks
SET deleted_at = NOW()
WHERE project_id = $1
AND due_date < NOW()
AND deleted_at IS NULL;
-- "Delete the tasks of a user who's leaving"
UPDATE tasks
SET deleted_at = NOW()
WHERE author_id = $1
AND deleted_at IS NULL;
These operations are atomic (one transaction), and the AND deleted_at IS NULL prevents re-deleting already-deleted rows. PostgreSQL returns the number of affected rows, which is useful to confirm the operation deleted what you expected:
result = await session.execute(
update(Task)
.where(Task.author_id == user_id, Task.deleted_at.is_(None))
.values(deleted_at=func.now())
)
print(f"Soft-deleted {result.rowcount} tasks")
Why does this implementation matter in real work?
1. The measured before-and-after is what you sell in a code review. When you propose migrating a table to soft delete with a partial index, showing numbers (28ms → 0.48ms) is what convinces the tech lead. Without numbers, it's opinion. This capsule trains you to produce those numbers.
2. The partial index is a reusable pattern. You'll learn to write it here and you'll see it in multi-tenancy (module 4: partial index per tenant), in optimistic locking (module 6: partial index by active state), and in zero-downtime migrations (module 5: adding a partial index with no lock via CREATE INDEX CONCURRENTLY). Once internalized, you apply it in many contexts.
3. The TIMESTAMPTZ vs TIMESTAMP detail avoids multi-region bugs. Any SaaS app serving customers in multiple time zones is going to run into this. Making the right call on day 1 saves you a painful migration later.
4. Knowing how to read the plan separates you from juniors. "Filter: (deleted_at IS NULL)" + "Rows Removed by Filter" is the signature of the anti-pattern. Seeing it in production and knowing the fix is a partial index makes you valuable.
Traps and common mistakes
Mistake 1 (conceptual): assuming any index "helps" with soft delete
Symptom: "I have an index on author_id, so soft delete is fine."
Why it's wrong: a normal index on author_id covers both active and deleted rows. PostgreSQL can use it to filter by author_id, but afterward it has to discard the deleted ones in the post-scan filter. The latency is proportional to the delete ratio.
How to tell: look at the EXPLAIN ANALYZE. If you see Filter: (deleted_at IS NULL) with Rows Removed by Filter: > 0, the index isn't partial. The higher the Rows Removed, the more wasted cost.
How to fix it: create the partial index with WHERE deleted_at IS NULL. If the original index has a lot of cardinality on its own, you can keep both (the partial one for the endpoint's query, the normal one for queries that do need to see deleted rows, like audits or reports).
Mistake 2 (practical): forgetting ANALYZE after the index change
Symptom: you create the partial index, run the query, and the plan doesn't use it. You still see a Seq Scan or the old index.
Why it happens: the planner uses statistics from pg_statistic. After creating a new index, the statistics don't reflect its selectivity until ANALYZE runs (manually or via autovacuum).
How to tell: run ANALYZE tasks; and re-run the EXPLAIN. If the plan changes, it was missing stats.
How to fix it: run ANALYZE table; after index changes. In migrations, include op.execute("ANALYZE tasks;") after creating the index.
Mistake 3 (conceptual): putting WHERE deleted_at IS NULL in the composite index before the filter columns
Symptom: you create CREATE INDEX ... ON tasks (deleted_at, author_id, created_at) WHERE deleted_at IS NULL expecting it to be more efficient.
Why it's wrong: putting deleted_at as the index's first column contributes nothing when the WHERE deleted_at IS NULL filter is already in the partial index's predicate. The partial index only contains rows with deleted_at IS NULL, so indexing that column is redundant. It also takes up space.
How to tell: review your index definitions with \d tasks in psql. If you see deleted_at as the first column in a partial index with WHERE deleted_at IS NULL, it's redundant.
How to fix it: the correct definition is CREATE INDEX ... ON tasks (author_id, created_at DESC) WHERE deleted_at IS NULL. The predicate column goes in the WHERE, not in the list of indexed columns.
Mistake 4 (practical): bloat accumulated from frequent updates to deleted_at
Symptom: after months with soft delete, the table is bloated (it takes up 3x the space of the live data), and a normal VACUUM doesn't reclaim it.
Why it happens: each UPDATE SET deleted_at = NOW() creates a new version of the row. The old one stays dead until VACUUM runs. If the deleted rows also receive later updates (recoveries and re-deletions, for example), the bloat accumulates. PostgreSQL reuses the space for new inserts, but if the pattern is "many updates, few new inserts," the space never gets fully reused.
How to tell:
SELECT
schemaname || '.' || tablename AS table,
pg_size_pretty(pg_relation_size(schemaname || '.' || tablename)) AS table_size,
n_dead_tup,
n_live_tup,
ROUND(100.0 * n_dead_tup / NULLIF(n_dead_tup + n_live_tup, 0), 1) AS dead_pct
FROM pg_stat_user_tables
WHERE tablename = 'tasks';
If dead_pct > 30%, you have significant bloat.
How to fix it:
- Make sure autovacuum is active and configured for that table (
ALTER TABLE tasks SET (autovacuum_vacuum_scale_factor = 0.05)lowers the threshold for tables with many updates). - In extreme cases,
VACUUM FULLreclaims the space but takes an exclusive lock (downtime). A no-downtime alternative:pg_repack(an extension). - Long term, evaluate whether the table should migrate to an archive table (capsule 07) so the deleted rows don't accumulate bloat in the operational table.
Exercises
Exercise 1: measure the degradation on your machine
Run the capsule's setup (1M rows, 60% deleted). Create the normal index first and measure the typical query. Then drop it, create the partial index, and measure the same query. Report Execution Time, Buffers: shared hit, and Rows Removed by Filter.
See solution
-- Setup
\timing
-- Anti-pattern: normal index
DROP INDEX IF EXISTS idx_tasks_author_created;
DROP INDEX IF EXISTS idx_tasks_author_created_active;
CREATE INDEX idx_tasks_author_created ON tasks (author_id, created_at DESC);
ANALYZE tasks;
EXPLAIN (ANALYZE, BUFFERS)
SELECT id, title FROM tasks
WHERE author_id = 42 AND deleted_at IS NULL
ORDER BY created_at DESC LIMIT 50;
-- Pattern: partial index
DROP INDEX idx_tasks_author_created;
CREATE INDEX idx_tasks_author_created_active
ON tasks (author_id, created_at DESC)
WHERE deleted_at IS NULL;
ANALYZE tasks;
EXPLAIN (ANALYZE, BUFFERS)
SELECT id, title FROM tasks
WHERE author_id = 42 AND deleted_at IS NULL
ORDER BY created_at DESC LIMIT 50;
Example output (MacBook Pro M2, PostgreSQL 16):
| Metric | Normal index | Partial index | Improvement |
|---|---|---|---|
| Execution Time | 28.29 ms | 0.48 ms | 59x |
| Buffers shared hit | 2,412 | 54 | 45x |
| Rows Removed by Filter | 1,483 | 0 | — |
Your numbers will vary by hardware and by the concrete distribution of the random seed, but the shape of the change (order of magnitude) is invariant.
Exercise 2: compare index sizes
Measure the size of the normal index versus the partial one over the same table and columns. How much does each take up? How does that relate to the ratio of active vs deleted rows?
See solution
-- Create both indexes at the same time
CREATE INDEX idx_normal ON tasks (author_id, created_at DESC);
CREATE INDEX idx_parcial ON tasks (author_id, created_at DESC) WHERE deleted_at IS NULL;
ANALYZE tasks;
-- Compare sizes
SELECT
indexname,
pg_size_pretty(pg_relation_size(indexname::regclass)) AS size,
pg_relation_size(indexname::regclass) AS size_bytes
FROM pg_indexes
WHERE indexname IN ('idx_normal', 'idx_parcial');
Example output:
indexname | size | size_bytes
-------------+--------+------------
idx_normal | 32 MB | 33554432
idx_parcial | 13 MB | 13631488
Analysis:
- Normal index: 32 MB to index 1M rows.
- Partial index: 13 MB to index 400k active rows (40% of 1M).
- Ratio: 13/32 ≈ 0.40 — exactly proportional to the ratio of indexed rows.
Operational implications:
- Lower
shared_buffersusage: the partial index fits in memory more easily, leaving more room for other indexes and hot data. - Less maintenance IO: VACUUM and ANALYZE over the partial index are faster.
- Smaller backup size: the index is included in pg_dump (not with
--schema-only, but yes with a binary backup).
The higher the delete ratio, the greater the relative saving. With 90% deleted, the partial index would be ~10% of the normal one's size. With 10% deleted, the saving is marginal (90% of the size).
Exercise 3: apply the pattern to an existing table with no downtime
You have a comments table in production with 5M rows. You want to add deleted_at and the partial index without taking an exclusive lock. Design the migration sequence (you don't need Alembic yet; pure SQL is fine).
See solution
Migration 1 — Add a nullable column:
-- This is fast on PostgreSQL 11+ (no table rewrite because the DEFAULT is NULL).
ALTER TABLE comments ADD COLUMN deleted_at TIMESTAMPTZ NULL;
PostgreSQL 11+ optimizes ADD COLUMN ... NULL into a metadata operation: it doesn't rewrite the table, it just records the new column in the catalog. Short lock (milliseconds).
Migration 2 — Create the partial index with no lock:
CREATE INDEX CONCURRENTLY idx_comments_post_active
ON comments (post_id, created_at DESC)
WHERE deleted_at IS NULL;
CONCURRENTLY avoids an exclusive lock. It takes longer (PostgreSQL makes two passes over the table), but the app keeps serving requests during the creation.
Caveats:
CREATE INDEX CONCURRENTLYcan't run inside a transaction. If you use Alembic, you needop.execute()with autocommit.- If the creation fails halfway (OOM or a conflict, for example), the index is left in an
INVALIDstate and has to be dropped:DROP INDEX idx_comments_post_active;and recreated. - Verify with:
SELECT indexname, indexdef FROM pg_indexes WHERE tablename = 'comments';
Migration 3 — (Optional) backfill of historical soft deletes:
If you need to mark as deleted records that show up as "deleted" in some parallel log from before the column existed, do it in batches:
-- In batches so as not to take a long-lived lock
UPDATE comments
SET deleted_at = '2024-01-01'::timestamptz -- sentinel date
WHERE id IN (SELECT id FROM old_deleted_log LIMIT 10000)
AND deleted_at IS NULL;
-- Repeat until there are no more
Migration 4 — Change the app's code:
Only after the index is created and verified. Change DELETE to UPDATE SET deleted_at = NOW(). Add WHERE deleted_at IS NULL to the existing queries (or, better, automate it with the mechanism from capsule 04).
Summary of the ordering:
- ALTER TABLE ADD COLUMN (fast, short lock).
- CREATE INDEX CONCURRENTLY (slow but no exclusive lock).
- Verify the index and the plan.
- Historical backfill if it applies (in batches).
- Deploy the new code.
Module 5 (zero-downtime migrations) goes deeper into this pattern with Alembic.
Exercise 4: detect accumulated bloat
After simulating many updates to deleted_at, check the table's bloat. How much "dead" space is there? How does it compare to the "live" space?
See solution
Simulate frequent updates:
-- Simulate 50k recoveries and re-deletions
DO $$
BEGIN
FOR i IN 1..10 LOOP
UPDATE tasks SET deleted_at = NULL
WHERE id IN (SELECT id FROM tasks WHERE deleted_at IS NOT NULL LIMIT 5000);
UPDATE tasks SET deleted_at = NOW()
WHERE id IN (SELECT id FROM tasks WHERE deleted_at IS NULL LIMIT 5000);
END LOOP;
END $$;
Measure the bloat:
-- Let autovacuum run (or force one)
VACUUM ANALYZE tasks;
SELECT
schemaname || '.' || relname AS table,
pg_size_pretty(pg_relation_size(relid)) AS table_size,
n_dead_tup,
n_live_tup,
ROUND(100.0 * n_dead_tup / NULLIF(n_dead_tup + n_live_tup, 0), 1) AS dead_pct,
last_vacuum,
last_autovacuum
FROM pg_stat_user_tables
WHERE relname = 'tasks';
Example output:
table | table_size | n_dead_tup | n_live_tup | dead_pct | last_vacuum
-------------------+------------+------------+------------+----------+---------------------
public.tasks | 145 MB | 45123 | 1000000 | 4.3 | 2026-05-02 14:23:11
Analysis:
n_dead_tup: rows marked as dead (old MVCC versions pending VACUUM).dead_pct < 5%: healthy bloat, autovacuum keeps the table clean.dead_pct > 30%: a problem. Consider tuningautovacuum_vacuum_scale_factorfor that table:
ALTER TABLE tasks SET (
autovacuum_vacuum_scale_factor = 0.05,
autovacuum_analyze_scale_factor = 0.02
);
This lowers the threshold so autovacuum runs more often (when dead_pct goes past 5%), keeping the bloat low.
If the bloat has already accumulated:
VACUUM FULLreclaims it but requires an exclusive lock (downtime).pg_repack(an extension) reclaims it with no exclusive lock, but you have to install it.- Migrating to an archive table (capsule 07) eliminates the problem in the operational table for good.
The lesson: soft delete + a well-configured autovacuum is sustainable for mid-sized tables. For large tables with high churn, evaluate an archive table.
Exercise 5: the defensive filter in the UPDATE
Why does the capsule recommend UPDATE tasks SET deleted_at = NOW() WHERE id = $1 AND deleted_at IS NULL instead of WHERE id = $1? Show with a concrete case what happens without the defensive filter.
See solution
Case without the defensive filter:
-- Delete task #100 at 10:00
UPDATE tasks SET deleted_at = NOW() WHERE id = 100;
-- Now: tasks.deleted_at = '10:00'
-- By mistake, the endpoint gets called again at 11:00 (double-clicked button, retry, etc.)
UPDATE tasks SET deleted_at = NOW() WHERE id = 100;
-- Now: tasks.deleted_at = '11:00'
Problems:
- Loss of temporal information: the real deletion date was 10:00; now it says 11:00. If your auditor asks "when was it deleted?", the answer is wrong.
- Unnecessary update: PostgreSQL creates a new version of the row (MVCC) for nothing. Bloat accumulated from idempotent operations that shouldn't have an effect.
- Triggers fire twice: if you have
BEFORE UPDATEorAFTER UPDATEtriggers, they run on every call, even when there's no real change. If the trigger inserts into an audit log, you record two deletions when there was one.
Case with the defensive filter:
-- Delete task #100 at 10:00
UPDATE tasks SET deleted_at = NOW() WHERE id = 100 AND deleted_at IS NULL;
-- Output: UPDATE 1 (1 row affected)
-- Repeated call at 11:00
UPDATE tasks SET deleted_at = NOW() WHERE id = 100 AND deleted_at IS NULL;
-- Output: UPDATE 0 (0 rows affected, there was no change)
Benefits:
- Natural idempotence: calling the endpoint twice has the same effect as calling it once.
- No unnecessary bloat: the second UPDATE doesn't generate a new version.
- Triggers don't fire on repeat calls: the trigger only runs when there's a real change.
- The client can know it did nothing:
result.rowcount == 0indicates the row was already deleted, which lets you handle the case (return a 404, or a 200 with an "already deleted" message).
Implementation in SQLAlchemy:
result = await session.execute(
update(Task)
.where(Task.id == task_id, Task.deleted_at.is_(None))
.values(deleted_at=func.now())
)
if result.rowcount == 0:
raise HTTPException(status_code=404, detail="Task not found or already deleted")
General lesson: any soft delete UPDATE has to be idempotent with a defensive filter. The same applies to recovery (UPDATE SET deleted_at = NULL WHERE deleted_at IS NOT NULL).
Summary and next step
In this capsule you learned:
deleted_at TIMESTAMPTZ NULLis the canonical pattern.TIMESTAMPTZfor timezone correctness,NULLas the representation of "active."- The partial index is part of the pattern, not an optional optimization. Without it, soft delete has a cost that grows with the delete ratio.
- The before-and-after is measurable: ~28ms with a normal index vs ~0.5ms with a partial one over 1M rows with 60% deleted. A 50-60x improvement is typical.
- The delete UPDATE has to be idempotent with the defensive
AND deleted_at IS NULLfilter. It prevents loss of temporal information and unnecessary bloat. - Bloat is a real cost of the pattern. A well-configured autovacuum keeps it low on mid-sized tables; large tables with high churn require archive tables (capsule 07).
- Migrating to soft delete with no downtime combines
ALTER TABLE ADD COLUMN NULL(short lock) +CREATE INDEX CONCURRENTLY(no exclusive lock). Module 5 goes deeper.
Before moving on you should be able to:
- Design the
deleted_atcolumn for a new table with the right type and default without thinking about it. - Create the corresponding partial index in a single line of SQL.
- Read an
EXPLAIN ANALYZEand spot the anti-pattern (Filter + Rows Removed by Filter). - Justify the defensive filter in the soft delete UPDATE with concrete reasons.
Next capsule — Soft delete in SQLAlchemy: mixins and events. You're going to take this pattern to the ORM. You'll see the three mechanisms SQLAlchemy offers to automate the WHERE deleted_at IS NULL filter (the before_compile event listener, a custom query class, a mixin with @declared_attr), you'll compare concrete trade-offs, and you'll implement the recommended pattern: a mixin + an event listener with a clean escape hatch. It's the capsule that keeps your team from writing WHERE deleted_at IS NULL by hand in every new query.
Resources
- PostgreSQL Documentation — Partial Indexes — the official docs. Soft delete is mentioned explicitly as the canonical case.
- PostgreSQL Documentation — Date/Time Types — the difference between
TIMESTAMPandTIMESTAMPTZand why the second one wins. - PostgreSQL Wiki — VACUUM and bloat — the official reference on bloat, VACUUM, and autovacuum tuning.
- Cybertec PostgreSQL — "Index-only scans and VACUUM" — a deep understanding of how VACUUM impacts indexes, relevant to soft delete.
- Brandur Leach — "Postgres Tip: Avoiding Slow
OFFSET" — general PostgreSQL patterns that apply here; a direct tip on partial indexing. - Markus Winand — "Indexing IS NULL" — the chapter of the online book on how PostgreSQL treats
IS NULLin indexes, the technical context for understanding why the partial index works. - Heroku Engineering — "Postgresql at scale: bloat and partial indexes" — Heroku's operational guide.
Module 2 — SQL Patterns for Production APIs Guide
Next capsule: Soft delete in SQLAlchemy — mixins and events.