Module 2: Napkin Estimation (Back-of-the-Envelope)
4. Storage: how much disk in five years
Description
The second capacity number answers a question with very concrete consequences: how much disk does Enlace need, and for how long? Unlike the QPS, which is a rate (requests per second, which comes and goes with the traffic), storage is an accumulation: every URL that's created stays on disk and adds up forever —or until it expires—. That's why storage is estimated over a time horizon (here, five years of retention) and grows month by month to a total.
In this lesson you multiply three things —how many records per month, for how many months, and how much each record weighs— to reach Enlace's number: ~6 TB at five years. Then you refine it with the two factors people forget and that can double or triple the count: the indexes (the structures that make search fast take up space) and the replication (keeping several copies so as not to lose data). And you learn to read what that number means: 6 TB "fits but is planned", a very different answer from "fits with plenty to spare" or "doesn't fit, you have to spread it out".
Connection to the module: this lesson builds the second row of the capacity table. It uses lesson 2's kit (powers of 10, base-10 units) and a new piece of data from the prompt: the record size (~1 KB) and the retention (5 years). The number it produces —6 TB, or 27 TB with indexes and replicas— is the one that in module 5 will justify the decision of whether Enlace needs sharding (spreading the database across several machines) or not. The internal structure of the Link record (what fields it has, why it weighs 1 KB) belongs to module 3; here we take "1 KB per record" as a given input and don't discuss its anatomy in depth.
The filing cabinet that never empties
Imagine a law firm that files every case it opens. The capacity question isn't "how many cases do they open per second?" —that would be the QPS— but "how many filing cabinets do I need to buy, and by when?". And that question has a different nature: the cases don't go away. If they open 100 cases a month and each takes up a centimeter of shelf, by the end of the first year they've filled 12 meters of shelf; by the end of the second, 24; and so on, adding up, because nothing is removed from the archive. The cabinet fills up cumulatively, and to know how many to buy you have to project the total over several years, not measure the rate of a single instant.
There's one more detail the lawyer knows and the novice forgets: the case isn't just the case's paper. It's the paper plus the folder's index (the tab, the label, the entry in the registry book that lets you find the case without going through all of them). That index also takes up space, sometimes as much as the case itself. And if the firm is serious, it keeps a copy of each case in another office in case the first one burns down —which doubles or triples the total space—. So the real archive is bigger than the sum of the papers: it's papers × index factor × copies factor.
A system's storage is exactly this filing cabinet. The records accumulate month by month over a retention horizon; the indexes that make them searchable take up their own space; and the replicas that protect them multiply the total. Estimating well is not forgetting any of the three.
The three factors of storage
Every storage calculation, in any system, is the same product of three things:
storage = (records per unit of time) × (retention time) × (size per record)
For Enlace, each factor comes from the prompt:
- Records per unit of time: 100 million new URLs per month. Each
shortencreates aLinkrecord. - Retention time: 5 years = 60 months. It's how long we keep each record before (maybe) expiring it.
- Size per record: ~1 KB. It's how much a complete
Linkrecord weighs on disk.
Multiply the three and you have the raw storage. Then you adjust for indexes and replicas. Let's go part by part.
Where does "1 KB per record" come from?
Before multiplying, it's worth knowing what's inside that 1 KB, even though the complete anatomy of the Link record belongs to module 3. A record stores, roughly:
| Field | Content | Approximate size |
|---|---|---|
short_code | the short code (7 chars base62) | ~7 bytes |
long_url | the original URL | ~500 bytes (the prompt's average) |
created_at | when it was created | ~8 bytes |
expires_at | when it expires | ~8 bytes |
clicks | visit counter | ~8 bytes |
| subtotal | ~531 bytes |
The fields add up to about ~531 bytes, dominated by the long_url. Why then do we say "~1 KB" and not "~531 bytes"? Because a real database doesn't store just the data: it adds overhead per row (headers, pointers, alignment), and above all it maintains indexes that take up space separately. Rounding to 1 KB per record is a comfortable and slightly conservative napkin assumption that absorbs that overhead without having to break it down. It's a good example of smart rounding: 531 bytes would fake a precision we don't have about the real size of the URLs, while ~1 KB is a round number that leaves margin for what we're not counting in detail. The fine structure of the record and why the short_code is exactly 7 characters is module 3.
It's worth noting that the long_url —the field that dominates the size— is text, and text compresses well: URLs share a lot of prefix (https://www., repeated domains, similar paths), and an ordinary compression algorithm can often reduce them to half or less. That means the "~500 bytes per URL" is a generous upper bound; in practice a database that compresses could store the data in quite a bit less. We don't put compression in the main calculation —for napkin math the conservative "uncompressed" assumption is best—, but it's a real lever that plays in our favor: if the 6 TB ever felt tight, compressing is one of the first cards before resorting to sharding. Noting it mentally is part of keeping the number "alive" and not just computed.
Worked example: Enlace's 6 TB
With the three factors, the raw calculation:
records in 5 years = 100M/month × 12 months/year × 5 years = 6,000M records
raw storage = 6,000M × 1 KB = 6 × 10⁹ × 10³ B = 6 × 10¹² B = 6 TB
Let's run it, step by step:
# Enlace's raw storage at 5 years.
records_per_month = 100_000_000
months = 12 * 5 # 5 years = 60 months
bytes_per_record = 1_000 # ~1 KB, base 10
total_records = records_per_month * months
total_bytes = total_records * bytes_per_record
print(f"total_records = {total_records:,} = {total_records/1e9:.0f} billion")
print(f"total_bytes = {total_bytes:,}")
print(f"total_TB = {total_bytes / 1e12:.1f} TB")What to expect.
total_records = 6,000,000,000 = 6 billion
total_bytes = 6,000,000,000,000
total_TB = 6.0 TB
~6 TB of raw data at five years, over 6 billion records. Notice the exponent arithmetic behind it: 10⁸/month × 60 months ≈ 6 × 10⁹ records, and 6 × 10⁹ × 10³ B = 6 × 10¹² B = 6 TB. You counted zeros: 8 + (the 6×10¹ of the months) → 9, and then 9 + 3 (the KB) → 12, which is TB. Without multiplying anything long.
And here, again, the number brings a design conclusion. 6 TB is a manageable but not negligible amount. To calibrate: a common SSD today is 1 to 4 TB, so 6 TB fits on a couple of disks, or on a single database server with several disks. That is, Enlace is not obligated to shard for space —it doesn't need to spread its data across many machines just because it won't fit—. That's important news, because sharding is one of the most complexity-expensive decisions in the whole design (module 5), and discovering with a calculation that it's not needed for space saves you that complexity. But 6 TB isn't "forget about it" either: it's a number that's planned and watched, above all because the missing factors are going to make it grow.
Storage grows: the movie, not the photo
The 6 TB is the photo of the end: what Enlace takes up after five years. But storage is a movie, not a photo —it grows month by month from zero—, and seeing the growth curve matters for planning when to buy disk. Since Enlace ingests at a constant rate (100M/month), the growth is linear:
records_per_month = 100_000_000
bytes_per_record = 1_000
for year in range(1, 6):
recs = records_per_month * 12 * year
tb = recs * bytes_per_record / 1e12
print(f"end of year {year}: {recs/1e9:>4.1f} billion records = {tb:.1f} TB")What to expect.
end of year 1: 1.2 billion records = 1.2 TB
end of year 2: 2.4 billion records = 2.4 TB
end of year 3: 3.6 billion records = 3.6 TB
end of year 4: 4.8 billion records = 4.8 TB
end of year 5: 6.0 billion records = 6.0 TB
Cumulative storage (raw), in TB
6 TB ┤ ●
┤ ●
┤ ●
3 TB ┤ ●
┤ ●
┤ ●
1 TB ┤ ●
┤ ●
0 └──┬────┬────┬────┬────┬────┬────┬────┬────┬────┬──
0 6 12 18 24 30 36 42 48 54 months
The line rises 1.2 TB per year, adding evenly. This has two useful readings. The first: Enlace doesn't need 6 TB on day one; it starts with almost nothing and reaches 1.2 TB at the end of the first year. You can buy disk as it grows, not all at once. The second, more important: if the service grows (more than 100M/month over time), the line curves upward and reaches 6 TB before five years. Our estimate assumes constant ingestion; a good estimator notes that assumption ("6 TB if the rate stays at 100M/month") because business growth is exactly what can invalidate the projection. The 6 TB photo is correct under the assumption; the movie reminds you which assumption it depends on.
The two factors people forget: indexes and replicas
The 6 TB is the raw data. The disk you'll actually buy is more, for two reasons almost all novices overlook.
Indexes. A database is useless if you can't find a record fast. To resolve a short_code into a long_url without going through the 6 billion records one by one, the database maintains an index on short_code —a structure (typically a B-tree) that points from each code to its row—. That index takes up its own space, and it's not little: for a table like this, indexes can add on the order of +50% to the size. It's a napkin factor; the exact number depends on how many indexes you have and on which columns, but ignoring it entirely underestimates the disk systematically.
Replicas. Keeping a single copy of the data is playing to lose it: if that disk (or that machine) dies, everything's gone. That's why serious systems keep several copies on different machines —replication, the topic of module 5—. A typical factor is ×3: three complete copies, so you can lose two and still have the data. Replication multiplies all the storage (data and indexes) by the number of copies.
Let's put the three floors of the calculation together:
raw = 6e12 # 6 TB of raw data
with_indexes = raw * 1.5 # +50% for indexes
with_replication = with_indexes * 3 # x3 copies
print(f"raw = {raw/1e12:.0f} TB")
print(f"+ indexes (x1.5) = {with_indexes/1e12:.0f} TB")
print(f"+ replication (x3) = {with_replication/1e12:.0f} TB")What to expect.
raw = 6 TB
+ indexes (x1.5) = 9 TB
+ replication (x3) = 27 TB
Enlace's honest provisioning number isn't 6 TB: it's on the order of ~27 TB when you count indexes and three replicas. It's still perfectly manageable —27 TB fit on a handful of servers—, but it's 4.5 times the naive number, and that difference is what separates a capacity plan that works from one that falls short halfway through the year. The rule: the raw number is the starting point, not the answer. Report it, but adjust it for indexes and replicas before deciding how much disk to buy.
How to report it well: "~6 TB of raw data at 5 years; ~9 TB with indexes; ~27 TB provisioned with ×3 replication". The three figures tell the complete story —the data, the searchable data, and the searchable and safe data— and make clear the assumptions (index factor, replica factor) that anyone can adjust.
A clarification, because they're confused: a replica isn't the same as a backup. The three replicas of the ×3 factor are live and synchronized copies that serve traffic and protect against the death of a machine —if a server goes down, another replica responds instantly—. A backup is a copy frozen in time that protects against something else: an accidental deletion or a corruption, cases where the replicas don't help (because they'd faithfully replicate the error). A serious system has both, and each one adds its own space. For the napkin estimate we put the replica factor (×3) in the main calculation and note separately that backups add still more disk —typically a few more complete copies, depending on the backup retention policy—. We don't break it down here, but knowing that a replica and a backup are different things avoids underestimating the total space and is the kind of nuance that distinguishes a mature estimate.
Expiration changes the movie: when the disk stabilizes
Our 6 TB calculation assumes that nothing is deleted in five years: everything that comes in stays. But Enlace's prompt mentions expiration as an option, and if records expire, the movie is different and ends better. The key is that, with expiration, storage doesn't grow without a ceiling: it stabilizes when what comes in equals what goes out.
Suppose Enlace expires URLs at 2 years. Then, at any given moment, only the records of the last 2 years (24 months) live on disk, not those of all 5. Each month 100M new ones come in and the 100M that turned 2 years old go out (expire), so the total stays put at the accumulation of the window:
records_per_month = 100_000_000
bytes_per_record = 1_000
for retention_months in (24, 36, 60):
steady = records_per_month * retention_months * bytes_per_record
print(f"retention {retention_months} months ({retention_months//12} years): {steady/1e12:.1f} TB at steady state")What to expect.
retention 24 months (2 years): 2.4 TB at steady state
retention 36 months (3 years): 3.6 TB at steady state
retention 60 months (5 years): 6.0 TB at steady state
With expiration at 2 years, Enlace stabilizes at ~2.4 TB forever, instead of growing indefinitely. The curve rises linearly during the first 2 years (up to 2.4 TB) and then flattens: each new URL replaces one that expires. This completely changes the capacity plan —2.4 TB stable is much more comfortable than "grows 1.2 TB/year without end"— and it's why the retention policy is a design decision with direct impact on the disk. We use "5 years, nothing expires" as a conservative assumption (the worst case for space) so the number doesn't depend on a policy that may not exist; but a good estimator always asks "does the data expire?", because the answer can divide the disk by two or more.
Which factor rules: sensitivity to the assumption
An estimate has several assumptions, and not all weigh equally. It's worth knowing which of the three factors moves the result most, because that's where it's worth fine-tuning the assumption and not the others. In Enlace's storage, the three factors enter as multipliers, so a percentage change in any one changes the total by the same percentage —but the one with the most real uncertainty is the record size, because the "~1 KB" was the coarsest rounding—. Let's see how the total moves if the record weighed differently:
records = 6e9 # 6 billion (fixed)
for kb in (0.5, 1.0, 2.0):
tb = records * kb * 1e3 / 1e12
print(f"record of {kb:>3} KB -> {tb:>4.1f} TB raw")What to expect.
record of 0.5 KB -> 3.0 TB raw
record of 1.0 KB -> 6.0 TB raw
record of 2.0 KB -> 12.0 TB raw
The total is linearly sensitive to the record size: doubling the weight doubles the disk. This says two things. First, that the "~1 KB" is the assumption most worth revisiting if you want to fine-tune the count —if you measured that the real URLs average 800 bytes and not 500, the number would drop noticeably—. Second, and more reassuring: even in the worst reasonable case (2 KB per record), Enlace is 12 TB raw, which is still manageable. The estimate is robust: even if you're wrong by a factor of 2 in the most uncertain assumption, the design conclusion ("fits on a few machines, doesn't need sharding for space") doesn't change. A good estimator does this mental test —"and if my weakest assumption were double?"— and checks that the answer holds. If it holds, the number is solid; if the conclusion flips with a factor of 2, you have to measure that assumption better before deciding.
A bit of intuition: the storage of other systems
As with the QPS, Enlace's 6 TB make sense when compared. What dominates storage is the record size, and there systems differ by orders of magnitude according to what they store:
| System | Typical size per record | Why |
|---|---|---|
| Enlace (URLs) | ~1 KB | short text: a URL and some metadata |
| A messaging system (text) | ~1 KB per message | short text, like Enlace |
| A photo service | ~1–5 MB per photo | compressed image: a thousand times a URL |
| A video service | ~1 GB per hour of video | millions of times a URL |
For the same number of records, storing photos weighs ~1,000 times more than storing URLs, and storing video ~1,000,000 times more. That's why a URL shortener with 6 billion records takes up 6 TB (fits on a few machines), while a photo service with the same number of records would take up petabytes (thousands of TB, which really do force massive distributed storage). This comparison teaches you to read the type of system: when the record is text, storage is rarely the problem; when it's multimedia, it's usually the central problem of the design. Enlace, being text, has the luxury that the disk is a minor factor —and that frees the design to concentrate on the read, which really is its challenge—.
Where does this data live? A note on the storage tiers
Enlace's 6 TB don't all have to live on the same type of medium, and this connects with the module's other numbers. There's a storage hierarchy, from fastest and most expensive to slowest and cheapest:
- RAM (memory): very fast (nanoseconds), very expensive per byte, volatile. Only a fraction of the data fits. Here lives the cache —lesson 6's working set, ~333 MB—, not the 6 TB.
- SSD (solid-state disk): fast (microseconds), medium price. Here lives Enlace's database: the 6 TB (or 9 with indexes) fit comfortably.
- Hard disk / object storage (cheap): slow (milliseconds), very cheap per byte. For cold data, files, backups.
The point for estimation: it's not enough to ask "how many bytes?", it also matters "of what type?". Enlace's 6 TB on SSD are perfectly affordable; the same 6 TB all in RAM would be absurdly expensive and unnecessary —which is why only the hot part is cached—. The storage estimate (this lesson) gives you the total that goes to SSD; the memory estimate (lesson 6) tells you what fraction of that is worth moving up to RAM. The two numbers work together: one sizes the disk, the other the cache, and the difference between them (6 TB on disk against ~333 MB in RAM) is exactly what makes caching cheap and worth it.
Common mistakes
Forgetting indexes and replicas and reporting only the raw data (systematic underestimation). What happens: someone computes "6 TB" and buys 6 TB of disk, and a few months later runs out of space because the indexes and replicas took up 4.5 times more. Why it happens: the raw calculation (records × size) is the one that comes out directly, and the index and replica factors have to be remembered to add. How to detect it: if your storage number is exactly "records × size" with no multiplier, you forgot two factors. How to fix it: always multiply by the index factor (~1.5) and by the replica factor (~3, or whichever you use) before provisioning. The raw is for understanding the data volume; the provisioned is for buying disk.
Estimating storage as a rate instead of an accumulation (confusing the type of number). What happens: the person treats storage like the QPS and reports "Enlace stores 100M records per month" without projecting the total over the retention. Why it happens: it comes from applying the same mold as the QPS (which really is a rate) to storage (which is an accumulation). How to detect it: if your storage number has units of "per month" or "per second", you estimated it as a rate, not an accumulation. How to fix it: storage is estimated over a horizon (the retention) and gives a total in bytes, not a rate. Multiply by the retention months; that factor is what turns the ingestion rate into the disk accumulation.
Ignoring expiration when the prompt includes it (undeclared assumption). What happens: someone computes 6 TB assuming that nothing is deleted, when the prompt says URLs can expire. Why it happens: the raw "everything accumulates forever" calculation is the simplest and is taken by default. How to detect it: if your estimate grows without a ceiling and the system has expiration or limited retention, you overestimated. How to fix it: if records expire (say, at 2 years), storage stabilizes at the accumulation of the retention window, not the historical total. For Enlace we use 5 years of full retention as a conservative assumption (the worst case for space), but a good estimator states: "6 TB assuming nothing expires in 5 years; with expiration at 2 years, the disk would stabilize at ~2.4 TB". The retention assumption changes the number, so it's stated.
Exercises
Exercise 1 — The storage of a new system. A notes service stores 50 million new notes per month, each ~2 KB, with 3-year retention. (a) How many records in total? (b) How much raw storage? (c) With indexes (+50%) and ×3 replication, how much to provision? Show the arithmetic.
See solution
- (a)
50M/month × 12 × 3 = 50M × 36 = 1,800M = 1.8 billion records(1.8 × 10⁹). - (b)
1.8 × 10⁹ × 2 KB = 1.8 × 10⁹ × 2 × 10³ B = 3.6 × 10¹² B = 3.6 TBraw. - (c) With indexes:
3.6 × 1.5 = 5.4 TB. With ×3 replication:5.4 × 3 = 16.2 TBprovisioned.
Complete report: "~3.6 TB raw at 3 years; ~5.4 TB with indexes; ~16.2 TB with ×3 replication". Note that the heavier record (2 KB vs Enlace's 1 KB) and the shorter retention (3 vs 5 years) partly offset each other; the method is identical, only the three factors change.
Exercise 2 — The growth movie. With Enlace's numbers (100M/month, 1 KB, constant ingestion), in which year does it cross the 4 TB threshold of raw data? And what would happen to that crossing if from year 3 the service doubled its ingestion to 200M/month?
See solution
With constant ingestion, the accumulation rises 1.2 TB/year (100M/month × 12 × 1 KB = 1.2 TB/year). The 4 TB is crossed between year 3 (3.6 TB) and year 4 (4.8 TB), roughly at 4 / 1.2 ≈ 3.3 years.
If in year 3 the ingestion doubles to 200M/month, the rate goes from 1.2 to 2.4 TB/year from that point. At the end of year 3 you have 3.6 TB (accumulated at the old rate); from there you add 2.4 TB/year, so the 4 TB is crossed shortly after year 3, almost immediately. The lesson: business growth accelerates the disk filling up, and a projection made with constant ingestion is optimistic if the service takes off. That's why the ingestion assumption is stated and revised: the movie depends on it.
Exercise 3 — Sharding yes or no? Based only on the storage, argue whether Enlace needs to spread its database across several machines (sharding) for reasons of space. Consider the provisioned number (~27 TB) and the fact that a modern database server can have on the order of 10–20 TB of disk. Does the read QPS from lesson 3 change your answer?
See solution
For space, Enlace is at the limit but not obligated to shard. The raw data (6 TB) or even with indexes (9 TB) fits comfortably on a single 10–20 TB server. The provisioned number with replicas (27 TB) doesn't fit on one machine, but that's not sharding: those are copies of the same data on different machines (replication), not the data spread out. Each replica is still ~9 TB and fits on one server. So for pure space, Enlace can live on a ~9 TB primary database with two ~9 TB replicas each, without sharding.
But the QPS can push toward sharding, and that's the subtlety. Sharding isn't done only for space; it's also done for load: if a single primary can't handle the writes, or if the reads exceed what the replicas can serve, you have to spread. For Enlace the writes are trivial (40–120/s), so no need by write; and the reads (4,000–12,000/s) are better resolved with a cache (module 4) than with sharding. Conclusion: Enlace probably doesn't need sharding, neither for space (it fits) nor for load (cache and replicas are enough). This is exactly the kind of decision that estimation lets you make with grounds instead of out of fear, and module 5 develops it. Being able to argue it with numbers —and not just intuit it— is the module's goal.
Summary and next step
In this lesson you produced the second row of the capacity table: the storage. You learned that, unlike the QPS (a rate), storage is an accumulation over a retention horizon, and that it comes from multiplying three factors: records per month × retention months × size per record. For Enlace: 100M × 60 × 1 KB = 6 billion records = ~6 TB raw. You saw the growth movie (1.2 TB/year, linear under constant ingestion) and why the ingestion assumption matters. And you added the two factors people forget: indexes (~+50%, so the data is searchable) and replication (×3, so it's safe), which bring the provisioned number to ~27 TB. The design conclusion: 6 TB "fits but is planned"; Enlace doesn't need sharding for space.
Before moving on you should be able to: estimate storage as the product of the three factors; project the growth curve and name the ingestion assumption; adjust the raw by indexes and replicas before provisioning; and argue with numbers whether a system needs sharding for space.
The third number is the bandwidth: how many bytes per second come in and out of Enlace over the network. It's a calculation that reuses the QPS from lesson 3 (bandwidth = QPS × payload size) and that, for Enlace, has a happy ending —a number so small it confirms the network isn't its bottleneck—. Lesson 5 computes it and, in passing, teaches you to read what type of system you have on your hands by comparing the bandwidth of a URL shortener with that of a video service.
Resources
- Martin Kleppmann, Designing Data-Intensive Applications, Chapter 3, "Storage and Retrieval" — dataintensive.net. Explains why indexes take up space and speed up reads (B-trees, LSM-trees), exactly the index factor we add here. To understand where the "+50%" comes from. In English.
- PostgreSQL documentation, "Database Physical Storage" — postgresql.org/docs/current/storage.html. The real detail of per-row and per-page overhead in a concrete database, which is what we round with the "~1 KB per record". Useful if you want to see where the overhead the rounding absorbs comes from.
- The System Design Primer, "Back-of-the-envelope" section (URL shortener storage calculation) — github.com/donnemartin/system-design-primer. It walks through a storage estimate almost identical to Enlace's, with the same records × retention logic. Free, in English.