8. Project: choose and justify Enlace's consistency model and its SLO

Description

The moment has come to bring the seven lessons together into a single deliverable. In this project you will do what an engineer does when it is their turn to close out the reliability and consistency part of a design: make decisions and justify them with numbers. Not "let's make it reliable and that's it", but a defensible document that answers four concrete questions, each backed by a lesson of the module and by arithmetic that gets executed:

1. The reliability plan: what are Enlace's single points of failure and how do I eliminate them? (lessons 2 and 3)
2. The CAP/PACELC classification: what is Enlace under partition and what is it the rest of the time? (lessons 4 and 5)
3. The consistency model: what guarantee do I put on each operation, and why is eventual enough for resolve? (lesson 6)
4. The SLO: what SLI do I measure, what objective do I aim for, how much error budget does it give me, and does my dependency ceiling allow it? (lesson 7)

The deliverable is a reliability design document —a diagram of the topology with its SPOFs marked, a table of consistency decisions per operation, and an SLO sheet with the executed numbers—. You will build a small calculator in Python that produces the SLO sheet from your decisions, run it, and defend each choice. When you finish you will have a template that serves to reason about the reliability of any system, not just Enlace's.

Connection to the module: this is the capstone. It uses the SPOFs and redundancy of lesson 2, the failover of lesson 3, the correct CAP of lesson 4, the PACELC of lesson 5, the measured eventual consistency of lesson 6 and the nines/error budget of lesson 7 —all at once— to produce a real artifact. It is also the bridge to the capstone of the complete guide (module 8, module-08-project-design-enlace-end-to-end): the reliability and consistency decisions you produce here are one of the components that end-to-end design integrates with the estimation, the cache, the replicas and the load balancing of the previous modules.

The project brief

You are the engineer in charge of Enlace's reliability and consistency. The team gives you the anchor numbers —the same ones from the whole guide— and the scaled topology of modules 4 to 6, and asks you for a design document. These are the input data, fixed:

Input datum	Value	Where it comes from
Reads per second (qps_read)	~4,000	100:1 ratio (module 2)
Writes per second (qps_write)	~40	module 2
Read:write ratio	100:1	canonical requirement
Code space	base62⁷ ≈ 3.5×10¹²	module 3
Worst-case replication lag	~2 s	module 5
Dependency availability	compute 99.95%, db 99.9%, CDN 99.99%	providers (brief datum)

And these are the decisions you make (the design levers): which components to make redundant, what failover configuration for each one, the CAP/PACELC classification, the consistency model per operation, and the SLI/SLO/SLA with its numeric objective. The project consists of choosing each one, justifying it, and computing the consequences.

The design steps

Before seeing the reference solution, here is the process. Do it yourself first; the solution is afterward so you can compare.

Step 1 — Map the SPOFs and their elimination

Draw Enlace's topology (load balancer → apps → cache + primary/replicas) and mark each single point of failure. For each one, decide the redundancy (lesson 2) and the failover configuration (lesson 3: active-active for the stateless parts, active-passive for the database). Remember the N+1 rule: one spare unit more than you need.

Step 2 — Classify Enlace in CAP and PACELC

Decide the "if Partition" branch (AP or CP?) and the "Else" branch (EL or EC?) for the resolve path, and justify each one with the cost of getting it wrong. Remember that the classification can be per operation: resolve (read) and the uniqueness of shorten (write) do not have to be the same.

Step 3 — Choose the consistency model per operation

For each Enlace operation —resolve, the creator's read right after shorten, and the uniqueness of the short_code when generating it— choose the model (eventual, read-your-writes, strong-by-construction) and justify it. Lean on the propagation window calculation of lesson 6 (how many codes in flight, what collision probability).

Step 4 — Set the SLI, the SLO and the SLA, and verify the ceiling

Choose the SLI (what you measure), the SLO (your internal goal) and the SLA (what you promise, ≤ SLO). Compute the error budget of the SLO. And —crucially— verify the dependency ceiling: multiply the availability of your series dependencies and check whether it gives you margin for the SLO; if not, propose the architecture change that raises it (lesson 2).

Reference solution

Here is a complete and defensible design document. It is not the only correct answer —another SLO objective or a different redundancy are also defensible—, but it is a solid proposal with each decision justified.

Deliverable 1 — The reliability plan

The topology with its SPOFs marked and its remedy:

                       ┌──────────────────┐
   clients ──────────► │  load balancer   │  SPOF ► redundant: 2 balancers
                       │  (x2, floating VIP)│        with floating virtual IP,
                       └────────┬─────────┘        network-level failover (fast)
                    ┌───────────┼───────────┐
                    ▼           ▼           ▼
                ┌───────┐   ┌───────┐   ┌───────┐   no-SPOF: stateless (module 6),
                │ app 1 │   │ app 2 │   │ app 3 │   active-active, N+1 (one extra).
                └───┬───┘   └───┬───┘   └───┬───┘   Trivial failover: the LB takes the
                    └───────────┼───────────┘        downed one out via health check.
                    ┌───────────┼───────────┐
                    ▼           ▼           ▼
                ┌───────┐   ┌────────┐  ┌────────┐  database: primary + replicas,
                │ cache │   │primary │─►│replica │  active-passive. Failover: promote
                │ (aside)│  │        │  │ (x N+1)│  an up-to-date replica (lessons 3, 5).
                └───────┘   └────────┘  └────────┘  Cache-aside: if the cache goes down,
                                                    the apps go to the db (degrades, not down).

Component	SPOF?	Remedy	Failover configuration
Load balancer	Yes (at the door)	2 balancers + floating virtual IP	Network failover, fast (the most critical)
App servers	No	Already redundant (stateless), N+1	Active-active: the LB takes the downed one out (trivial)
Cache	Partial	Cache-aside: if it goes down, the apps go to the db	Degrades latency, does not take the service down
Db primary	Yes	Replicas ready to promote	Active-passive: promote replica (to design carefully due to split-brain → resilience guide)
Read replicas	No	N+1 replicas	Active-active among themselves: the read LB distributes

Key note inherited from Enlace's asymmetry: during a primary failover, the replicas keep serving reads, so only the writes degrade (shorten, 1% of the traffic). A primary failover is a minor degradation, not a total outage.

Deliverable 2 — The CAP/PACELC classification

Enlace (resolve path) is PA/EL.

• if Partition → PA (available). Under a network partition between the primary and a replica, resolve keeps responding with whatever copy it has, even if it may be a few seconds stale. Cost of the alternative (CP): refusing to redirect a user who clicked —error or wait—, a much worse experience than a slightly stale URL (lesson 4).
• Else → EL (low latency). With a healthy network —99.99% of the time— each resolve is served from the closest replica in ~1 ms, without coordination. Paying coordination (EC) would cost ~14 ms extra per read, across 4,000 reads/s, to guarantee an exactness a redirect almost never needs (lesson 5).

It is the same class as Cassandra and DynamoDB, and not by chance: Enlace has the archetypal profile —read-heavy, data tolerant to lag, priority on responding always and fast— those systems exist for.

Deliverable 3 — The consistency model, per operation

Operation	Model	Justification
resolve (general read)	Eventual	Only ~80 short_codes in flight at any instant; probability 2×10⁻¹¹ that a random read hits one; and a freshly created code nobody else knows yet. Eventual is more than enough (lesson 6).
Creator's read after shorten	Read-your-writes	The only real stale case is the creator resolving their link before it propagates. Their reads are served from the primary for a little while. There is no need for strong consistency for everyone, just this guarantee for the author.
Uniqueness of the short_code when generating it	Strong by construction	Here eventual does NOT suffice: two equal codes for different URLs is irreversible corruption. It is guaranteed by design —disjoint ranges per node (id_generator, module 3)— so that two nodes never collide, without coordinating on the hot path.
clicks counter (if it exists)	Eventual	A count a few seconds behind does no harm to anyone; coordinating each increment at 4,000/s would be an unnecessary bottleneck.

The cross-cutting lesson: consistency is chosen per operation, not for the whole system. Enlace is almost entirely eventual, with read-your-writes for the author and a strong-by-construction guarantee only on uniqueness.

Deliverable 4 — The SLO sheet, executed

First the calculator that produces the numbers:

# enlace_reliability_and_slo.py — the SLO sheet, executed
SECONDS_PER_YEAR  = 365 * 24 * 3600
SECONDS_PER_MONTH = 30  * 24 * 3600

def downtime(a, period):
    return (1 - a) * period

def human(seconds):
    if seconds >= 3600:  return f"{seconds/3600:.2f} h"
    if seconds >= 60:    return f"{seconds/60:.1f} min"
    return f"{seconds:.1f} s"

def parallel(a, n):
    return 1 - (1 - a) ** n

# --- The chosen SLO and its translation ---
slo = 0.999                                   # internal objective for resolve
print("Enlace's SLO on resolve:")
print(f"  objective = {slo*100}%   downtime = {human(downtime(slo, SECONDS_PER_YEAR))}/year")
print(f"  error budget = {human(downtime(slo, SECONDS_PER_MONTH))}/month")

# --- The dependency ceiling, before and after making the db redundant ---
compute, db, cdn = 0.9995, 0.999, 0.9999
ceiling_before = compute * db * cdn
print(f"\nDependency ceiling (in series):")
print(f"  before:  {ceiling_before*100:.3f}%  ->  {'SUFFICES' if ceiling_before>=slo else 'DOES NOT SUFFICE'} for SLO {slo*100}%")

db_redundant = parallel(0.999, 2)             # two dbs at 99.9% in parallel (lesson 2)
ceiling_after = compute * db_redundant * cdn
print(f"  redundant db ({db_redundant*100:.4f}%):")
print(f"  after: {ceiling_after*100:.3f}%  ->  {'SUFFICES' if ceiling_after>=slo else 'DOES NOT SUFFICE'} for SLO {slo*100}% with margin")

What to expect. When you run it:

Enlace's SLO on resolve:
  objective = 99.9%   downtime = 8.76 h/year
  error budget = 43.2 min/month

Dependency ceiling (in series):
  before:  99.840%  ->  DOES NOT SUFFICE for SLO 99.9%
  redundant db (99.9999%):
  after: 99.940%  ->  SUFFICES for SLO 99.9% with margin

And the SLO sheet that summarizes the decision:

Element	Value	Justification
SLI	% of resolve successful in <100 ms, monthly	resolve is 99% of the traffic and what matters to the user (being redirected fast)
SLO	99.9% (43.2 min/month of error budget)	Serious but achievable; absorbs failovers and deploys. Chasing 99.99% would multiply the cost needlessly for a shortener
SLA	99.5% toward paying customers	≤ SLO, to have a cushion: a bad month breaks the SLO (internal alarm) without breaking the SLA (no penalty)
Dependency ceiling	99.84% in series → does not suffice	The db must be made redundant (the weak link at 99.9%): raises the ceiling to 99.94%, now with margin over the SLO

The most important finding of the calculation: the 99.9% SLO is not sustainable with the series dependencies as they are (ceiling 99.84%, below the objective). The action that fixes it is the one from lesson 2: make the database redundant, the weakest link, which raises the ceiling to 99.94% and gives margin. An SLO without verifying the dependency ceiling is a promise you do not control; verifying it turns the project into an honest design.

Common mistakes

Choosing "all strong" or "all eventual" to simplify (of granularity). What happens: someone wants a single label for all of Enlace and applies the same model to the read and to code generation —opening the door to collisions if they choose eventual, or making resolve needlessly expensive if they choose strong—. Why it happens: a single label is more convenient to communicate. How to spot it: if operations with different cost of incoherence (an old redirect is harmless; a duplicated code is corruption) get the same model, you are miscalibrated. How to fix it: deliverable 3 is a table per operation, not a single label.

Promising an SLO without verifying the dependency ceiling (of architecture). What happens: a 99.9% SLO is set and computing the product of the series dependencies is forgotten, guaranteeing that the system will break the objective for reasons external to its own code. Why it happens: the availability of one's own service is looked at and not the complete chain. How to spot it: if your SLO is greater than the product of the availabilities of everything on the critical path, you already lost. How to fix it: do the ceiling calculation before committing to the SLO (as in deliverable 4) and, if it does not suffice, make redundant or take off the path the weakest link.

Designing the primary's failover naively (of reliability). What happens: the plan says "if the primary does not respond, automatically promote a replica", without any mechanism against split-brain —the primary could be alive but isolated, leaving two primaries writing—. Why it happens: automatic promotion looks simple. How to spot it: if your failover does not name fencing or consensus, it has the split-brain hole. How to fix it: at this design level, recognize that the primary's failover needs fencing and leader election by consensus, and mark it as a border with the resilience guide —do not leave it as a promote and done—.

Exercises

Exercise 1 — Defend your SLO. An executive tells you: "99.9% allows 8.76 hours of downtime a year. That seems like a huge amount to me. I want us to promise 99.999% (five nines)". Respond with two technical and cost arguments for why five nines is not appropriate for Enlace, leaning on the table of lesson 7 and on the dependency ceiling.

See solution

1. The cost per nine grows ~10× and the use case does not buy it. Going from 99.9% (8.76 h/year) to 99.999% (5.3 min/year) is three more nines: each one divides the downtime by ten and multiplies the effort (100% automatic failover with no human intervention, multi-region redundancy, 24/7 on-call). For a URL shortener —where a URL that takes a moment to redirect, very occasionally, causes no real harm— that expense is disproportionate. Five nines is the standard of emergency telephony, not of a redirect.
2. The dependency ceiling makes it impossible without a major rearchitecture. Enlace's series dependencies give a ceiling of 99.84% (or 99.94% making the db redundant). Promising 99.999% (which requires a ceiling of at least 99.999%) would require making every dependency redundant across multiple regions and providers, an enormous cost —and even so, 5.3 min/year leaves no margin for any human error—. Promising it without that work would be signing an SLA we will break the first month for reasons outside our control.

Counterproposal: 99.9% as internal SLO with the db redundant (ceiling 99.94%, with margin), and an SLA toward customers below (99.5%) to have a cushion. Serious, achievable and honest.

Exercise 2 — Recompute the ceiling. The infrastructure team improves the compute provider from 99.95% to 99.99%, but the database is still a lone primary at 99.9% and the CDN 99.99%. (a) Compute the new series dependency ceiling. (b) Does it suffice for a 99.9% SLO? (c) Which link still rules and what would you do?

See solution

• (a) Ceiling = 0.9999 × 0.999 × 0.9999 = 0.998801, that is 99.88%.
• (b) It does not suffice. 99.88% is below the 99.9% SLO. Improving the compute barely moved the ceiling.
• (c) The database still rules (the weakest link, 99.9%), exactly like the weak-link rule of lesson 2: in the product, the smallest term dominates, so improving the compute (which was already strong) barely changes the result. What has to be done is make the database redundant: two at 99.9% in parallel give 99.9999%, and the ceiling rises to 0.9999 × 0.999999 × 0.9999 ≈ 0.99980, that is 99.98%, now with margin over the SLO. Always make the weakest link redundant first.

Exercise 3 — Design the reliability of a new feature. Enlace adds "premium" links with expiration: a short_code that stops working on a date expires_at. A paying customer demands that, at the exact instant a link expires, it stops redirecting worldwide. Analyze: (a) what consistency model does that requirement demand? (b) does it clash with Enlace's PA/EL classification? (c) propose a design that meets the requirement without making all of resolve strongly consistent.

See solution

• (a) "At the exact instant, worldwide, it stops redirecting" is a requirement of strong consistency (linearizability): every replica, asked right after the expiration, must agree that the link is no longer valid. It is the opposite of eventual, where different replicas could differ for a few seconds.
• (b) Yes, it clashes. Enlace is PA/EL precisely because resolve tolerates lag; a requirement of instant and global expiration asks for coordination that PA/EL deliberately avoids. Meeting it with strong consistency on every resolve would kill the low latency of the 4,000 reads/s.
• (c) The design that reconciles both: do not make all of resolve strong, but bound the requirement. Options: (1) Accept a tolerance window —negotiate with the customer that the expiration is effective "in seconds", not "at the exact nanosecond"—, which keeps eventual and probably satisfies the real intent. (2) If the exact instant is non-negotiable, treat the expiration as a strongly consistent write (invalidate the short_code on the primary and in the cache in a coordinated way) but keep resolve eventual for everything else: only the premium links with expiration pay the cost of coordination, and only at the moment of expiring, not on every read. It is, once again, choosing consistency per operation and per case: 99.99% of the traffic remains eventual and fast; only this specific feature pays for the guarantee it really needs.

Summary and next step

In this project you brought the six lessons together into a defensible design document for Enlace. You produced the four deliverables: the reliability plan (the SPOFs marked in the diagram —load balancer and primary— and their remedy with redundancy and failover, remembering that a primary failover only degrades the writes thanks to the 100:1 asymmetry); the CAP/PACELC classification (PA/EL, available under partition and fast the rest of the time, justified with the cost of each alternative); the consistency model per operation (eventual for resolve, read-your-writes for the creator, strong-by-construction for uniqueness); and the executed SLO sheet (SLI of resolve, SLO of 99.9% with 43.2 min/month of error budget, SLA below to have a cushion, and the dependency ceiling check that revealed the database must be made redundant to sustain it). The finding that ties everything together: an SLO without verifying the dependency ceiling is a promise you do not control.

With this you close module 7. You now know how to reason, with numbers and without myths, about the reliability of a system (SPOF, redundancy, failover), its consistency tradeoff (CAP, PACELC, strong versus eventual) and its availability promise (SLI/SLO/SLA, nines, error budget). You know how to find the weakest link, choose consistency per operation, and translate a percentage into honest downtime hours.

What comes next is the capstone of the whole guide. In module 8 (module-08-project-design-enlace-end-to-end) you will integrate everything you built over eight modules —how to approach the problem (M1), capacity estimation (M2), the data model and ID generation (M3), the cache (M4), the replicas and sharding (M5), the load balancing and statelessness (M6), and the reliability and consistency you just closed (M7)— into a single end-to-end Enlace design: the diagram, the capacity table, the read and write path, and the list of justified tradeoffs. The reliability and consistency decisions of this project are one of the pieces that final design assembles. And from there, the guide points you toward its ecosystem siblings —resilience, events, architectural styles— to keep going deeper into each border we marked here.

Resources

• Google SRE Book — Chapter 3 (Embracing Risk) and Chapter 4 (SLOs) — how a real team decides how much reliability is enough (neither too much nor too little) and how the error budget guides that decision; the framework that supports your SLO choice in deliverable 4.
• Designing Data-Intensive Applications, Martin Kleppmann — Chapters 5 and 9 (integrative review) — the two chapters that cover replication (reliability, failover, lag) and consistency/consensus (linearizability, CAP); reading them together consolidates the decisions of deliverables 1 to 3 into a single framework.
• System Design Primer — availability, consistency and CAP checklist — a quick review of the patterns you used in the project (redundancy, failover, CAP, eventual consistency), useful as a checklist when defending your design document in an interview.