Module 2: Latency and Cost as Architecture

6. Async and streaming: when the user can't wait

Next →

Overview

By the end of this lesson you'll know how to attack the problem neither the cascade nor the cache solves: the perceived latency on the operations that do reach the LLM and take seconds. There are two techniques, and both start from a key distinction: the real time an operation takes isn't the same as the time the user perceives waiting. Streaming leverages that by showing the response token by token —the user sees text almost immediately instead of staring at a blank screen until the complete block is ready—, so the perceived latency plummets even though the real work takes the same. Async goes further: when an operation takes too long to make anyone wait, it pulls it off the critical path entirely —it queues the work, responds "in progress" instantly, and delivers the result when it's ready—. You're going to measure both: how streaming lowers the perceived latency from 1500 ms to 303 ms, and how async lowers it from 2100 ms to 15 ms by moving the work elsewhere.

This matters because the most common latency mistake isn't that the LLM is slow —that's a fact from lesson 2— but putting it synchronous in the critical path when it takes seconds. Synchronous means the user is left waiting, blocked, until the LLM finishes. With a millisecond operation, that's fine; with a two-second one, the user stares at a frozen screen and wonders whether something broke. Streaming and async are the two architecture responses to that wait: one makes it tolerable by showing progress, the other eliminates it by taking the work out of sight. Knowing which to use —and when synchronous simply isn't acceptable— is what separates an AI feature that feels nimble from one that feels broken, even when the model takes exactly the same in both.

Connection with the module: this lesson completes the set of techniques. Lesson 3 set the two budgets; lessons 4 (cascade) and 5 (cache) attacked the cost one; this one attacks the latency one —and specifically the tail latency lesson 4 left unresolved (the cascade lowers the median latency, but the hard queries still take the same in the strong model)—. Streaming is the answer to that tail: it doesn't speed up the generation, but it makes the user perceive a fast response. Async is the answer to the operations that shouldn't be in the critical path at all. Lesson 7 is going to compose cost (cascade + cache) and latency (streaming) into a single request path. And the idea of "pulling the work off the critical path" connects with the events and queues pattern of the ecosystem's architecture guides —here we apply it to the AI component—.

The kitchen that lets you know when the dish is ready

Think of it this way. You go into a restaurant and order a dish that takes a while. There are three ways the kitchen can handle your wait, and each feels radically different even though the dish takes the same to cook.

The first, the worst: the waiter takes your order, goes to the kitchen, and doesn't come back until the whole dish is ready. You sit at the table staring at the wall, no bread, no water, no sign anything is happening, fifteen minutes. The dish took fifteen minutes —a fact—, but your experience was fifteen minutes of uncertainty. That's the synchronous blocking: the user waits for the complete result with no signal, staring at a blank screen.

The second, much better: the waiter brings you bread immediately, then the appetizer, then the main course in parts as they come out. The main course still takes fifteen minutes total, but you're eating from the first minute —the wait was filled with visible progress, and you didn't even notice the fifteen minutes—. That's streaming: the response arrives in parts, the user sees something almost immediately, and even though the total work takes the same, the perceived wait plummets.

The third, for dishes that really take a while: the kitchen gives you a pager —"when it's ready, it vibrates"— and you go to your table, chat, check your phone, live your life. You're not waiting at the counter; the dish cooks out of your path, and they let you know when it's ready. That's async: the operation is pulled off the critical path entirely, the user gets an instant acknowledgment ("in progress") and goes about their business until the result arrives.

The dish takes the same in all three cases —the kitchen doesn't cook faster—. What changes is how the wait is handled, and that decides everything. Streaming and async are an AI feature's bread-in-parts and pager: they don't make the LLM faster, but they make the wait tolerable or invisible. This lesson is learning which of the three each operation uses —and why the first, the waiter who disappears, is a design mistake when the dish takes seconds—.

Worked example: streaming and async, measured

We're going to measure the two techniques over two real Mercado operations. The latency budget is 800 ms (search's). There's no real LLM: the total latency is computed with the stub —remember from lesson 2 that latency grows with the output tokens—.

Part A, streaming: a 400-token support agent response with the strong model. Blocking, the user sees nothing until the last token. With streaming, they see the first token almost immediately —the time-to-first-token (TTFT)— and read while the rest is generated.

Part B, async: the "describe your product" generator for sellers, 600 tokens with the strong model. Synchronous, the seller waits for everything. Async, the job is queued, "in progress" is responded instantly, and the text arrives later.

# Lesson 06 — async and streaming where the user can't wait for the complete block
# LLM STUB: everything SIMULATED. Zero network, zero API, zero keys.

MODELS = {
    "cheap":  dict(usd_in=0.0008, usd_out=0.004, base_ms=90,  ms_per_tok=0.4),
    "strong": dict(usd_in=0.008,  usd_out=0.040, base_ms=300, ms_per_tok=3.0),
}
def total_latency(model, out_tokens):
    m = MODELS[model]
    return m["base_ms"] + out_tokens * m["ms_per_tok"]

LATENCY_BUDGET_MS = 800

# --- Part A: streaming — time-to-first-token (TTFT) vs total time ---
# The support agent's response: 400 output tokens with the strong model.
# Blocking: the user sees NOTHING until the last token.
# Streaming: the user sees the first token almost immediately and reads as it's generated.
m = MODELS["strong"]
OUT = 400
blocking_ms = total_latency("strong", OUT)          # waits for the complete block
ttft_ms     = m["base_ms"] + 1 * m["ms_per_tok"]     # first token: base + 1 token

print("== Part A: streaming (support agent response, 400 tokens) ==")
print(f"{'mode':<12}{'perceived_ms':>14}{'verdict_vs_budget':>20}")
print(f"{'blocking':<12}{blocking_ms:>14.1f}{('OVER' if blocking_ms>LATENCY_BUDGET_MS else 'OK'):>20}")
print(f"{'streaming':<12}{ttft_ms:>14.1f}{('OVER' if ttft_ms>LATENCY_BUDGET_MS else 'OK'):>20}")
print(f"  perception: the user sees text at {ttft_ms:.0f} ms instead of waiting {blocking_ms:.0f} ms "
      f"(-{(1-ttft_ms/blocking_ms)*100:.0f}%)")

# --- Part B: async — pull the heavy work off the request path ---
# The "describe your product" generator for sellers: 600 tokens with the strong model.
GEN_OUT = 600
sync_ms  = total_latency("strong", GEN_OUT)          # the seller waits for EVERYTHING
ENQUEUE_MS = 15                                       # enqueue the job and respond "in progress"
async_ms = ENQUEUE_MS                                 # perceived by the seller

print("\n== Part B: async ('describe your product', 600 tokens) ==")
print(f"{'mode':<12}{'perceived_ms':>14}{'real_work_ms':>14}{'verdict_vs_budget':>20}")
print(f"{'sync':<12}{sync_ms:>14.1f}{sync_ms:>14.1f}{('OVER' if sync_ms>LATENCY_BUDGET_MS else 'OK'):>20}")
print(f"{'async':<12}{async_ms:>14.1f}{sync_ms:>14.1f}{('OVER' if async_ms>LATENCY_BUDGET_MS else 'OK'):>20}")
print(f"  the real work ({sync_ms:.0f} ms) doesn't disappear: it moves off the critical path;")
print(f"  the seller gets 'in progress' in {async_ms} ms and the text arrives when it's ready.")

What to expect. When you run it:

== Part A: streaming (support agent response, 400 tokens) ==
mode          perceived_ms   verdict_vs_budget
blocking            1500.0                OVER
streaming            303.0                  OK
  perception: the user sees text at 303 ms instead of waiting 1500 ms (-80%)

== Part B: async ('describe your product', 600 tokens) ==
mode          perceived_ms  real_work_ms   verdict_vs_budget
sync                2100.0        2100.0                OVER
async                 15.0        2100.0                  OK
  the real work (2100 ms) doesn't disappear: it moves off the critical path;
  the seller gets 'in progress' in 15 ms and the text arrives when it's ready.

Read the two parts, because each shows a different idea of what can be done with the wait.

Streaming lowers the perceived latency without speeding up anything. The support agent's response takes 1500 ms to generate completely (300 base + 400 tokens × 3 ms) —that violates the 800 ms budget, and the blocking user stares at a frozen screen for a second and a half—. With streaming, the first token appears at 303 ms (300 base + 1 token), and from there the user reads while the rest is generated. The perceived latency —what the user waits before seeing something— drops 80%, from 1500 to 303 ms, and now it fits within the budget. And here's the crucial thing: the real work didn't change. The model still takes 1500 ms to produce the 400 tokens; streaming doesn't speed it up by a microsecond. The only thing that changed is when the user sees the first result —and that's what decides whether the feature feels fast or broken—. Streaming is the bread that arrives immediately while the dish cooks.

Async takes the work out of sight entirely. The "describe your product" generator takes 2100 ms (300 base + 600 tokens × 3 ms). Synchronous, the seller waits those 2100 ms blocked —it violates the budget by more than double—. Async, the system queues the work and responds "in progress" to the seller in 15 ms (what it takes to enqueue), and the text arrives when it's ready. The seller's perceived latency drops from 2100 to 15 ms —it fits the budget with room to spare—. And again, the crucial thing: the 2100 ms of real work doesn't disappear; it moves. The model still generates the 600 tokens in 2100 ms, but it does it off the critical path, in the background, while the seller goes about their business. Async is the pager: the operation cooks separately and they let you know.

When to use each. The difference between the two parts isn't the model or the tokens: it's whether the user needs the result to continue. In the support agent, the user wants to read the response now —they can't continue without it—, so streaming is used: the result arrives, but in parts, so the wait is tolerable. In "describe your product," the seller doesn't need the description at that instant —they requested it, they can keep editing other things and check it in a minute—, so async is used: it's pulled off the path entirely. The rule: if the user waits for the result to continue, use streaming (show it in parts); if the user doesn't have to wait for it, use async (take it off the path). And the mistake both correct is the same: the synchronous blocking on a seconds-long operation —the waiter who disappears—.

Going deeper: why streaming works, and what async gains besides latency

Why streaming lowers the perception so much. The key is in lesson 2: the LLM's latency grows with the output tokens —each token adds time—. That means the response is produced incrementally, token by token, not all at once at the end. The model already has the first token at ~300 ms; what takes 1500 ms is finishing the 400. Streaming simply delivers each token when it's ready instead of waiting to have them all. Since we humans read slower than the model generates, by the time you finish reading the first paragraph, the next one has already arrived —the wait disappears inside the reading—. Streaming isn't a visual trick: it leverages the incremental nature of generation to turn "wait for everything and then read" into "read while it's generated." That's why almost every LLM chat interface streams: without it, every response would be a blank screen for several seconds.

Async gains more than latency: robustness. Pulling the work off the critical path doesn't just improve the perceived latency; it makes the system more robust. When the "describe your product" generation runs in the background, queued, good things happen: if the LLM is slow or down at that moment, the job waits in the queue and is processed when the model comes back —the seller doesn't see an error, only that their description takes a bit longer—. If synchronous, by contrast, a down LLM means the seller gets an error to their face. The queue decouples the user from the LLM's availability at that exact instant. (In-depth resilience patterns —retries, circuit breakers, the mechanics of queues— are from the resilience guide and this guide's module 5; here it's enough to see that async buys latency and decoupling.) This is an extra reason to prefer async on heavy, non-urgent operations: it doesn't just feel instant, it withstands failures better.

The cost doesn't change with streaming or async. An honest nuance: neither streaming nor async saves money. The model generates the same tokens in both cases, so the cost per call is identical —these are latency techniques, not cost ones—. The cost saving comes from the cascade and the cache (lessons 4 and 5). It's important not to confuse the axes: if your problem is the bill, streaming/async don't touch it; if your problem is that the feature feels slow or blocked, cascade/cache don't fully touch it (the cascade lowers the median latency but not the perceived one on the tail). Each lever to its problem —and that's why lesson 7 composes them, each attacking its constraint—.

Async isn't free in complexity. The pager has a cost: setting up async means a queue, a worker that processes the jobs, a mechanism to deliver the result (polling, notification, websocket), and a handling of states ("in progress", "ready", "failed"). That's more pieces to operate than a simple synchronous call. That's why async is reserved for operations where it really pays off —heavy, non-urgent— and isn't used for everything. A search that takes 200 ms doesn't need async: it would be over-engineering. The rule: async for what takes seconds and the user shouldn't wait for; synchronous (with streaming if it generates text) for what the user does wait for.

Common mistakes

Synchronous blocking on a seconds-long operation (a perceived-latency mistake). What happens: the LLM is put synchronous in the critical path for an operation that takes one or two seconds, and the user stares at a frozen screen with no signal. The feature works —the response arrives— but it feels broken, and users leave believing it hung. Why it happens: synchronous is the easiest to code (call, wait, return) and in development, with one query, the couple of seconds is tolerated. How to spot it: if your AI feature generates text and the user sees a blank screen until it finishes, you didn't give it streaming. How to fix it: if the user waits for the result, stream it (show the first token in ~300 ms); if they don't wait for it, make it async.

Streaming where it should have been async (a critical-path mistake). What happens: a heavy, non-urgent operation —generating a long description, a report— is done with streaming, so the user sees the text appear, but they're stuck at the screen for the two seconds it takes, when they could have gone on to something else. Why it happens: streaming feels modern and is applied by default to everything that generates text, without asking whether the user really needs to keep watching. How to spot it: if the user doesn't need the result right now but you still have them waiting (even watching streaming), you left it in the critical path unnecessarily. How to fix it: for operations the user doesn't consume instantly, use async —take it off the path, give it a pager— instead of chaining it to the screen with streaming.

Believing streaming or async saves money (a wrong-axis mistake). What happens: someone puts in streaming expecting the cost to drop, and is surprised the bill doesn't change. Streaming and async are latency techniques; the model generates the same tokens, so the cost is identical. Why it happens: the two axes of the taximeter (time and money) get mixed and it's assumed that improving one improves the other. How to spot it: if you expected a cost saving from streaming/async, you confused the lever. How to fix it: use each lever for its axis —cascade and cache for cost, streaming and async for latency— and compose the ones that attack your two problems (lesson 7).

Exercises

Exercise 1 — Choose the technique. For each Mercado operation, say whether you'd use synchronous-blocking, streaming, or async, and why: (a) the support agent answering a customer message in a live conversation; (b) generating a weekly sales summary for the seller, sent by email; (c) semantic search returning sorted products (150 tokens, 750 ms).

See solution
  • (a) Live support agent → streaming. The customer is in a conversation and waits for the response to continue; they can't go on without it. But the response can take more than a second if it's long. Streaming is ideal: the customer sees the text appear almost immediately (TTFT ~300 ms) and reads while it completes, instead of staring at a frozen screen. It's exactly the case of Part A of the example.
  • (b) Weekly summary by email → async. The seller doesn't wait for this result in real time —they receive it by email when it's ready—. It's a heavy operation (generating a summary is a lot of text) and non-urgent. Async is correct: the work is queued, generated in the background (even if it takes several seconds, nobody cares), and delivered by email. It would be absurd to have the seller waiting at a screen while it's generated.
  • (c) 750 ms semantic search → synchronous-blocking (or streaming if applicable). 750 ms fits within the 800 ms budget, and the result (a list of products) is probably shown all at once, not as text read word by word. Synchronous is fine here —it's fast and the result is a list, not a narration—. If the search sometimes generates longer responses approaching the budget, streaming would help; but for 750 ms with a structured output, plain synchronous is adequate and not worth complicating. The rule: don't put in streaming or async where synchronous already fits within the budget —it would be over-engineering—.

Exercise 2 — Compute the TTFT. With the stub (strong: base 300 ms, 3 ms/token), compute the time-to-first-token of a response and the total time for three output lengths: 100, 400, and 1000 tokens. What happens to the TTFT as the response gets longer? And to the total time? What does that tell you about why streaming matters more on long responses?

See solution

TTFT = base + 1×ms_per_tok = 300 + 3 = 303 ms (constant!). Total time = base + out_tokens×3:

out_tokensTTFTtotal
100303 ms300 + 300 = 600 ms
400303 ms300 + 1200 = 1500 ms
1000303 ms300 + 3000 = 3300 ms

What happens to each: the TTFT is constant at 303 ms regardless of the length —the first token is always ready at ~300 ms, because it only depends on the base latency plus one token—. The total time, by contrast, grows with the output: 600 ms for 100 tokens, 3300 ms for 1000.

What that tells you about streaming: the gap between what the user waits with streaming (303 ms, constant) and without streaming (the total, growing) widens with the response length. For 100 tokens, streaming saves 600 − 303 = 297 ms (useful but modest). For 1000 tokens, it saves 3300 − 303 = ~3000 ms (enormous!). That's why streaming matters much more on long responses: blocking makes you wait the full 3.3 seconds, while streaming always shows text at 303 ms. On a short response synchronous is tolerated; on a long one, without streaming, it's a blank screen for several seconds. The output length —again the protagonist of lesson 2— decides how much streaming is worth.

Exercise 3 — The async that also toughens. The "describe your product" generator is async. One Monday, the LLM provider has a 20-minute outage. Describe what a seller who requests a description during the outage experiences, and compare it with what they'd experience if the operation were synchronous. What property, besides latency, did async give the feature?

See solution

With async (the real case): the seller requests the description, gets "in progress" in 15 ms (as always), and keeps editing their product. The job enters the queue, but since the LLM is down, it waits in the queue unable to be processed. When the LLM comes back 20 minutes later, the worker takes the queued job and processes it; the description arrives (delayed, but it arrives). The seller experiences, at most, "my description took a while to appear" —a minor annoyance, not an error—.

With synchronous (the counterfactual): the seller requests the description, the system calls the LLM in the critical path… and the LLM is down. The seller gets an error to their face: "the description couldn't be generated, try again." And if they retry during the outage, another error. Their experience is a broken feature for 20 minutes.

The extra property async gave the feature, besides lowering the perceived latency, is robustness / decoupling: the queue separates the seller from the LLM's availability at that exact instant. The work is done when the model can, not when the user requests it, so a temporary outage of the model is absorbed as a delay instead of propagating as an error. Pulling the work off the critical path doesn't just improve how the wait feels; it makes the system not go down when the LLM goes down —which is exactly the kind of resilience this guide's module 5 covers in depth—. Async is, at once, a latency technique and a first line of defense against the LLM's failures.

Summary and next step

In this lesson you attacked the perceived latency, the problem neither the cascade nor the cache solves. With the kitchen of three waits you saw the central distinction: the dish takes the same, but how the wait is handled decides everything. Streaming —the bread that arrives immediately— shows the response token by token: you measured that the support agent's TTFT drops from 1500 ms blocking to 303 ms (-80%), even though the real work still takes 1500, because the generation is incremental and the first token is already at ~300 ms. Async —the pager— pulls the work off the critical path: "describe your product" drops from 2100 ms synchronous to 15 ms perceived, by moving the work to the background. You learned when to use each (streaming if the user waits for the result, async if they don't), that the mistake both correct is synchronous blocking on seconds-long operations, that async also toughens (the queue decouples the user from the LLM's availability), and that neither saves money —they're latency levers, not cost ones—.

Before moving on you should be able to: distinguish real latency from perceived latency; choose between synchronous, streaming, and async depending on whether the user waits for the result; explain why streaming lowers the perception so much (incremental generation, constant TTFT) and why it matters more on long responses; and know that streaming/async attack latency, not cost, and that async brings decoupling for free.

What follows is putting it all together. You now have the four pieces —budget, cascade, cache, async/streaming— and in lesson 7 you're going to compose them into a single cost-aware request path, applied to Mercado's support agent. You're going to see in what order they're applied (cache first, then cascade, with the budget as gate), and you're going to measure —by executing— that the techniques accumulate but don't add up (they overlap over the same cheap traffic), and that no single one is enough but together they bring the feature within its budget. It's the module's synthesis.

Resources

← Previous5. Cache: many queries repeatNext →7. The cost-aware request path