My AI Feature Took 85.9 Seconds. The Model Was Only One Bottleneck.

One of Wimemo's AI plan requests took 85.9 seconds to answer.

That is not “a little slow.”

That is enough time for a user to decide the feature is fake, broken, or not worth trusting again.

My first instinct was to blame the model.

That felt reasonable. The feature was generating travel advice. Recent plan turns had an 8.9s median time to first token, a 21s p90, and a tail bad enough to produce that 85.9s outlier. On the surface, this looked like a familiar AI-product story: we picked a model that was too smart, too heavy, or too expensive for an interactive user flow.

That diagnosis was incomplete.

The model was only one bottleneck in a longer chain.

Once I separated model time from retry time, proxy buffering, pre-model context work, and output-shape overhead, the problem stopped looking like “LLMs are slow” and started looking like a real systems bug.

That distinction matters because users do not experience your inference stack in layers.

They experience one wait.

Wimemo is an iPhone app for private travel memory, but it now also includes AI features for planning and reflection. A user who asks for a travel plan is not benchmarking my model choice. They are testing whether the product feels dependable enough to help with a real trip. If the first useful answer appears too late, the details of which layer caused the delay are irrelevant. The product already lost.

The best evidence came from the mismatch between what the model was doing and what the user was seeing.

After I removed the old timeout-and-retry behavior, a live global request still timed out through OpenRouter. But a direct DeepSeek V4 Flash request returned its first model token in about one second. China exposed an even stranger clue: model TTFT was 1.37s, yet the client did not receive visible output until the 12.55s final payload.

At that point, “the model is slow” stopped being a satisfying story.

The user-visible delay was being created by multiple layers, and each one needed a different fix.

Layer 1: Retry Policy Turned One Slow Request Into A Minute-Long Lie

The 85.9-second outlier was not a pure inference event.

It was a policy mistake.

The old request path allowed a 60s timeout and then did a full retry. That sounds safe if you are thinking like infrastructure. It is terrible if you are thinking like an interactive product. A timeout is already evidence that the current request failed its user-facing contract. Retrying the same plan turn inside the same visible wait window did not save the experience. It only made the bad wait longer and less interpretable.

So the first rule changed:

Timeout is terminal for interactive AI work unless you have evidence that the retry can still land inside the user's patience budget.

In Wimemo's case, that meant a 25s total model budget, a 20s attempt timeout, and retries only for fast connection or upstream 5xx failures that still fit inside the same overall budget.

That one decision removed the most embarrassing tail, but it still did not make the feature feel fast.

Because the model was not the only thing the user was waiting for.

Layer 2: Streaming Was “Enabled,” But The Gateway Was Still Buffering

This was the most counterintuitive part of the incident.

The app already used SSE. I thought that meant the user would see text as soon as the model started producing it.

Not necessarily.

In China, the model began responding quickly, but the answer was being held until the end of the request. The transport layer was defeating the entire point of streaming. From the user's perspective, a fast-streaming model behind a buffering gateway is just another slow endpoint.

The fix was not “optimize the prompt.”

It was to make the stream unambiguously stream:

• Cache-Control: no-cache, no-transform
• Content-Encoding: identity
• X-Accel-Buffering: no
• an invisible padded open frame at the start of the SSE response so gateways flush early instead of waiting for a larger body

This is the kind of failure AI teams miss when they only measure total latency and model latency. The model can be fine. The user can still get a frozen screen.

Once I saw China return first byte / first answer / final at 3.13s / 3.99s / 6.35s, the lesson was obvious:

If the user cannot see the first token, TTFT is not the whole truth.

You need a first-byte metric and a first-meaningful-answer metric as separate numbers.

Layer 3: Pre-Model Work Was Quietly Eating The Budget

Before the model even started thinking, the system was doing too much.

Some of it was individually reasonable: fetching freshness state, assembling conversation context, preparing planning inputs. Together, it was enough to turn every request into a longer wait than it needed to be.

Two problems stood out.

First, the request path was reading freshness inputs through multiple calls when one service-side RPC could answer the same question. Second, turn persistence and context assembly were too serialized. The system was spending interactive time on bookkeeping that could overlap or be simplified.

The new path fixed both:

• freshness reads moved from eight PostgREST calls to one service-only RPC
• turn persistence overlaps with context assembly
• the current idempotent turn is excluded from conversation history because the message is already added separately
• the pre-model budget became explicit: 2s globally and 5s for the China self-hosted cold path

This changed how I think about AI feature performance.

I used to mentally divide the request into “before AI” and “during AI.”

That was too vague to be useful.

Now the chain is measured as:

1. pre-model
2. model TTFT
3. model total
4. tool time
5. post-model
6. user-visible total

That decomposition is not just observability polish. It is what turns a slow feature from folklore into an actual engineering problem.

Layer 4: Output Scope Was Too Ambitious For The First Draft

Even after transport and pre-model work improved, there was still a product judgment issue hiding in the request.

I was asking an interactive plan feature to act like a report generator.

The old mental model was: if a smarter model produces a deeper answer, use that by default. But reasoning depth has a latency cost, and not every first response deserves to pay it. OpenAI's own reasoning guidance and model guidance make the tradeoff explicit: more reasoning can improve quality, but it also increases latency and token use. That is a real product decision, not an abstract benchmark preference.

For Wimemo, the right default was not “maximize thought.”

It was “return the first useful draft inside the interaction budget.”

So ordinary plan creation changed shape:

• default to DeepSeek V4 Flash for interactive advice and plan creation
• keep the initial JSON compact
• return no day-by-day itinerary unless the user explicitly asks for it
• return exactly three preparation items and a short summary for the first pass
• repair near-valid provider JSON locally instead of sending the same task back for another expensive round trip

That is not dumbing the feature down.

It is matching the first response to the moment the user is in.

A user asking for an initial Tokyo trip draft usually wants orientation first, not exhaustive detail at the cost of a long blank wait. If they want itinerary depth next, the product can still give it. But it should earn the second step by making the first step feel alive.

The Result Was A Better Contract, Not Just A Faster Number

After the phase-two changes, the production check for the same plan flow measured:

• global: 2.02s first byte, 2.90s first answer, 5.11s final
• China: 3.13s first byte, 3.99s first answer, 6.35s final

Those results matter because they describe the experience the user actually receives, not just the time a model started emitting tokens on some internal trace.

They also changed the durable rule I would reuse on any AI product:

Bind interactive AI to a user-visible latency budget before you maximize reasoning depth.

That rule sounds obvious only after you have felt the alternative in production.

If I were handing this lesson to another founder or product engineer, the checklist would be:

1. Measure first byte, first meaningful answer, and final separately.
2. Treat timeout retry as a product decision, not only an infrastructure default.
3. Confirm that “streaming” actually reaches the client progressively.
4. Give pre-model work its own explicit budget.
5. Shrink the first response until it fits the moment the user is in.
6. Use the smartest model only where its extra reasoning survives contact with the latency budget.

Where Human Judgment Still Matters

There are still limits. Explicit multi-day itinerary requests may still take longer. Regional paths still differ. Direct DeepSeek JSON-object mode means structured-plan E2E coverage has to stay strong. And there will always be flows where depth matters more than speed.

But the thesis I am keeping is simple:

When an AI feature feels slow, the model is often only one layer of the wait. The product gets faster when you budget the whole chain.

I write these from building Wimemo, a private travel memory app. If you want the product context behind the engineering decisions, you can read more about Wimemo here.