Streaming, and the 125-second edge timeout that kills long calls

A long AI API call has a failure mode that has nothing to do with the model, the prompt, or your code, and everything to do with the infrastructure between you and the provider: at roughly 125 seconds, an edge in the path gives up on you.

We found it on real traffic — a customer's long analysis calls dying at almost exactly two minutes with an error page their SDK could not parse. The page was Cloudflare's 524, "a timeout occurred", rendered as HTML. Not the provider's JSON error envelope; the edge's error page, which means the provider may well have still been generating a perfectly good response that now had nowhere to go.

Why 125 seconds#

Cloudflare's proxy enforces a Proxy Read Timeout of about 100–125 seconds (documented, with around a second of stated imprecision) on the leg between its edge and the origin it is proxying for. When your request path includes Cloudflare — and for an AI API call it very plausibly does twice, since api.anthropic.com sits behind Cloudflare's edge and your own infrastructure may too — a connection that produces no bytes for that long is cut, and the caller receives a 524.

Here is the trap: a non-streaming AI call is byte-silent by design for its entire generation. The provider accepts the request, thinks for three minutes, and sends the whole response at once. From the edge's point of view that is indistinguishable from a dead origin. The connection is not idle in any sense that matters to you — the model is working — but no byte has crossed the wire, and no byte crossing the wire is the only signal the edge trusts.

Worth stating what is not the killer here, because the obvious suspect is innocent: a Cloudflare Worker has no wall-clock limit on an HTTP request, and no time limit on an individual subrequest — waiting on a fetch consumes no CPU budget. The compute platform is fine. It is the proxying edge in front of an origin that cuts, and it cuts on silence, not on duration.

Streaming is the fix, not an optimisation#

With "stream": true, the provider starts emitting bytes within a second or two — SSE chunks, one delta at a time — and never goes quiet long enough for any edge to classify the connection as dead. The same three-minute generation completes fine, because the timeout is on silence, not on total duration. This reframes a decision most teams make on UX grounds: streaming is not just for chat interfaces that want tokens to appear live. Past some call length, streaming is the only reliable transport. Batch jobs, background summarisation, agent steps with big outputs — the calls least likely to have streaming turned on are exactly the calls long enough to need it.

Streaming has its own metering traps — on some providers a streamed call reports zero tokens unless you ask for usage explicitly — but they are all solvable, and none of them kills the call.

What a proxy in the path does about it#

Vigil sits between the app and the provider, which puts the edge cut inside our blast radius: if the upstream leg dies at ~125s, what does the customer see? Two decisions from building that path are transferable.

Cut just under the documented ceiling, on purpose. The proxy arms a 115-second deadline on the upstream fetch for non-streaming calls — under the only documented cutoff in the path, with headroom for the edge's stated imprecision. The point is to convert an unparseable HTML death into a parseable structured error the SDK can act on. The direction of the number matters more than its value: every call that completes today still completes, and only calls that were going to die at the edge anyway get a better death. Cutting lower than the documented ceiling would start failing 90-second calls that work today — a regression dressed as a fix.

Disarm the timer when the response arrives. The tempting implementation, AbortSignal.timeout(), stays armed after the response resolves — fire it at 115s while a slow body stream is still being read and you sever a response that was flowing fine. A manual timer that is disarmed the moment headers arrive kills only calls that are actually stuck. Timeout code has to know the difference between "no response yet" and "response arriving slowly"; conflating them turns a guard into a new failure mode.

And when the edge beats you to it anyway, what arrives is HTML where JSON should be — which your error handling has to survive; that is its own post — and the classified result lands per call on the dashboard rather than in a support thread three days later.