"The AI API is down" and "the AI API is slow" produce the same feeling — you're staring at a spinner — but they're different problems with different fixes. LLM calls are legitimately slow sometimes (long outputs stream token by token), and a client timeout set too aggressively will turn ordinary latency into a fake outage. Before you page anyone, rule out the boring explanations.
Why LLM calls are slow even when everything's fine ¶
- Output length. Generation time scales with tokens produced. A long answer simply takes longer — that's throughput, not an outage.
- Time-to-first-token vs. total time. A model can start streaming quickly but take many seconds to finish. If you measure only total time, a long-but-healthy response looks like a hang.
- Cold capacity / queueing. During demand spikes, requests queue. Slower, but up.
- Big context. Large prompts take longer to process before the first token appears.
The tell-tale differences ¶
| Signal | Slow (up) | Down |
|---|---|---|
| Response eventually arrives | Yes, just late | No — it errors or never connects |
| Status code | 200 (streaming) | 5xx, or connection refused/timeout |
| Multi-region probe | All regions reachable | All (or most) regions fail |
| Provider status feed | Green (maybe "degraded performance") | Acknowledged incident |
| Smaller/shorter request | Returns fast | Still fails |
A 60-second triage ¶
- Send a tiny request. Ask for one sentence with a short prompt. If it returns quickly, the provider is up and your slow calls are just large — the fix is your side (streaming, timeouts, smaller context).
- Probe from four regions. A multi-region check of the API endpoint tells you whether the endpoint is reachable everywhere or failing globally. Reachable everywhere + your calls slow = latency, not outage.
- Read the code. A 200 that's slow is throughput. A 5xx is an outage. A 429 is throttling — up, but refusing you.
- Check the board. The AI status board and the provider's feed will say "degraded performance" during a slow-but-up period versus a full incident.
Fixes for "slow but up" ¶
- Stream responses so users see tokens immediately instead of waiting for the whole answer — this alone removes most perceived "hangs."
- Set timeouts to time-to-first-token, not total time, so a long-but-healthy stream isn't killed mid-answer.
- Shrink context and output caps where you can; both directly cut latency.
- Add a fallback to a faster model or a queued retry so a slow window degrades gracefully instead of erroring.
FAQ ¶
How do I know if an AI API is down or just slow?
Send a tiny request and probe the endpoint from multiple regions. If the small request returns and the endpoint is reachable everywhere, it's up and your slow calls are large or queued. If requests error with 5xx or the endpoint fails across regions, it's a genuine outage. The status code is the tiebreaker: slow-200 is latency, 5xx is down, 429 is throttling.
My request times out but the provider says it's healthy — why?
Usually your client timeout is shorter than the response needs. LLM outputs stream over many seconds; if you cap total request time too low, a normal long answer gets killed and looks like an outage. Base your timeout on time-to-first-token and stream the rest.
Does "degraded performance" on a status page mean it's down?
No — it means up but slower or with elevated errors. Requests still succeed, just late or after retries. It's the middle state between fully healthy and a full incident, and it's usually best handled with streaming, backoff, and a bit of patience rather than treating it as an outage.