This is not a hypothetical. On this site, /check/netflix.com once read "DEGRADED, HTTP 200, 100.00% uptime, 3930/3930 probes succeeded" while /outages read "netflix.com — Down for 61h 35m". Same service, same minute, same database. Both pages were rendering honestly from their own source.
How both pages were right ¶
Four probe methods run against every target: HTTP/1.1 with strict TLS, HTTP/2, a GET with a browser User-Agent, and a HEAD with a curl User-Agent. They differ by request shape on purpose, because sites treat request shapes differently — and one of them failing while the others load the page is a fact worth showing.
The rollup called that state degraded. The check page rendered it as degraded, correctly, next to a 200 and a full uptime history.
The incident machine did something else. It treated anything that was not exactly up as a failed check. So a single flaky method out of four crossed the failure threshold, opened an incident, and that incident stayed open for two and a half days — because resolving it required consecutive clean checks and the flaky method kept flaking.
Nobody wrote a bug. Two subsystems held different definitions of the same word, and neither knew the other existed.
The four ingredients ¶
Every false-positive alert I have seen since has had some subset of these.
1. Two definitions of "down"
The root cause here. If the dashboard, the alert rule, the status page and the uptime figure each decide independently what counts as a failure, they will disagree, and the disagreement surfaces as an alert nobody can reconcile with what they are looking at. The fix is not better thresholds; it is one definition, in one place, that every surface reads.
2. A single vantage point treated as global evidence
A probe that fails from one network proves that one network could not reach the target. It does not prove the target is down, and the difference matters most for exactly the sites people care about: airlines, banks and streaming services routinely refuse datacenter traffic while serving browsers normally. An alert that says "down for everyone" from one egress is overstating its evidence. More on why even several vantage points do not settle it.
3. A timeout shorter than the thing being measured
A check with a five-second timeout against a service whose cold start takes eight seconds will page you on every deploy. The alert is real in the sense that the check failed; it is false in the sense that nothing was wrong. Choosing thresholds without masking real outages.
4. No debounce, or debounce in only one direction
Alerting on a single failed check makes every transient blip a page. But the asymmetry matters more than most people expect: this system requires two consecutive failures to open an incident and two consecutive successes to close one, and the second half is what stops a flapping service from generating an alert storm. A counter that resets on any opposite result is what prevents fail-pass-fail-pass from ever crossing a threshold.
What actually fixed it ¶
Three changes, in order of how much they mattered.
- One shared definition of availability.
degradedmeans some probe methods failed while others loaded the site — the service served, so it counts as available. Impairment is reported as its own separate fact, never folded into an uptime percentage and never sufficient to open an incident. - An evidence model instead of a boolean. Each method now resolves to a class: reachable, service error, method unsupported, probe blocked, rate limited, or unknown. A method that does not apply — no HTTP/2 support, a HEAD refused with 405 — does not vote at all. Neither does one that was refused with a 403, because being denied tells you nothing about whether the service works.
- Unconfirmed by default. A down verdict from one egress is marked unconfirmed unless independent vantage points agree. The word "confirmed" is reserved for evidence that earns it.
The general lesson ¶
False positives are almost never a threshold problem, which is unfortunate because thresholds are the easy thing to adjust. They are usually a definition problem: two parts of a system quietly disagreeing about what a word means, with the disagreement only becoming visible when the two are shown on the same screen.
The diagnostic question is not "was this alert wrong?" but "what would have had to be true for this alert to be right?" — and then checking whether the system believes that. When the answer is "the incident machine thinks one failed method is an outage and the uptime figure does not", you have found something a threshold cannot fix.
What to do with an alert you suspect is false ¶
- Check the thing a user does, not the thing the monitor checks. If the homepage loads and the API is dead, both the alert and the dashboard can be right.
- Compare the failing check's definition of failure against the dashboard's. If they differ, that is your answer.
- Look at whether the failure is one method, one region, or everything. One of anything is rarely an outage.
- Do not silence it until you know which. A muted false positive and a muted real one look identical afterwards.