What 2.4 billion deliveries taught us about endpoint health

In September we delivered 2.4 billion events. We pulled a month of attempt logs to answer one question: can we predict which endpoints will fail before they do? The short answer is yes, and the signal is cheaper than we expected.

Failure is concentrated

0.6% of endpoints produced 71% of failed attempts. Those endpoints were not random: nearly all of them showed rising p95 latency for at least twenty minutes before the first 5xx. Latency drift is the earliest symptom of a backend that is about to tip over.

What we changed

The endpoint health view now scores each endpoint on latency trend, recent status mix and time since last success. A score below the threshold triggers a warning before the circuit breaker opens, so you can page the owner while the endpoint is still mostly working.

We also stopped treating a 429 as a failure in the score. Rate limiting is the endpoint telling you something useful, and we now schedule around it instead of counting it against them.