Back to the journal

Example postmortem

A rollout that pushed latency past budget

A fictional postmortem following a latency spike from detection through rollback and recovery.

Fictional example. The service, timeline, and measurements are illustrative.

Summary ↑

In this fictional incident, a new release of an order lookup service made an extra backend call on most requests. The service’s 95th-percentile response time rose from roughly 200 milliseconds to more than one second. A rollback brought it back near baseline.

Impact ↑

For about 17 minutes, users in the rollout cohort saw slower order lookups. Requests still succeeded at close to the normal rate, which meant the error-rate view alone gave responders an incomplete picture.

Timeline ↑

  • 09:38 UTC — Baseline. The order lookup service is serving traffic at roughly 200 milliseconds p95.

  • 09:41 UTC — Rollout expands. The release goes from a small cohort to 25% of traffic.

  • 09:46 UTC — Detection. A responder sees sustained latency above 1,000 milliseconds and compares the new and previous release cohorts.

  • 09:50 UTC — Rollback. The team returns all traffic to the previous version.

  • 09:58 UTC — Recovery. Latency returns near baseline and holds steady under normal traffic.

Contributing factors ↑

The release introduced a sequential lookup on a frequently used request path. Load testing covered overall throughput but used test data that rarely exercised the extra lookup. The rollout gate evaluated successful responses and missed the slowdown because it had no latency threshold.

Follow-up ↑

  • Add a latency gate that compares the new release with the previous version under similar traffic.

  • Include representative lookup patterns in load tests and record p95 for each common request path.

  • Add a version comparison to the on-call dashboard so responders can check whether degradation is concentrated in a rollout cohort.

The next staged release should demonstrate that the gate detects the same slowdown before the rollout expands.

See all journal posts