Reliability Engineering
The incident ended. What still needs fixing?
Restoring a service buys time. It does not explain the failure or establish that the next incident will be easier. We investigate the dependency behind the symptom and turn that understanding into changes the team can test.
Start with the failed request.
An alert names the service that noticed a problem. The cause may be several calls away. Useful telemetry lets someone follow that path without reconstructing it from separate dashboards.
A service restarting doesn’t establish that recovery worked. Pending work, retries and downstream state need checking too.
What the investigation needs to establish
Which dependency failed?
Connect the user-visible symptom to a specific failure path. Missing telemetry is an engineering finding, not a reason to stop at the nearest error log.
What can the team recover?
Test the procedure people are expected to use. Include the degraded state, the intervention and the checks that establish recovery.
When the same incident returns
The immediate fix became permanent
A restart or a larger limit can restore service while leaving the failure mode intact. We separate the immediate remedy from the work still needed.
The runbook has never been exercised
A plausible procedure can fail when a dependency is unavailable. Practising under agreed conditions exposes those assumptions.
Make the next incident less uncertain
Agree a failure scenario
Choose a dependency and an operational consequence the team needs to understand.
Exercise it
Observe detection and recovery under controlled conditions. Record where the procedure needs judgement or missing information.
Change the conditions
Use the findings to improve the software or its operation, with a named owner for the remaining work.