Engineering4 min read
Designing alerts nobody mutes
An alert that fires constantly teaches your team to ignore it. How to tune thresholds so a notification still means something.
PPriya RaghunathanOperations Lead
The muting spiral
Noisy alerts get muted. Muted alerts miss the one incident they existed for. The fix is not more discipline, it is fewer and better alerts.
Alert on impact, not on inputs
A spike in requests is not a problem. A spike in failed checkouts is. Point alerts at the outcome your team is accountable for.
Use bands, not single thresholds
Compare against the same hour last week rather than a fixed number. Traffic has a shape, and a flat threshold ignores it.
Every alert needs a runbook
If the person woken up does not know the first three steps, the alert is incomplete. Attach the steps to the alert itself.
Review the log monthly
Count how many alerts led to action. Anything below half needs retuning or retirement.
