How do you design alerting that doesn't cause alert fatigue — what does SLO-based alerting look like?

#Alert fatigue is a design failure, not a tooling problem

Alert fatigue happens when the volume and noise of alerts trains the humans receiving them to stop trusting alerts as a signal — they get silenced, snoozed, or acknowledged on autopilot, and by the time a genuinely serious page arrives it gets the same half-attention as the ninety false alarms that came before it. The root cause is almost never "we need a better paging tool." It's that the alerting rules were designed around what's easy to measure (a metric crossed a threshold) rather than around what actually matters (are users being harmed, right now, badly enough to justify waking someone up).

#Cause-based alerting: technically correct, practically exhausting

The naive approach to alerting is to instrument every plausible failure mode and page on each one individually: CPU above 80%, disk above 90%, a pod restarted, GC pause exceeded 200ms, one replica out of twenty is unhealthy, queue depth above some arbitrary number. This is cause-based alerting — you're alerting on a potential root cause of a problem, not on the problem itself.

The trouble is that causes are numerous, frequently self-correcting, and only sometimes actually produce user impact. A pod restarting is normal during a rolling deploy. CPU at 85% for two minutes during a traffic spike that autoscaling handles fine is not an incident. If every one of these pages a human, most pages are false positives relative to the thing you actually care about — whether customers can use the product — and that's exactly the pattern that trains people to ignore alerts.

#Symptom-based / SLO-based alerting: alert on user pain

The alternative is to alert on symptoms — the user-visible behavior your Service Level Objectives are defined against, typically availability (proportion of successful requests) and latency (proportion of requests under some threshold). Instead of "disk is at 90%," the alert is "the error rate for checkout-api has been above what's sustainable to still hit our monthly SLO." The cause — is it disk, is it a bad deploy, is it a downstream dependency — becomes something the on-call engineer investigates after being paged, using dashboards and traces, not something that pages them directly.

This single change collapses dozens of potential cause-based alerts into a small number of symptom-based ones per service, because almost every failure mode you'd otherwise page on individually — bad deploy, disk full, downstream outage, GC thrashing, a noisy neighbor — manifests as the same two or three user-visible symptoms: more errors, or slower responses. You still want dashboards and lower-urgency causes visible for the investigation phase; you just don't want a human's phone to ring for all of them.

#Burn-rate alerting: the mechanism that makes SLO alerting actually work

Here's the part that's easy to get wrong: you cannot alert "we breached our SLO" directly, because an SLO is defined over a long window (commonly 28 or 30 days), and by the time you've actually breached a 30-day SLO, the damage is done and the alert is useless — you want to know hours before the SLO would be breached, and you want to know how urgently.

The mechanism is burn rate: how fast you're consuming your error budget relative to the rate that would exhaust it exactly at the end of the SLO window. A burn rate of 1x means "consuming the budget at exactly the rate that empties it right at the boundary of the window — business as usual." A burn rate of 10x means "at this rate, the entire 30-day budget will be gone in three days." The key insight from Google's SRE practice (multi-window, multi-burn-rate alerting) is that you evaluate burn rate over multiple time windows simultaneously, because a short window alone is too noisy (a two-minute blip triggers a page for something that self-resolves) and a long window alone is too slow (a real fast-burning incident takes hours to page on if you only look at a 6-hour average).

The standard pattern uses two paired conditions:

• Fast burn — e.g., burn rate ≥ 14.4x sustained over a 1-hour window (with a 5-minute short window as a confirmation to avoid triggering on a single noisy sample) — pages immediately. At that rate, 2% of the entire monthly error budget is gone in one hour; left unchecked, the full budget is gone in about two days.
• Slow burn — e.g., burn rate ≥ 6x sustained over a 6-hour window — creates a ticket or a lower-urgency notification, not a page. It's real, but it's not "wake someone up at 3 a.m." urgent.

#What actually goes into an SLO-based alerting policy

The four golden signals from Google's SRE book — latency, traffic, errors, and saturation — are still the right dimensions to instrument, but the point of SLO-based alerting is that only errors and latency (the two your users directly experience) typically get to page a human; traffic and saturation stay as dashboard/diagnostic signals you look at after a symptom-based page, to find the cause.

🧠 Check your understanding
0/3 · 0/3 answered
1. What is the main problem with purely cause-based alerting (e.g., paging on every instance of CPU > 80%, disk > 90%, or a pod restart)?
It only works for stateless servicesMost individual causes don't actually translate into user-visible harm, so the volume of pages trains on-call engineers to ignore themIt's technically impossible to implement in PrometheusCause-based alerts cannot be sent through PagerDuty or Opsgenie
2. Why do burn-rate alerts typically evaluate two windows at once (e.g., a 1-hour window with a 5-minute confirmation) instead of a single window?
A single short window alone is too noisy (fires on transient blips) and a single long window alone reacts too slowly to a genuinely fast-burning incidentIt's purely a legacy convention with no technical reasonPrometheus requires at least two range vectors per alerting ruleTo reduce the load on the Prometheus server
3. In SLO-based alerting, what typically triggers a page directly to a human, versus what's investigated after the page?
Only symptom-level SLIs like error rate and latency page directly; underlying causes (disk, CPU, GC, individual pod health) are investigated afterward using dashboards and tracesOnly disk and memory metrics page; error rate and latency are considered too noisy to alert onPaging is randomized across all available metrics to avoid engineer biasEverything possible is paged directly, including CPU, disk, and GC metrics, for maximum coverage