--- title: Alerting That Does Not Cry Wolf description: "How we designed our alerting rules to minimize noise and maximize signal for a travel booking platform — severity thresholds, composite alerts, and pruning." canonical: https://nowah.xyz/blog/alerting-does-not-cry-wolf lastModified: "2026-08-07T03:50:48.751Z" --- # Alerting That Does Not Cry Wolf How we designed our alerting rules to minimize noise and maximize signal for a travel booking platform — severity thresholds, composite alerts, and pruning. 47 alerts fired last week. The on-call engineer acknowledged a few, investigated none, and resolved zero. They all resolved themselves. Transient spikes, scheduled maintenance, and a monitoring system with overly sensitive thresholds. Then on Friday night, alert number 48 fired. A real outage. Payment webhook processing stopped. The on-call engineer saw the notification, assumed it was another false alarm, and went to bed. The outage continued for two hours before someone happened to check the dashboard. Alert fatigue is a real operational risk. Too many alerts trains the on-call team to ignore alerts. Which means when a real incident happens, the response time degrades to however long it takes someone to manually notice. All those alerts meant to improve reliability actually made it worse. We overhauled our alerting system after that incident. Here's what we changed. ## The rule: every alert must require action ![Illustration for this section](https://pics.nowah.xyz/website-media/infrastructure-030-img-1.webp) Our new standard is simple. If an alert fires, the on-call engineer must need to do something. If the alert resolves itself without human intervention, it shouldn't have fired. This eliminated about 70% of our alerts. Transient spikes that self-correct? Removed. Scheduled maintenance windows that temporarily degraded metrics? Suppressed during maintenance. Load balancer [health check](/blog/health-check-hierarchy) failures that resolve when the container restarts automatically? Covered by the orchestrator, not by alerts. What remains are alerts that actually need a human: booking success rate dropping below 98%, [payment processing](/blog/launching-payment-processing-ai-handles-money) failing above 0.5%, queue depth growing with no worker activity, and database connection pool exhaustion. ## Severity-based thresholds Not all alerts need the same response. A 2% drop in booking success rate at 3 PM is different from a 2% drop at 3 AM. The first might have a larger blast radius because more travelers are booking. But both need the same response speed because every minute of degradation means failed bookings. We classify alert severity based on the metric and the threshold: **P1 (wake-up call).** Booking success rate below 98%. Payment processing success below 99.5%. Core service uptime violation. The on-call engineer gets paged immediately. These thresholds are set tight because the impact of these failures is immediate and financial. **P2 (urgent but not emergency).** AI agent response time above 10 seconds sustained. Notification delivery delayed by more than 10 minutes. Queue depth growing persistently. The on-call engineer gets a notification and has 15 minutes to respond. **P3 (next business day).** Non-critical service degradation. Dashboard rendering issues. Minor analytics delays. These create tickets, not pages. **P4 (informational).** Metrics that deviate from normal but don't indicate a problem. These go to a dashboard, not to a person. ## Composite alerts ![Supporting diagram](https://pics.nowah.xyz/website-media/infrastructure-030-img-2.webp) Simple threshold alerts generate false positives. A brief CPU spike triggers an alert that resolves before the engineer even sees it. A single health check failure triggers an alert, but the service recovered on the next check. Composite alerts combine multiple signals to reduce false positives. Our P1 booking alert doesn't fire on a single data point. It fires when: - Booking success rate is below 98% AND - The condition has persisted for at least 2 minutes AND - More than 10 booking attempts have been made in that window (to avoid false positives during low-traffic periods). All three conditions must be true simultaneously. A 1-second dip that recovers doesn't trigger the alert. A sustained drop during active booking traffic does. For queue health, the composite is: queue depth above threshold AND worker processing rate below normal AND the condition persists for 2+ minutes. This catches real queue problems while ignoring brief spikes that workers catch up with quickly. ## Escalation policies An alert without an owner is useless. Our escalation policy ensures every alert gets acknowledged within a defined window. When a P1 fires, the on-call engineer has 5 minutes to acknowledge. If they don't acknowledge (asleep, phone on silent, in a dead zone), the alert escalates to the backup on-call. If the backup doesn't acknowledge within another 5 minutes, it escalates to the engineering lead. Acknowledgment doesn't mean resolution. It means "I'm looking at it." The engineer might acknowledge, investigate for 15 minutes, and then escalate manually because the issue is outside their expertise. That's fine. The point is that someone is engaged. For P2 alerts, the acknowledgment window is 15 minutes before escalation. P3 and P4 don't have escalation because they create tickets, not pages. ## Quarterly alert review Every quarter, we review every active alert rule. The review asks: **Did this alert fire in the last quarter?** If not, is it still relevant? Alert rules for features that were deprecated, infrastructure that was replaced, or thresholds that no longer apply should be removed. **When it fired, was action required?** If the alert fired 12 times and the engineer took action zero times, the alert is noise. Either the threshold needs adjustment, the alert needs to be removed, or the underlying issue needs to be fixed permanently. **Was the threshold appropriate?** If the alert fired at the right time but with the wrong severity, adjust it. A P1 that should have been a P2 degrades trust in P1 alerts. **Are there incidents that should have had alerts but didn't?** This is the other failure mode. Review recent incidents and check whether earlier alerting would have reduced impact. If so, add or tighten alert rules. The quarterly review keeps the alert system honest. Alert rules accumulate over time. Without pruning, the system gradually becomes noisier, and alert fatigue returns. ## The noise-to-signal checklist If your alerting system is noisy, here's the audit process. Count alerts over the last month. How many fired? How many required human action? The ratio of actionable alerts to total alerts is your signal-to-noise ratio. Target above 80%. Identify the top 5 noisiest alerts. These are the ones that fire most frequently. For each one, ask: can the threshold be raised? Can the alert be composite (requiring sustained condition)? Can the underlying issue be fixed permanently? Check acknowledgment rates. If alerts are frequently unacknowledged before escalation, the team has stopped trusting the system. Fix the noise problem first, then re-establish the expectation that alerts are acknowledged. Test your P1 alerts work. Send a test P1 during a non-critical time and verify the on-call engineer receives it, acknowledges it, and follows the escalation path. Don't assume the alerting system works just because you configured it. The goal isn't zero alerts. It's zero useless alerts. Every alert that fires should make the on-call engineer think "I need to look at this," never "Not this again." --- Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. [Plan your next trip](https://app.nowah.xyz).