Incident Response for a Travel Booking Platform

2:47 AM. The alert fires. Payment webhook processing has stopped. The queue depth graph shows a cliff: jobs are piling up with zero being processed. Looking at the dashboard, roughly 200 bookings in the last 20 minutes have confirmed reservations but no confirmation emails. Travelers have been charged but haven't received their booking details.

This is a P1 incident. The booking path itself is working (travelers can still book), but the post-booking experience is broken. People are paying for flights and getting silence in return. Every minute that passes, more travelers join the "where's my confirmation?" group.

Here's how we handle it.

Severity classification

Not all incidents are equal. A slow analytics pipeline is annoying for us but invisible to travelers. A broken payment flow means people can't book trips. We classify incidents by severity to determine the response speed and team size.

P1 (Critical). The booking or payment path is broken, or traveler-facing services are down. Booking success rate drops below 98%. Payment processing drops below 99.5%. Response: immediate, all hands, external communication.

P2 (High). Significant degradation to traveler-facing features. AI agent response time exceeds acceptable thresholds. Notification delivery is delayed by more than 10 minutes. Response: within 15 minutes, on-call team, internal communication.

P3 (Medium). Non-critical feature degradation. A specific search route returns errors. Document upload is slow. Response: within 1 hour, assigned engineer.

P4 (Low). Minor issues with no traveler impact. Internal dashboard broken. Non-critical monitoring gap. Response: next business day.

The classification depends on two dimensions: how many travelers are affected and whether the booking path is involved. A bug affecting 1% of users that blocks bookings is P1. A bug affecting 50% of users that delays analytics by an hour is P3.

The first 10 minutes

When a P1 fires, the on-call engineer has 10 minutes to assess the situation and start mitigation. Not fix. Mitigate.

The sequence:

Minute 0-2: Acknowledge the alert. Open the monitoring dashboard. Identify what's broken and what the blast radius is.

Minute 2-5: Check recent deployments. Most incidents correlate with recent changes. If a deploy happened in the last hour, it's the prime suspect. If it looks like the cause, roll back immediately. Don't investigate first, roll back first.

Minute 5-10: If it's not a deploy, start gathering data. Which services are affected? Are health checks passing? Is it an internal failure or an external dependency issue? Start a communication thread for the team.

The key discipline in the first 10 minutes is bias toward action. Roll back the deploy. Restart the stuck worker. Kill the runaway process. Fix first, understand later.

Escalation

The on-call engineer handles P2-P4 incidents alone. P1 incidents escalate.

If the on-call engineer can't identify or mitigate the issue within 15 minutes, they escalate to the engineering lead. If the engineering lead can't resolve it within 30 minutes, it escalates further. Each escalation brings in someone with broader system knowledge and authority to make decisions (like taking a service offline or switching to a backup provider).

We don't escalate based on "I need help." We escalate based on time. If 15 minutes pass without mitigation, the issue is harder than one person can handle quickly. Fresh eyes and more experience speed up resolution.

Communication during incidents

Internally, we run a dedicated incident channel. Every action, observation, and decision gets logged there in real time. This creates a timeline that the retrospective can reference later.

For traveler-facing communication, we have decision thresholds:

If the booking path is broken for more than 5 minutes, we post to the status page.

If travelers have been affected (charged without confirmation, bookings in limbo), we prepare a direct communication plan. Usually this means a batch of notification messages once the issue is resolved: "Your booking for [flight] is confirmed. We apologize for the delayed confirmation."

We never send "we're having issues" messages during an incident. They cause more anxiety than they relieve. We wait until we can say "the issue is resolved, here's what happened, and here's what we're doing about it."

Blameless retrospectives

Within a week of resolving any P1 or P2 incident, we run a retrospective. The format is structured and blameless.

Timeline. We reconstruct exactly what happened, minute by minute. When was the first indicator? When was it detected? When was it mitigated? When was it resolved?

Root cause. Not "who" caused it, but "what" caused it. A code change, a configuration error, an external dependency failure, a capacity limitation. We dig until we find the systemic cause, not the proximate trigger.

Contributing factors. What made the incident worse or harder to resolve? Missing monitoring? Unclear runbooks? Gaps in test coverage?

Action items. Concrete steps to prevent recurrence or improve response time. Each action item has an owner and a deadline.

The blameless part is critical. If engineers fear blame, they hide mistakes. Hidden mistakes become repeat incidents. Our retrospectives focus on systems and processes, not individuals. "The deployment pipeline didn't catch this" is useful. "Alice deployed a bad change" is not.

Building a culture around incidents

Incidents are inevitable. A system that never has incidents either isn't doing anything interesting or isn't monitoring well enough to detect them.

What matters is how quickly we detect, how fast we mitigate, and how effectively we learn. Each incident is an investment in future reliability, but only if we extract the lessons.

Our incident runbook is a living document. After each retrospective, the runbook gets updated with new detection patterns, mitigation steps, and communication templates. Over time, the runbook becomes a codified version of the team's collective experience.

The minimum viable runbook

If you're building incident response for a booking platform:

Define severity levels based on traveler impact. The booking path is always the highest priority.

Establish on-call rotation. Someone needs to be reachable at 2 AM. This is non-negotiable for a platform that handles real bookings.

Bias toward rollback. If a recent deploy is suspect, roll back first and investigate after. The cost of an unnecessary rollback is small. The cost of extended downtime while investigating is large.

Log everything during the incident. The real-time log becomes the retrospective timeline. Without it, your retrospective relies on memory, which is unreliable under stress.

Run retrospectives within a week, not a month. The details fade quickly. The emotional energy to implement changes fades even faster. Do the retrospective while the incident is still fresh.

Make retrospectives blameless in practice, not just in policy. If leadership punishes the engineer who made a mistake, the next engineer will cover up theirs. The systemic fixes that prevent recurrence only happen when people are honest about what went wrong.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.