--- title: "The First 48 Hours: Monitoring Plan for an AI Travel Agent Launch" description: "Monitoring plan for the first 48 hours after ship — dashboards, on-call, rollback triggers, and what counts as a launch-day success." canonical: https://nowah.xyz/blog/post-launch-monitoring-first-48-hours lastModified: "2026-08-07T08:24:30.707Z" --- # The First 48 Hours: Monitoring Plan for an AI Travel Agent Launch Monitoring plan for the first 48 hours after ship — dashboards, on-call, rollback triggers, and what counts as a launch-day success. The first two days after cutover are an ops shift, not a marketing afterparty. This is the dashboard, on-call, and rollback plan we will run — written now so nobody invents process under pager noise. ## Three monitoring layers ![Illustration for this section](https://pics.nowah.xyz/website-media/launch-006-img-1.webp) **Infrastructure monitoring** is the foundation. Latency, error rates, database connections, background job queue depth, upstream provider availability. This layer tells you whether the system is running. It catches outages, performance degradation, and resource exhaustion. Every software team needs this, and it is the least interesting layer for AI products because the most dangerous failures return 200 OK. **Semantic monitoring** is the AI-specific layer. This is where you catch the agent saying things that are wrong, irrelevant, or potentially harmful. We run automated checks on agent responses looking for hallucinated airport codes, impossible routes, prices that diverge significantly from recent search results, and dates that do not match traveler requests. These checks run in near-real-time against a sample of production conversations. Semantic monitoring is harder to build than infrastructure monitoring because you need to define what "correct" means for each type of agent response. A flight search result is correct if the origin, destination, date, and cabin class match the query. A hotel recommendation is correct if the location, dates, and price range match. An itinerary suggestion is correct if the activities are real, the timing is feasible, and the locations are in the right city. **Business monitoring** tracks outcomes. Booking completion rates, revenue per session, support ticket volume, traveler satisfaction signals. This layer tells you whether the product is actually working for travelers, even if the servers are up and the agent is technically correct. A launch that is technically flawless but tanks booking completion is still a failed launch. ## Alert thresholds We classify alerts into four severity levels, each with a different response expectation. **P0: Service down.** The agent is not responding, payments are failing, or the app is unreachable. Response: page the on-call engineer immediately. Target resolution: under 15 minutes. **P1: Error rate above 5 percent.** Something is systematically failing for a meaningful fraction of travelers. Response: investigate immediately, consider halting the rollout. Target resolution: under one hour. **P2: Quality degradation.** Semantic monitoring detects a pattern of incorrect or suboptimal agent responses. Response: investigate within four hours, escalate if the pattern is growing. Target resolution: within 24 hours. **P3: Metric anomaly.** A business metric is trending differently than expected but not yet at a level that indicates a clear problem. Response: review in the next standup. Target resolution: within 48 hours. The first 48 hours focus on P0 and P1 with heightened sensitivity. We lower our alert thresholds temporarily, meaning we investigate anomalies that we would normally classify as P3 at full severity, because launch-day patterns are unfamiliar and what looks like noise might be a signal. ## The war room ![Supporting diagram](https://pics.nowah.xyz/website-media/launch-006-img-2.webp) For major launches, we set up a physical and virtual war room. The room has three screens: one for each monitoring layer. The people in the room include the engineering lead responsible for the feature, the product manager who defined success criteria, the support lead who handles traveler-facing issues, and the on-call SRE who manages infrastructure response. Each person has a specific focus. Engineering watches semantic quality and investigates anomalies. Product tracks business metrics and makes rollout progression decisions. Support monitors incoming tickets and surfaces patterns that dashboards miss. SRE ensures infrastructure health and manages [incident response](/blog/incident-response-travel-platform). The war room operates on a rotation schedule. No one sits in the room for 48 straight hours. We run 8-hour shifts with handoff documents that capture the current state, active investigations, and decisions pending. Handoffs happen at the start of each shift with a 15-minute overlap. ## The 48-hour timeline **Hours 0 through 4:** Maximum vigilance. Every metric is watched. Every anomaly is investigated. The rollout is at its initial percentage and the decision to promote has not been made. The team is looking for immediate failures that evaluations and [private testing](/blog/beta-testing-real-travelers-synthetic-data-misses) missed. **Hours 4 through 12:** Pattern recognition. Enough data has accumulated to distinguish real issues from noise. The team reviews a sample of conversations manually, checking for semantic accuracy and traveler satisfaction signals. The first promotion gate decision may happen at the end of this window if everything looks clean. **Hours 12 through 24:** The overnight test. Traveler behavior changes at night. Different time zones come online. The agent needs to perform consistently across the full daily cycle. The overnight shift watches for degradation that only appears during low-traffic periods when caching is cold and database connections are fewer. **Hours 24 through 36:** Sustained stability check. If the first 24 hours were clean, the team transitions from active watching to monitoring. Dashboards are still live but the posture shifts from "looking for problems" to "confirming health." The second promotion gate decision typically happens here. **Hours 36 through 48:** Handoff to operations. If no P0 or P1 incidents have occurred and metrics are stable, the war room demobilizes. Monitoring transitions from launch mode to standard operations mode with regular alert thresholds. A post-launch report is drafted capturing what was observed, what was investigated, and what improvements to make for the next launch. ## Building a monitoring runbook Every launch uses a runbook customized from a template. The runbook specifies which dashboards to watch, what the expected ranges are for each metric, what constitutes an anomaly versus normal variation, who to contact for each type of issue, and what the rollback procedure is if something goes wrong. The runbook is written before the launch, reviewed by the team, and available to anyone in the war room. It is not a generic document. It is specific to the feature being launched, referencing the exact metrics that matter for that capability and the exact failure modes that evaluations identified as risks. We update the runbook template after every launch with what we learned. The Istanbul airport hallucination added "verify geographic data accuracy" as a semantic monitoring check. A payment timing issue added "monitor payment completion latency P95" to the infrastructure layer. Each incident improves the runbook for the next launch. After 48 hours, the feature is either stable in production and operating under normal monitoring, or it has been rolled back with a post-mortem scheduled. Either outcome is acceptable. What is not acceptable is uncertainty: a feature that is live but nobody is confident it is working correctly. The 48-hour window eliminates that uncertainty, one way or the other. --- Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. [Plan your next trip](https://app.nowah.xyz).