Launch Day Failure Modes: Incidents We Design Against Before Ship

Launch day is when weak assumptions meet real travelers. We have not had a public launch day yet. What follows is the incident catalog we design against: agent mistakes, payment edge cases, notification storms, and how we will communicate when something does break.

Incident one: the hallucination

The agent told a traveler that a specific flight existed: a direct route between two cities, departing at a specific time, at a specific price. The traveler was excited. The flight was exactly what they wanted. There was one problem. The flight did not exist.

The root cause was a failure in the agent's tool-calling layer-layer-ai-agent-search-flights). The agent had searched for flights and received results. But when it composed the response, it merged details from two different flights into a single recommendation. The departure time came from one flight. The price came from another. The route was an interpolation that matched neither. The result looked perfectly plausible, which is what made it dangerous.

Detection happened when the traveler attempted to proceed with booking and the system could not find the flight the agent had described. The booking layer queried the actual inventory and returned nothing matching the agent's description. The traveler reported the discrepancy through the in-app feedback button.

The fix had two parts. First, we added a verification step between the agent's response and the traveler-facing message. Before presenting any flight option, the system cross-references the agent's description against the actual search results. If there is a mismatch, the response is blocked and the agent re-generates from the verified data. Second, we added evaluation cases specifically testing for merged-result hallucinations, with over fifty new test cases covering various combinations of similar flights.

The deeper lesson was that an agent with seventy-plus tools can produce results that look correct to a human reader but are fabricated from fragments of real data. Plausible hallucinations are more dangerous than obvious ones because they pass the casual inspection test. Our verification layer now catches these before they reach the traveler.

Incident two: the double charge

A traveler completed a booking and was charged twice. Not approximately twice. Exactly twice. The same amount, the same card, within three seconds of each other.

We had built a multi-layer idempotency checks specifically to prevent this. The first layer generates a unique booking attempt identifier before any payment is initiated. The second layer uses the payment processor's own idempotency key to prevent duplicate charges. The third layer checks the booking confirmation from the travel data provider before marking the booking as complete.

The edge case was a network timeout. The first payment request succeeded at the processor level but the response timed out before reaching our server. Our system, believing the payment had failed, retried with a new request. The idempotency key should have prevented the duplicate, but the retry logic generated a new key because it treated the timeout as a fresh attempt rather than a retry of the same attempt.

The fix was straightforward: the idempotency key is now generated at the booking attempt level, not the payment request level. Any retry for the same booking attempt uses the same idempotency key, regardless of whether the retry is triggered by a timeout, an error, or a user re-click. The three layers now overlap completely, with no gap for a network timeout to exploit.

The traveler received an immediate full refund for the duplicate charge, a personal apology from our team, and a credit toward their next booking. The incident took ninety minutes to fully resolve from detection to refund confirmation.

Incident three: the notification storm

A single traveler received over two hundred push notifications in ten minutes. Their phone was buzzing continuously. They thought the app had been hacked.

The cause was a status update loop. A flight the traveler had booked experienced a series of minor status changes: gate assignment, then gate change, then gate reversion. Each status change triggered a notification. But the status was also being polled by our background job system, and a timing issue caused the same status changes to be processed multiple times. Each processing generated a new notification. The loop amplified three genuine status changes into over two hundred notification deliveries.

The notification service had no per-traveler rate limit. It had rate limits on the API layer to prevent abuse, but the notification dispatch was an internal system call that bypassed API rate limiting. The rate limits we had built protected against external abuse but not against internal amplification.

The fix was a per-traveler notification rate limit: no more than a configurable number of notifications per traveler per hour, with intelligent deduplication that collapses identical or near-identical notifications into a single delivery. We also added circuit-breaking to the status polling system so that rapid successive changes are batched rather than individually processed.

The traveler was contacted, apologized to, and offered a direct communication channel to our team. They were remarkably understanding once we explained what happened. Honest explanation of a technical failure is almost always met with more patience than silence or evasion.

Why we publish post-mortems

Publishing post-mortems serves three purposes. First, it holds us accountable. A post-mortem that is published externally is written more carefully and more honestly than one that stays internal. The knowledge that others will read it drives thoroughness.

Second, it builds trust with travelers. Every traveler knows that software has bugs. When a company openly explains what went wrong and what they fixed, the traveler gains confidence that the company takes reliability seriously. Silence after an incident, on the other hand, suggests that the company either does not know what happened or does not care.

Third, it helps other teams building AI products. The hallucination incident, the idempotency gap, and the notification storm are not unique to our platform. Any team building an AI agent with tool-calling capabilities could encounter similar issues. Sharing our learnings reduces the industry's collective failure rate.

Systemic fixes

Each incident produced a specific technical fix. But the more important outcome was the systemic changes to our launch process.

After the hallucination incident, we added a mandatory verification layer between agent output and traveler-facing responses for any content that includes specific factual claims: prices, times, routes, availability. The agent can be creative in its conversation. It cannot be creative with facts.

After the double charge incident, we reviewed every payment path for idempotency coverage. We found two additional edge cases where the layers did not fully overlap and fixed them preemptively. We also added integration tests that simulate network timeouts during payment processing and verify that retries use the same idempotency key.

After the notification storm, we implemented rate limiting on all internal notification dispatch, not just external API endpoints. Any system that sends notifications to travelers is now subject to per-traveler rate limits, regardless of whether the trigger is external or internal.

Post-mortem culture

Every P0 or P1 incident gets a written post-mortem within seventy-two hours. The template is consistent: incident summary, timeline, root cause, resolution, prevention measures, and lessons learned. The post-mortem is reviewed by the engineering lead and the product manager before publication.

The most important rule of our post-mortem culture is that post-mortems are blameless. The question is never "who made the mistake?" It is always "what about our system allowed this mistake to happen?" Systems fail. People do not fail; systems fail to support people. When we find a human error in the incident timeline, we ask what guardrail was missing that would have caught the error before it reached production.

This culture means that engineers report incidents immediately and honestly. There is no incentive to hide a problem or minimize its severity because there is no blame attached to discovering it. The incentive is to report fast, fix fast, and prevent recurrence.

Things will break again. We will publish those post-mortems too. The goal is not perfection. It is continuous improvement driven by radical honesty about what went wrong.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.