How We Test an AI Agent Before Letting It Book Real Flights

A passing test suite means almost nothing for an AI agent that books flights. Traditional software tests verify deterministic behavior: given input X, the system produces output Y. An AI agent that searches live flight inventory, reasons about traveler preferences, and executes real bookings operates in a space where the same input can produce different outputs on every run. The airline changed a price. A route sold out. The language model weighted a preference differently. The world moved between test runs.

This is the testing challenge we face every time we prepare to ship a new capability at Nowah. Here is how we handle it.

Anatomy of an evaluation dataset

Our evaluation datasets are organized into four categories, each testing a different failure surface.

Common scenarios cover the booking patterns that represent 80 percent of real traveler requests. One-way domestic flights. Round-trip international itineraries. Weekend getaways with flexible dates. Business trips with tight schedule requirements. These are the bread-and-butter interactions that must work flawlessly because most travelers will never encounter anything else.

Edge cases are the requests that fall outside the typical pattern. Infant travelers requiring lap seats. Wheelchair assistance requests that need to propagate through the booking. Pet transport with carrier size requirements. Multi-city itineraries with tight connections. Travelers with multiple passports who need visa information for specific nationalities. Each edge case was added to our dataset because a real traveler encountered it and the agent handled it poorly.

Adversarial inputs test the agent's robustness against intentionally confusing or malicious requests. Prompt injection attempts. Contradictory requirements in a single message. Requests designed to make the agent ignore safety guidelines. Queries that try to extract system information. We run these not because we expect most travelers to be adversarial, but because the small percentage who are can cause outsized damage.

Safety checks cover scenarios where the agent's response has consequences beyond a bad search result. A traveler mentioning a medical emergency. Requests to book travel to regions with active travel advisories. Age-appropriate content handling for family travel. These evaluations have the strictest pass rate requirement: 99 percent or above, with zero tolerance for critical failures.

Tool-calling accuracy

The agent has a large set of tools it can invoke. When a traveler says "find me direct flights from San Francisco to Tokyo next Thursday morning, business class," the agent needs to select the correct search tool, construct parameters with the right origin airport code, destination code, date, cabin class, and stop count, execute the search against live inventory, and return results that match the request.

We measure tool-calling accuracy across four dimensions:

Tool selection. Did the agent pick the right tool? When asked about flights, it should call the flight search tool, not the hotel search tool. This sounds obvious, but complex multi-part queries can confuse the reasoning. "I need a flight to Paris and a hotel near the Eiffel Tower" requires two different tools called in sequence.

Parameter accuracy. Did the agent pass the correct parameters? The difference between "SFO" and "SJC" is forty miles and an entirely different trip. Date parsing is another minefield. "Next Thursday" depends on when the conversation happens and what timezone the traveler is in.

Execution order. When the agent chains multiple tools, did it call them in a sensible order? Searching for hotels before confirming the flight destination wastes API calls and confuses the traveler. The agent should establish the destination first, then search for accommodations.

Error recovery. When a tool call fails, how does the agent respond? APIs time out. Rate limits are hit. Inventory disappears between search and booking. The agent needs to handle each failure mode gracefully: retry with backoff for timeouts, queue and wait for rate limits, inform the traveler and suggest alternatives for disappeared inventory.

Multi-step reasoning evaluation

The hardest evaluations involve conversations where the agent must chain four or more tool calls to complete a task. A traveler says: "Plan a trip to Tokyo. I need flights from New York, a hotel near Shibuya for five nights, and I want to know if I need a visa."

The agent must: search flights from JFK/EWR/LGA to NRT/HND, search hotels near Shibuya with availability matching the flight dates, check visa requirements for the traveler's nationality, and present all three results in a coherent response with pricing.

Each step depends on the previous one. The hotel dates depend on the flight itinerary. The total trip cost depends on both. If any step fails, the agent needs to explain what worked, what did not, and what options the traveler has.

We evaluate these multi-step chains by tracking whether the agent reaches the correct end state through a reasonable path, not whether it takes the exact same path every time. There are multiple valid approaches to the Tokyo trip query. The evaluation checks that the end result is complete and accurate, with some flexibility in the specific sequence of tool calls.

Payment flow testing

Payment is where mistakes cost real money. We run a dedicated test suite against our payment infrastructure in test mode for every release. The suite covers:

Successful payment for a standard booking. Payment requiring 3D Secure verification. Payment declined by the card issuer. Duplicate payment attempts. Payment for a booking where the price changed between search and checkout. Refund processing for cancelled bookings. Multi-currency conversions.

Each test verifies not just that the payment was processed correctly, but that the multi-layer idempotency checks prevented duplicates. The booking attempt layer records the intent. The payment processor enforces exactly-once processing. The travel data provider confirms the booking. All three layers must agree for a booking to succeed.

We run these tests against real payment test modes, not mocks. Mocks cannot reproduce the timing-dependent edge cases that cause real production issues, like a payment that succeeds on the processor side but times out on our side, leaving an ambiguous state that the idempotency system must resolve.

Safety evaluation

Safety evaluations run separately from quality evaluations because they have different thresholds and different consequences for failure.

When a traveler mentions a medical emergency, the agent must recognize it and provide appropriate guidance, including suggesting they call emergency services. It should not attempt to book a flight to a hospital.

When a traveler asks about a destination with an active travel advisory, the agent must surface the advisory clearly before proceeding with any search. It should not present the advisory as a minor footnote below the flight results.

When an adversarial input attempts prompt injection, the agent must stay within its defined behavior. It should not reveal model instructions, ignore safety guidelines, or execute actions outside its authorized scope.

We test these scenarios with variations. The medical emergency is not always phrased as "I'm having a medical emergency." Sometimes it is "my chest hurts and I need to get to a doctor." Sometimes it is embedded in a longer message about trip changes. The agent needs to recognize the signal regardless of how it is phrased.

Safety evaluations must pass at 99 percent or above. If a new feature degrades safety performance even by a fraction of a percent, it does not ship until the degradation is understood and resolved.

The decision matrix

After running all evaluations, we have a matrix of pass rates across four dimensions (accuracy, latency, cost, safety) and four evaluation categories (common, edge, adversarial, safety). Translating this matrix into a ship or no-ship decision requires judgment, but we have codified the non-negotiable gates.

Safety below 99 percent: no ship, full stop. Quality below 95 percent on common scenarios: no ship. Quality below 85 percent on edge cases: ship with monitoring and a plan to improve, but only if common scenarios pass. Latency exceeding the budget by more than 50 percent: no ship until optimized. Cost exceeding the budget by more than 100 percent: no ship until the cost driver is understood.

If all gates pass, we move to private testing with a minimum of 50 real travelers over at least two weeks. Beta exposes the failures that no evaluation dataset catches, because real travelers interact with the agent in ways that even the most creative test designers cannot anticipate.

Only after beta confirms that the feature works in the wild do we begin the staged rollout to general availability. One percent first. Then ten. Then fifty. Then everyone.

It is a slower process than we would like. But an AI agent that books real flights with real money for real travelers does not get second chances with trust. We would rather ship slowly and correctly than quickly and apologetically.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.