How to Benchmark an AI Travel Agent

The AI industry is obsessed with benchmarks. MMLU. HumanEval. MATH. HellaSwag. These benchmarks measure important things: knowledge recall, code generation, mathematical reasoning, common sense inference.

None of them tell you whether an AI agent can book a flight.

This is a real problem. When we evaluate models for Nowah, the standard benchmarks are nearly useless as predictors of production performance. A model that scores 90% on MMLU might hallucinate flight prices. A model that aces HumanEval might fumble multi-step tool chains. The capabilities that matter for a travel booking agent, task completion across multi-turn conversations with real-time data and financial transactions, are not measured by any public benchmark.

We had to build our own evaluation framework. I want to share what we measure, why we measure it, and how our metrics differ from standard AI benchmarks.

Why standard benchmarks miss the point

MMLU measures whether a model knows facts. For travel booking, factual knowledge from training data is actively harmful. An LLM that "knows" the price of a flight from SFO to NRT is reciting stale data. The actual price changes by the minute. We do not want the model to recall facts about travel. We want it to query live APIs for real-time data.

HumanEval measures code generation ability. Our agent does not write code. It calls pre-defined tools with structured parameters. The skill being tested is different: not "can you write a Python function?" but "can you select the right tool and populate its parameters correctly?"

MATH measures mathematical reasoning. Travel has math (price comparison, date arithmetic, currency conversion), but it is elementary arithmetic within a complex constraint satisfaction problem. The hard part is not the math. It is managing the constraints.

The common thread is that standard benchmarks measure model capabilities in isolation. A travel agent operates at the intersection of multiple capabilities: language understanding, tool use, planning, memory, and real-time data integration. The system-level performance matters more than any individual capability.

Task completion rate: the north star

Our most important metric is task completion rate: what percentage of users who express a travel booking intent end up with a confirmed reservation?

This is the metric that captures everything. If the agent misunderstands the query, task completion fails. If it picks the wrong tool, fails. If it loses context mid-conversation, fails. If it presents options the user rejects, fails (indirectly, through user abandonment). If the booking API errors and recovery fails, fails.

We segment task completion by complexity:

• Simple search and book (one-way flight, single traveler): target above 90%
• Standard round-trip with hotel: target above 85%
• Multi-city itinerary: target above 75%
• Complex group booking: target above 65%

These targets might seem low compared to, say, 95% function calling accuracy. But task completion is an end-to-end metric that compounds many individual accuracy rates. If the agent needs 5 tool calls and each has 95% accuracy, the chain completion rate is 0.95^5 = 77%. Add conversation understanding, preference matching, and user interaction quality, and 75-85% task completion for moderate complexity is ambitious.

Preference alignment

Task completion tells you whether the agent booked a flight. Preference alignment tells you whether it booked the right flight.

We measure this several ways:

Confirmation accept rate tracks how often users book the agent's top recommendation versus asking for alternatives. Higher accept rates mean the agent's preferences are well-calibrated. If users consistently ask for alternatives, the agent is misjudging their preferences.

Modification rate tracks how often users modify the agent's initial search parameters. If the agent suggests direct flights and the user says "actually, connections are fine if it's cheaper," the initial search parameters did not match the user's priorities.

Post-booking satisfaction is measured through follow-up signals. Does the user search for the same route after booking (indicating regret)? Do they rebook within 24 hours (indicating error)? Do they return for their next trip (indicating overall satisfaction)?

Return usage rate, specifically rebooking within 90 days, is our strongest signal of agent quality. Users who come back trusted the agent enough to use it again. Users who do not were either dissatisfied or did not travel again in that window.

Efficiency metrics

Two bookings are not equal if one took 4 conversational turns and the other took 25. Efficiency matters both for user experience and for cost.

Turns to completion measures the number of conversational turns from initial intent to confirmed booking. We track this by complexity class and look for trends. Rising turns-to-completion usually means the agent is asking too many clarifying questions or failing to retain context.

Tool calls per task measures the total number of API calls the agent makes. More calls mean more latency and more cost. We target 3 to 7 tool calls per complex query, with the exact number depending on the task type.

Time to booking measures wall-clock time from first message to confirmation. This includes user think time, so it is not purely an agent performance metric. But dramatic increases suggest the agent is creating friction.

Token consumption measures the total tokens used per conversation. This is a direct cost metric. Longer conversations with more tool calls consume more tokens. We optimize for shorter, more efficient conversations that still maintain quality.

Building eval datasets for travel

Our evaluation datasets consist of realistic travel scenarios with expected outcomes. Each scenario has:

• Input: A user message or multi-turn conversation
• Expected behavior: Which tools should be called, with what parameters
• Expected output: What the agent should present to the user
• Evaluation criteria: Task completion, factual accuracy, preference match, tone

We build scenarios at multiple difficulty levels:

Unit-level scenarios test individual capabilities. "Extract the departure date from this message." "Select the correct tool for a hotel search." These have clear right/wrong answers.

Integration scenarios test multi-step chains. "Search for flights, rank them, present the top 3, and handle the user selecting option B." These test tool chaining and response coherence.

Adversarial scenarios test edge cases. "The user asks for a flight to an airport that does not exist." "The user provides contradictory dates." "The user asks the agent to do something outside its capabilities." These test error handling and graceful degradation.

Regression scenarios test specific failures we have observed in production. Every time a real user encounters a problem, we create a test case for it. This library grows continuously and prevents us from reintroducing fixed issues.

Online vs offline evaluation

Eval datasets are necessary but not sufficient. They test the agent against scripted scenarios. Production is unscripted.

We run online evaluation continuously in production:

Shadow testing runs new model versions alongside the production model. Both process the same queries. We compare outputs without exposing the new version to users. This catches regressions before they affect anyone.

A/B testing exposes a percentage of users to model or prompt changes and measures the impact on task completion, satisfaction, and efficiency metrics.

Anomaly detection flags conversations where the agent behaves unexpectedly: unusually long conversations, tool call failures, user abandonment, or booking errors. These get reviewed and added to the regression dataset.

Feedback loops connect production metrics back to model improvement. If task completion drops after a model update, we can trace it to specific scenario types and update the eval dataset to cover the gap.

The feedback loop

The evaluation framework is not a one-time assessment. It is a continuous improvement cycle.

Production data reveals failure patterns. Failure patterns become eval scenarios. Eval scenarios drive model and prompt improvements. Improvements get deployed and measured against production metrics. The cycle repeats.

This is why online evaluation matters more than offline evals for a production AI system. The eval dataset is always trailing reality. Users find edge cases that we did not anticipate. Market conditions change. New routes get added. Airlines change their policies. The agent has to handle a constantly evolving environment, and the evaluation system has to evolve with it.

Function calling accuracy exceeds 95% for well-defined schemas. That is the starting point, not the goal. The goal is a system where a user can say "plan my trip" and end up with a booking they are happy with, and where our metrics tell us exactly how close we are to that goal and where the gaps remain.

Building eval datasets for travel

The eval dataset is the most important artifact in the evaluation framework. It determines what gets measured, which determines what gets improved.

We build eval datasets from three sources.

Production conversations. Real user interactions are the gold standard. We anonymize conversations, extract the intent and outcome, and turn them into eval scenarios. A user who successfully booked a multi-city trip becomes a test case: given this intent, does the agent produce a valid itinerary? Production data captures the real distribution of user requests, including edge cases we would never think to write.

Synthetic scenarios. We generate test cases that cover specific capabilities or failure modes. "Book a round-trip flight for a family of 4 with an infant, one passenger needs wheelchair assistance, and they have a lap dog." This tests multiple tool interactions, special service requests, and passenger data collection. Synthetic scenarios are essential for coverage because production data alone is sparse for unusual cases.

Adversarial inputs. We test the agent's ability to handle inputs designed to cause failures. Contradictory constraints ("I want a direct flight under $200 from SFO to Tokyo"). Ambiguous requests ("somewhere warm, maybe December, not sure about dates"). Inputs that test hallucination resistance ("is there a direct flight from Des Moines to the Maldives?"). These adversarial cases push the agent's boundaries and expose weaknesses that normal usage might not reveal.

Our current dataset has grown to over 2,000 scenarios across all three categories. It grows weekly as production monitoring identifies new failure patterns.

Each scenario in the dataset includes:

Input. The user message or conversation history.

Expected behavior. What the agent should do. This is not always a single correct answer. For a flight search, "calls the flight search tool with correct parameters" is the expected behavior, not a specific set of results.

Evaluation criteria. How to score the agent's output. Binary pass/fail for tool selection. Numeric scores for response quality. Boolean checks for hallucination (did the agent claim facts not in the tool results?).

Difficulty rating. Simple, moderate, complex, or adversarial. This lets us track performance by difficulty tier and identify where improvements are needed most.

Offline vs online evaluation

Offline evaluation runs the agent against the eval dataset in a controlled environment. It is fast, reproducible, and covers a wide range of scenarios. We run it before every model or prompt change.

But offline evaluation has blind spots. The eval dataset is always a simplified approximation of real usage. Real users do not follow scripts. They change their minds. They provide incomplete information. They ask follow-up questions that the eval dataset does not anticipate.

Online evaluation fills these gaps by measuring the agent in production.

The cost dimension

Evaluation is not just about quality. It is about quality per dollar.

An agent that completes 95% of tasks but costs $2 per interaction is different from one that completes 90% of tasks at $0.20 per interaction. The right tradeoff depends on the task value. For a $500 flight booking with a 12% take rate, the $2 interaction cost is easily justified. For a $50 ground transport booking, it might not be.

We track cost efficiency metrics alongside quality metrics:

Cost per completed task. Total inference cost divided by the number of successfully completed bookings. This is the economic north star.

Token efficiency. Total tokens consumed divided by task completion rate. Lower is better. Verbose agents that use 15,000 tokens to accomplish what a lean agent does in 5,000 are wasting money.

Retry cost. When errors occur and the agent retries, the additional token cost of recovery. Frequent retries indicate tool schema problems or model behavior issues.

We optimize for the intersection of quality and cost. Sometimes a cheaper model with 88% task completion is better than a frontier model with 93% task completion at 5x the cost. It depends on the use case.

Evaluation as a competitive advantage

I will make an argument that might seem counterintuitive: the evaluation framework is more defensible than the model.

Models improve quickly and are increasingly commoditized. A frontier model from one provider is roughly equivalent to a frontier model from another for most travel tasks. If your product's quality depends entirely on the model, you are vulnerable to every competitor who uses the same model.

But the evaluation framework is proprietary. Our eval dataset contains thousands of travel scenarios derived from real production conversations. Our metrics are tuned to our specific product. Our feedback loop between production monitoring and eval improvement is a process that takes years to develop.

When we switch underlying models (which we do periodically as new models become available), the evaluation framework immediately tells us how the new model performs against our specific benchmarks. We can make model decisions in hours, not weeks, because the eval framework provides definitive answers.

Companies without robust eval frameworks are flying blind. They switch models based on vibes, public benchmarks, and hope. That is not a sustainable approach for a product that handles real money.

The human baseline

One question we return to regularly: how does the agent compare to a human travel agent?

We periodically run experiments where human travel agents and our AI agent receive the same booking requests. We compare task completion rate, time to completion, preference match, and user satisfaction.

The results are nuanced. For simple bookings (round-trip flight, single destination), the AI agent is faster and comparably accurate. For complex bookings (multi-city, group, unusual requirements), human agents still win on quality, though the gap narrows with each model generation.

For availability and consistency, the AI agent wins decisively. It is available 24/7. It does not have bad days. It does not forget to check a preference. It processes requests in minutes, not hours.

The trajectory is clear: the AI agent improves measurably with each evaluation cycle. Human agents do not improve at the same rate. The crossover point, where the AI agent matches or exceeds human agents across all complexity levels, is not reached yet but is visible on the roadmap.

The human comparison also reveals an important insight about what users value. Human agents excel at empathy and creative problem-solving. AI agents excel at consistency, speed, and memory. The ideal evaluation framework measures both sets of strengths rather than evaluating the AI only on dimensions where humans are strong. A user who gets a consistent, fast, preference-aware booking at 2 AM has received value that no human agent can provide, regardless of how the AI scores on creative trip suggestions.

Regression testing as institutional memory

Every bug we fix becomes a test case. Every user complaint that reveals a failure becomes an eval scenario. Every edge case we discover in production gets encoded in the dataset.

This accumulation is powerful. After a year of production operation, the eval dataset contains hundreds of scenarios that represent real-world failures. It is institutional memory of everything that has gone wrong, codified as automated tests.

When we change the model instructions, update tool schemas, or switch models, the regression suite tells us whether we have reintroduced any previously fixed failure. This is invaluable for preventing the "whack-a-mole" pattern where fixing one thing breaks another.

The regression dataset has another benefit: it grounds architectural decisions in data. When someone proposes changing the ranking algorithm, we run the new algorithm against all historical scenarios and compare outcomes. The debate moves from "I think this is better" to "here is how it performs against 500 real scenarios."

We invest significant engineering effort in keeping the eval infrastructure fast and reliable. The full suite runs in under 10 minutes. If it took an hour, developers would skip it. Speed matters for adoption.

The evolving benchmark landscape

Standard LLM benchmarks (knowledge tests, coding challenges, reasoning puzzles) are useful for comparing base models but tell you almost nothing about whether a model will work well as a travel agent. We learned this the hard way by choosing a model that scored well on public benchmarks but performed poorly on tool selection for our specific use case.

The travel agent benchmark space is still immature. There is no MMLU equivalent for "can this model book a multi-city trip while respecting a budget and accounting for visa requirements." We had to build our own, and we think every serious AI product company will need to do the same.

The investment in custom evaluation is not glamorous. It does not make for good demos or conference talks. But it is the single most important technical investment a production AI team can make. Without it, you are improving your product based on intuition rather than evidence. With it, every change is measured, every regression is caught, and every improvement is quantified. The teams that invest early in evaluation infrastructure will iterate faster and ship better products than teams that rely on manual testing and user feedback alone. We learned this lesson the hard way and would not build another AI product without establishing the eval framework before writing the first line of agent code.

The best AI travel booking experience is not the one that scores highest on any single benchmark. It is the one that consistently completes real travel tasks for real users. That is what we measure, that is what we optimize for, and that is the standard against which we hold ourselves accountable.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.