Measuring AI Quality at Scale

We launched our AI travel agent with a comprehensive eval suite. Hundreds of test scenarios covering everything from simple one-way flight searches to complex group trips with split itineraries. We felt confident. Then real users showed up and did things our evals never imagined.

One person asked for "flights that avoid going over water" because of a phobia. Another asked to find hotels "near where Anthony Bourdain ate in that Tokyo episode." A third asked to rebook their entire honeymoon after their wedding got postponed by two weeks, preserving everything else. None of these appeared in our eval dataset. All of them are perfectly reasonable requests from real travelers.

This experience taught us something I now consider foundational: offline evals catch roughly 60% of quality issues. The other 40% only surface in production. You need both.

The monitoring gap

Pre-launch evaluation is necessary. I am not arguing against it. If you ship an AI agent without testing it against known scenarios, you are being reckless. But the nature of AI agents makes pre-launch testing fundamentally incomplete.

The output space is enormous. A traditional software product has deterministic outputs for given inputs. You can test every code path. An AI agent produces different responses to the same input depending on conversation history, memory state, time of day, available inventory, and the inherent randomness of language model sampling. Covering even a representative fraction of this space pre-launch is impractical.

User behavior is unpredictable. Real users combine features in ways your team never considered. They ask ambiguous questions. They change their mind mid-sentence. They paste URLs into the chat. They test boundaries. The creativity of real users exceeds the imagination of any QA team.

The environment changes. Flight prices change. Hotel availability shifts. Airlines update their schedules. A volcano erupts in Iceland. These external changes affect agent quality in ways that no static eval dataset captures.

Task completion rate: the north star

If I had to pick one metric for AI agent quality, it would be task completion rate. Did the user accomplish what they came to do?

This sounds simple, but defining "completion" for an AI travel agent is subtle. A user asks for flights to Paris. The agent returns three options. The user says "thanks" and leaves. Did they complete their task? Maybe. They might have gotten the information they needed and will book later. Or they might have given up because none of the options were good.

We define task completion hierarchically:

• Search completion: the agent successfully returned relevant results (measurable from tool call success)
• Engagement completion: the user interacted with the results, asked follow-ups, or refined their search (measurable from conversation length and message content)
• Booking completion: the user actually booked something (directly measurable)
• Satisfaction completion: the user rated the interaction positively or returned for their next trip (measurable from feedback and retention)

Each level tells you something different. Search completion without engagement completion suggests the results were not relevant. Engagement without booking suggests a friction point in the booking flow. Booking without satisfaction suggests the agent made a sale but not a happy customer.

User satisfaction signals

Explicit feedback, like asking users to rate their experience, is valuable but noisy. People who are extremely happy or extremely unhappy rate. Everyone in the middle skips the rating prompt. This creates a bimodal distribution that overstates both success and failure.

Implicit behavioral signals are more reliable. We track:

• Recommendation acceptance rate: when the agent suggests three flight options, how often does the user accept one versus asking for different options? This directly measures whether the AI's ranking and personalization are working.
• Conversation efficiency: how many turns does it take to reach a booking? Shorter is generally better, but not always. A user who spends fifteen turns planning a three-week Asia trip is having a different experience than one who needs fifteen turns for a round-trip to LA.
• Return usage: does the user come back for their next trip? This is the ultimate satisfaction metric, but it has a long feedback loop since people book travel infrequently.
• Abandon rate by stage: where in the conversation do users stop engaging? A high abandon rate after flight results suggests the search quality or presentation needs work. A high abandon rate at the payment step suggests a UX problem in the booking flow.

Regression detection

The scariest quality failure is the one that happens slowly. A model update slightly degrades the agent's ability to handle multi-city trips. The change is subtle enough that no individual user complains, but task completion rate for complex itineraries drops from 78% to 71% over a week.

We built automated regression detection that monitors our key metrics on rolling windows. When task completion rate, recommendation acceptance rate, or any other tracked metric drops below a threshold for more than a few hours, an alert fires. The alert includes a comparison of recent conversations against baseline examples so an engineer can quickly diagnose whether the drop is real and what caused it. The time sensitivity matters. A quality regression that runs for a day affects thousands of conversations. One that runs for a week affects tens of thousands. We aim to detect and respond to regressions within hours. This requires real-time metric computation, not daily batch jobs.

A/B testing AI changes

A/B testing AI agents is harder than A/B testing traditional product changes. The core reason: output variance.

When you A/B test a button color change, the outcome difference between groups is entirely attributable to the button color. When you A/B test a prompt change in an AI agent, the outcome difference between groups is a combination of the prompt change and the natural variance of model output. Two users asking the same question might get different quality responses regardless of which experiment group they are in.

This means you need larger sample sizes to achieve statistical significance. In our experience, roughly 3-5x the sample size of a deterministic A/B test for the same confidence level.

We also learned to stratify by query complexity. A prompt change might improve complex queries while degrading simple ones, or vice versa. If your A/B test lumps all queries together, you might see a neutral result that hides significant positive and negative effects in different segments.

The other challenge is defining the success metric for an A/B test of an AI change. "Which version is better" is not always obvious. Version A might have higher task completion but lower user satisfaction ratings. Version B might produce longer conversations (is that good or bad?) with higher booking rates. You need to decide upfront what success looks like.

The feedback loop

Production metrics are not just for monitoring. They are the primary input to agent improvement.

Every week, we review a sample of conversations where quality metrics were low. What went wrong? Did the agent misunderstand the query? Did the tool return bad data? Did the ranking algorithm weight the wrong factors? Did the response format confuse the user?

These reviews generate concrete improvements: prompt adjustments, tool refinements, new eval scenarios, and occasionally architecture changes. The improvements ship, and the cycle repeats.

This feedback loop is, I think, the single most important operational practice for running AI agents in production. The teams that instrument their agents thoroughly and review production data regularly build better products faster than teams that optimize in isolation and hope their improvements survive contact with real users.

The best AI travel booking experience is the one that gets measurably better every week. That only happens if you are measuring the right things.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.