A/B Testing AI Behaviors: Why It Is Harder Than You Think

We ran an A/B test where both variants won on different metrics and we had no tiebreaker. Variant A had a higher booking completion rate. Variant B had higher post-booking satisfaction. Variant A was faster to first result. Variant B found cheaper flights on average. The data told us both options were better, depending on what you optimized for.

This is the fundamental challenge of A/B testing AI agent behaviors. The outcome space is multi-dimensional, the inputs are non-deterministic, and the thing you are measuring is often qualitative. Here is what we have learned about running experiments that actually produce useful answers.

Why non-determinism breaks sample size calculations

Traditional A/B testing assumes that the treatment is consistent. Every user in Variant A sees the same button color, the same layout, the same copy. The variance in outcomes comes from user behavior, not from the treatment itself.

AI agent behavior introduces variance on both sides. The same traveler sending the same query can receive different responses because the language model is non-deterministic. Flight A might rank higher than Flight B in one response and the reverse in the next, even with identical inputs. This means the variance in your experiment includes both user behavior variance and treatment variance, and the treatment variance can be substantial.

The practical consequence is that you need three to five times the sample size of a traditional A/B test to reach statistical significance. An experiment that would converge in two weeks with a button color change might take six to ten weeks with an AI behavior change. Most teams do not have the patience or the traffic for that, so they call results early and make decisions on noisy data.

We address this by running experiments longer than feels comfortable and by defining primary metrics in advance rather than shopping across metrics after the fact. If we say booking completion rate is the primary metric before the experiment starts, we commit to that metric even if other metrics look more favorable for the variant we prefer.

Controlling for prompt variation

The same traveler intent can be expressed in wildly different ways. "Find me cheap flights to Paris" and "I want to go to Paris, what's the most affordable option?" are semantically identical but syntactically different. The agent may handle them differently, producing different results for what should be the same experiment condition.

This is the prompt variation problem. In a traditional A/B test, the input is controlled: the user clicks a button. In an AI experiment, the input is natural language, and natural language is inherently variable.

We partially control for this by clustering queries by intent rather than by text. Our ranking experiments, for example, group all "search for flights from A to B on date C" queries together regardless of how the traveler phrased the request. This reduces the noise from prompt variation but does not eliminate it, because the agent's parsing of the intent introduces its own variance.

Measuring qualitative outcomes

The hardest metrics to measure are the qualitative ones. "Did the traveler feel confident in the agent's recommendation?" is arguably the most important outcome for a travel booking agent, and it is nearly impossible to A/B test directly.

We use proxies. Time spent reviewing the booking confirmation before clicking "confirm" is one proxy: shorter review times suggest higher confidence. The rate at which travelers modify the agent's top recommendation before booking is another: fewer modifications suggest better preference matching.

But proxies are imperfect. A traveler who reviews the confirmation quickly might be confident, or they might be in a rush. A traveler who does not modify the recommendation might be satisfied, or they might not know how.

For experiments where qualitative outcomes matter, we supplement quantitative data with direct feedback. A small percentage of travelers in each variant receive a one-question survey after their booking. This gives us a qualitative signal that, combined with the quantitative proxies, produces a more complete picture.

Ethical considerations

There is an ethical dimension to A/B testing AI behaviors that traditional experiments do not face. If Variant A provides better travel recommendations, is it fair to show Variant B to travelers who might book a worse trip as a result?

For low-stakes experiments like copy changes or display order, this is not a concern. But for experiments that affect the quality of recommendations or the reliability of bookings, we limit the scope and duration. We do not run booking quality experiments at 50/50 splits for months. We run them at small percentages for the minimum duration needed to detect a meaningful difference, and we define a stopping rule that halts the experiment if one variant is clearly harming traveler outcomes.

We also ensure that the control group always receives a good experience. The question an experiment answers should be "is the new approach better?" not "does the new approach work at all?" Both variants must be production-quality.

Practical guidelines

After running dozens of AI behavior experiments, here are the guidelines we follow.

Define the primary metric before the experiment starts, and commit to it. Run experiments for at least two weeks to capture weekday and weekend variance. Use three to five times the sample size you would use for a traditional A/B test. Segment by traveler type: business travelers, leisure travelers, and group planners respond differently to the same changes. Supplement quantitative metrics with qualitative feedback for experiments that affect recommendation quality. Set stopping rules that protect travelers from clearly inferior variants. And accept that some experiments will be inconclusive. An inconclusive result is more useful than a false positive from a noisy dataset.

The A/B test where both variants won on different metrics? We eventually broke the tie by asking a different question. Instead of "which variant is better?" we asked "which variant's strengths align with our product direction?" That is a judgment call, not a data call, and sometimes that is the best you can do.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.