Private Testing with Real Travelers: What Synthetic Data Misses

One of our private testers broke the agent with a request that no evaluation dataset contained. She asked: "I need to fly from Boston to Barcelona, but my husband is coming from Chicago and meeting me there. He is allergic to shellfish so we need a hotel that will not kill him. Also we are celebrating our anniversary so something romantic but not cheesy."

Our synthetic test data had round-trip flights, multi-city itineraries-itineraries-complex-planning), hotel preferences, dietary restrictions, and occasion-based recommendations each as separate test cases. Nobody had combined all five into a single conversational turn. The agent handled the flights fine, found hotels, but completely ignored the shellfish allergy and the anniversary context, treating them as unrelated background noise.

That is what synthetic data misses. Real travelers do not organize their requests into neat test categories. They talk like people, and people layer requirements, emotions, and context into single messages that no test designer would construct.

Why synthetic flight data and real inventory behave nothing alike

Our evaluation datasets include carefully constructed flight search scenarios. Origin, destination, date, cabin class, stop preference. Each scenario is designed to test a specific aspect of agent behavior. And each scenario operates against a static understanding of what the inventory looks like.

Real flight inventory changes by the minute. A fare that exists at 9:00 AM may be sold out by 9:15. Seasonal pricing creates patterns that static test data cannot capture. Codeshare flights appear under multiple airline codes, confusing the agent when it tries to present distinct options. Airlines add and remove routes without notice, and the agent needs to handle "that route does not exist" gracefully when a traveler asks for a connection that was available last week.

Our private testers surfaced all of these issues because they were searching for real trips on real dates. One tester searched for a flight to Bali during peak season and discovered that the agent was returning prices 30 percent lower than what was actually available because the data had a caching lag. Another tester found that the agent confidently presented a direct route that had been discontinued a month earlier. A third discovered that codeshare flights were appearing as duplicate options with different prices for what was the same seat on the same plane.

None of these would appear in synthetic testing because synthetic tests assume the inventory data is accurate at test time. Real inventory data is a moving target.

The diversity problem

Private testing revealed that real travelers ask questions in ways no prompt engineer anticipates. Our evaluation datasets were written by engineers who understood the agent's capabilities and unconsciously framed their test queries in a way the agent could parse. Real travelers do not do this.

A traveler asked: "What's the flight situation like for Easter?" The agent did not know when Easter was in the current year. Another asked: "Can you get me something like what I booked last time but cheaper?" The agent's memory had the previous booking, but it struggled to interpret "something like" as a fuzzy constraint rather than an exact match. A third asked: "My boss just told me I need to be in London Monday. Help." The agent treated this as a leisure travel query instead of recognizing the urgency and business context.

These are not edge cases. They are how normal people talk about travel. The gap between how engineers test an agent and how travelers use it is enormous, and the only way to close that gap is to put the agent in front of real travelers with real trips.

Setting up a structured beta program

Unstructured private testing produces unstructured feedback. We learned this early when our first beta cohort generated hundreds of messages about the experience, but most of them were general impressions rather than actionable reports. "It was pretty good" tells us nothing. "The agent suggested a hotel in the wrong neighborhood and I did not realize until I looked at the map" tells us everything.

Our beta program has a deliberate structure. We recruit travelers across five categories: business travelers who book frequently with tight constraints, leisure travelers who are flexible and exploratory, group planners coordinating multiple people, international travelers dealing with visas and currency, and travelers with accessibility needs. Each category exercises different agent capabilities and different failure surfaces.

We set a minimum of 50 testers over at least two weeks. This is not arbitrary. Two weeks captures the variation between weekday and weekend booking patterns, gives travelers time to complete real booking workflows not just searches, and provides enough data points to distinguish systematic issues from one-off anomalies.

Feedback collection combines structured surveys after each session, in-app feedback buttons that capture context alongside the report, analysis of chat transcripts to identify agent failure patterns that travelers did not explicitly report, and direct conversations with testers who encountered notable issues.

Bug triage

Not every beta issue is the same kind of problem. We triage beta feedback into three categories that each require different responses.

Agent failures are cases where the agent produced an incorrect result, called the wrong tool, or hallucinated information. These go directly to the engineering team for investigation. The fix might be a prompt adjustment, a tool parameter correction, or a new evaluation test case to prevent regression.

API failures are cases where the agent behaved correctly but the underlying data was wrong or unavailable. Flight prices that did not match, hotel availability that was stale, routes that no longer existed. These require investigation of our data pipeline and potentially changes to how we handle data freshness and fallback behavior.

UX confusion is the most insidious category. The agent worked correctly, the data was accurate, but the traveler did not understand what happened or how to proceed. Maybe the agent presented too many options without enough guidance. Maybe the booking confirmation did not make a key detail prominent enough. Maybe the agent used terminology the traveler did not recognize. UX confusion does not trigger an alert because nothing technically failed, but it erodes trust just as effectively as an error.

The three categories of bugs only real travelers surface

After running multiple beta cycles, we have identified three meta-categories of issues that synthetic testing consistently misses.

Timing bugs. Real travelers book anywhere from two weeks to eight months before travel. The agent's behavior needs to be appropriate across that entire window. Searching for a flight two months out returns different inventory, different pricing patterns, and different flexibility options than searching for next-week travel. Private testers naturally span this booking window in a way that synthetic tests, which typically use a fixed future date, do not.

Context bugs. Travelers carry context that they do not explicitly state. When someone says "same airline as last time," they expect the agent to know. When they mention "my usual seat," they expect the memory system to have retained that preference. Context bugs only appear when real travelers interact with the agent over multiple sessions, building up a relationship that synthetic single-turn tests cannot replicate.

Expectation bugs. Travelers have expectations formed by years of using other travel tools. When the agent behaves differently from what they expect, even if the agent's behavior is technically correct, the result is confusion and frustration. One beta tester expected the agent to automatically show the cheapest option first because that is what their usual booking site does. Our agent ranks by a preference-weighted algorithm that might surface a slightly more expensive flight that better matches the traveler's patterns. Technically better. Experientially confusing without explanation.

How beta feedback rewired our agent

Beta feedback does not just fix bugs. It changes priorities. After our first major beta cycle, we made three significant changes to the agent's behavior based on what we observed.

First, we added explicit date confirmation before any booking. Private testers frequently asked for flights using relative dates like "next weekend" or "the Thursday after Easter," and the agent's date interpretation was correct about 93 percent of the time. That sounds good until you realize that 7 percent means approximately one in fourteen bookings involved a date the traveler did not intend. Adding a confirmation step added friction but eliminated an entire category of expensive errors.

Second, we changed how the agent presents options. Originally, it showed all matching results in a single response. Private testers consistently reported feeling overwhelmed by more than five options in a conversation. We changed the default to showing the top three with an explicit offer to see more, and booking completion rates improved measurably.

Third, we adjusted the agent's memory system to distinguish between inferred preferences and stated preferences. The agent was inferring preferences from behavior, such as noticing a traveler always booked morning flights and then prioritizing morning departures. But sometimes the traveler was booking morning flights because of scheduling constraints, not preference. When the constraint changed, the agent's recommendation felt wrong. Now the agent asks to confirm inferred preferences rather than assuming they are permanent.

Each of these changes came directly from watching real travelers use the product. No evaluation dataset would have surfaced them. No amount of internal testing would have revealed them. Only real travelers with real trips and real money at stake showed us what the agent actually needed to be.

That is why private testing is not optional. It is the stage where the agent stops being a demo and starts being a product.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.