Testing AI Agents in Production

Traditional software testing works because software is deterministic. Same input, same output. Write an assertion, verify the output, move on. Test coverage gives you confidence.

AI agents break this model completely. Same input, different output every time. There are an infinite number of valid responses to "find me flights to Tokyo." The response quality depends on real-time inventory that changes by the minute. The conversation context means the same message produces different results depending on what was said previously.

After eighteen months of shipping an AI travel agent to production, I have developed strong opinions about what works for AI testing and what does not. The short version: you need a fundamentally different testing philosophy. Here it is.

The testing paradox

Traditional testing asserts on exact outputs. `expect(result).toBe("Tokyo Narita")`. This does not work for AI agents because:

• The agent might say "Tokyo Narita (NRT)" or "Narita International Airport" or "NRT - Tokyo" and all are correct.
• The flight options change every search, so the specific flights in the response cannot be hardcoded.
• The explanation text varies with each generation even when the underlying recommendation is the same.
• The conversation context means the response depends on the full history, not just the last message.

If you try to write traditional unit tests for an AI agent, you will spend more time updating the tests than improving the agent. The tests are brittle, the maintenance cost is high, and the coverage is illusory because passing a specific assertion does not mean the output is good.

Eval datasets for travel

Instead of asserting on exact outputs, we test properties of outputs. This requires eval datasets: curated sets of inputs with expected properties.

Our eval dataset has several hundred scenarios organized by category:

Common routes. SFO-NRT, JFK-LHR, LAX-CDG. The agent should return real flights with correct airports, reasonable prices, and proper date handling.

Edge cases. Dates that cross the international date line. Airports with similar codes. Routes with no direct service. Destinations requiring visas. The agent should handle these correctly or flag them clearly.

Multi-[city itineraries](/blog/launching-[multi-city](/blog/multi-city-flight-booking-ai-agents)-itineraries-complex-planning). Three-city Europe trips. Round-the-world routes. Trips with gaps between legs. The agent should produce coherent itineraries with valid connections.

Adversarial inputs. Requests for impossible trips. Attempts to confuse the agent with contradictory constraints. Inputs containing injection attempts. The agent should respond gracefully without breaking.

Preference-dependent scenarios. The same route for a budget traveler vs. a luxury traveler. The same destination for a couple vs. a family with kids. The agent should produce different recommendations based on the user profile.

Each scenario specifies properties to check, not exact outputs. For a flight search: did the agent return flights for the correct route? Are the dates correct? Are the prices within a reasonable range? Did it present three options? Did it provide a recommendation with explanation? These properties are verifiable regardless of which specific flights appear.

Property-based testing for tool correctness

The most reliable automated tests for an AI agent are property-based tests on tool calls. When the agent decides to search for flights, the tool call has specific parameters. These can be verified exactly:

• The origin airport code matches the user's departure city
• The destination airport code matches the requested destination
• The dates fall within the requested range
• The number of passengers is correct
• The cabin class matches the user's preference or default

Tool call correctness is the foundation of agent quality. If the agent calls the right tools with the right parameters, the results will be correct (assuming the tools themselves work). If the agent calls the wrong tool or passes wrong parameters, no amount of response quality compensates.

We run property-based tests on every agent change. A regression in tool call accuracy is caught before it reaches production.

Shadow mode

Shadow mode is the most valuable production testing technique we have implemented.

When we are evaluating a new model, a new prompt configuration, or a major agent change, we run the new version alongside the current production version. Both receive the same user messages. Both generate responses. Only the production version's response is shown to the user. The shadow version's response is logged for comparison. A comparison engine analyzes both responses for each conversation. It checks: Did the shadow version recommend different flights? Were the shadow recommendations better or worse by our quality metrics? Did the shadow version make any tool call errors that production did not? Did the response length or style change significantly?

Shadow mode lets us evaluate model changes on real production traffic with zero risk to users. If the shadow version performs better across a statistically significant sample, we promote it to production. If it performs worse, we discard it. If it is mixed, we investigate the specific scenarios where it differed.

The sample size needs to be large. Due to the non-deterministic nature of AI outputs, we typically need 1,000-5,000 shadow comparisons before we have confidence in a version change. This takes a few days of production traffic, which is why shadow mode runs continuously.

Regression monitoring

Shadow mode catches regressions before deployment. Regression monitoring catches regressions after deployment.

We continuously monitor key quality metrics in production:

• Task completion rate by query type
• Tool call success rate
• Recommendation acceptance rate
• Conversation abandonment rate
• Response latency percentiles

Each metric has a baseline range established from historical data. When a metric drops below its baseline for more than a few hours, an alert fires. The alert includes a sample of recent conversations showing the quality drop so engineers can diagnose the cause quickly. Common causes of production regressions:

• Model provider updated their model (they do this without warning sometimes)
• A data provider changed their API response format
• A prompt change had an unintended side effect
• Traffic patterns shifted (more complex queries than usual)

Regression detection must catch drops within hours to minimize user impact. A quality regression that runs for a day at our scale affects thousands of conversations. Early detection is not a nice-to-have; it is a requirement.

The human review loop

Automated testing catches quantifiable issues. Human review catches qualitative issues.

Every week, a team member reviews 50-100 randomly selected conversations. They assess qualities that automated metrics miss: Was the agent's tone appropriate? Did the recommendation reasoning make sense? Did the agent handle an ambiguous request gracefully? Would I, as a traveler, be satisfied with this interaction?

Human review is expensive and does not scale. But it catches issues that no automated system can: subtle tone shifts, slightly off recommendations, and creative user requests that the agent handled poorly. These insights feed back into prompt improvements, new eval scenarios, and architectural changes.

Testing an AI agent is not a phase. It is a permanent production system. Eval datasets grow over time. Shadow mode runs continuously. Regression monitoring never stops. Human reviews happen weekly. The agent is never "tested and done." It is always being tested, always being evaluated, always being improved.

This is the only way to maintain quality in a non-deterministic system. Accept that you cannot test every output. Test the properties that matter. Monitor continuously. Review regularly. And iterate faster than the quality can degrade.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.