Evaluating AI Agent Quality: Beyond Unit Tests

You cannot assert that an AI response equals "the correct answer." There is no single correct answer to "Find me a nice hotel in Paris." There are dozens of reasonable responses, and the difference between a good response and a bad one is subjective, contextual, and hard to formalize.

This is the fundamental challenge of testing AI agents. Traditional unit tests check that function A given input B produces output C. AI agent tests need to check that the agent, given a travel query, produces a response that is helpful, accurate, safe, and uses the right tools in the right order. That's a different kind of testing entirely.

We built an evaluation framework that runs before every deployment. It's not perfect. No AI evaluation system is. But it catches regressions, prevents unsafe behavior from shipping, and gives us quantitative confidence in the agent's quality.

Evaluation datasets

The foundation of our evaluation system is curated datasets of queries with expected behaviors. Not expected exact outputs, but expected behavioral patterns.

A flight evaluation entry might look like: "Query: 'Find me a direct flight from JFK to CDG next Thursday.' Expected behavior: calls flight search tool with correct airports and date, presents results sorted by relevance, includes price and duration, does not hallucinate flight numbers."

The evaluation doesn't check the exact text of the response. It checks that the right tools were called with the right parameters, that the response references actual tool results (not hallucinated data), and that the response structure is appropriate for the query type.

We maintain several evaluation suites:

Flight evaluations (`npm run eval:flights`) test flight search, comparison, and booking guidance across simple and complex queries.

Safety evaluations (`npm run eval:safety`) test resistance to prompt injection, information leakage, and unauthorized actions.

Verbose evaluations (`npm run eval:verbose`) test the quality and detail of the agent's explanations and recommendations.

Each suite can run independently, and they all run together as part of the deployment pipeline (`npm run eval`).

Evaluation dimensions

We evaluate along five dimensions:

Accuracy. Did the agent use correct information? Did it reference actual search results or make things up? Did the dates, prices, and airport codes match the tool results?

Safety. Did the agent resist adversarial inputs? Did it refuse to reveal system information? Did it enforce authorization boundaries?

Helpfulness. Did the agent answer the actual question? Did it provide enough detail to be useful? Did it guide the traveler toward a decision?

Tool call correctness. Did the agent call the right tools? Were the parameters correct? Were unnecessary tools avoided?

Latency. Did the agent respond within acceptable time? Our test configuration uses 60-second timeouts to account for external API latency, with 2 retries for flaky API calls.

Each dimension gets a score. The aggregate score determines whether the deployment proceeds. A regression in any single dimension can block the deploy if it drops below the threshold.

Adversarial testing

Safety evaluations are our most important evaluation suite. They test the agent against a library of adversarial inputs designed by our team based on known prompt injection patterns and travel-specific attack vectors.

The test cases include attempts to: override the agent's instructions, access other users' data, bypass booking authorization, extract the model instructions, and convince the agent to perform actions outside its scope.

For each test case, we assert that the agent refuses the request appropriately, doesn't leak system information, and doesn't attempt unauthorized tool calls. A single safety evaluation failure blocks the deployment entirely. We have zero tolerance for safety regressions.

We add new adversarial test cases whenever we discover a new attack pattern, either through our own red-teaming or from the broader AI security community. The evaluation suite grows over time as our understanding of the threat landscape deepens.

Regression testing for AI behavior

AI agents are non-deterministic. The same query can produce different responses on different runs. This makes regression testing harder than for deterministic code.

We handle this by running evaluations multiple times and checking for consistency. If the agent passes an evaluation 9 out of 10 times, that's good. If it fails 3 out of 10 times, we have a reliability problem even if the average score looks fine.

We also compare evaluation scores across versions. Before deploying a new agent configuration (prompt changes, tool modifications, model updates), we run the full evaluation suite on both the current and proposed versions. If the proposed version scores lower on any dimension, we investigate before proceeding.

This comparison catches subtle regressions. A prompt change that improves helpfulness might accidentally weaken safety. A tool modification that speeds up searches might reduce accuracy for edge case queries. Only by measuring all dimensions do we see the full picture.

Continuous evaluation in production

Pre-deployment evaluation is necessary but not sufficient. Production traffic is more diverse than any evaluation dataset. Real travelers ask things we didn't anticipate. Real network conditions create scenarios our test environment doesn't replicate.

We supplement pre-deployment evaluation with continuous production monitoring. The quality metrics described in our monitoring article (booking completion rate, tool call success rate, conversation length, explicit feedback) serve as ongoing evaluation signals.

When production quality metrics drift, we investigate. Usually the cause is an external dependency change (the travel data provider changed their response format, a new LLM version was deployed upstream) rather than our own code change. But we catch it through the same evaluation framework.

Building your own evaluation framework

If you're deploying an AI agent, here's the minimum evaluation setup.

Create evaluation datasets for your core use cases. Start with 20-50 test cases that cover the most common queries and the most dangerous edge cases.

Define behavioral expectations, not exact outputs. "The agent should call the search tool with these parameters" is testable. "The agent should respond with this exact text" is not.

Include safety evaluations from day one. Add adversarial test cases before you need them, not after an incident.

Run evaluations as a deployment gate. If you can only add one CI/CD gate, make it the safety evaluation suite. Quality regressions are fixable. Safety regressions are incidents.

Expand the evaluation suite continuously. Every bug report, every support ticket, every discovered edge case becomes a new test case. The dataset should grow with your understanding of how the agent is used.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.