Monitoring AI Agent Quality in Production

The AI agent returned a 200 OK. The server streaming stream delivered tokens at 40 per second. Latency was well within threshold. All system health metrics looked great. And the response recommended a flight that doesn't exist.

Traditional application performance monitoring tells you the system is healthy. It doesn't tell you the system is helpful. For a standard API, that distinction doesn't matter much. If the endpoint returns the right HTTP status code and the data matches the schema, it's working. For an AI agent, an endpoint can be perfectly healthy from a system perspective while producing useless or dangerous output.

We had to build a separate monitoring layer that tracks agent quality alongside system health. This is the layer that catches hallucinated flights, unhelpful responses, and degraded tool call patterns before travelers notice.

Quality metrics that matter

We track several metrics that traditional APM doesn't cover.

Booking completion rate measures how often a conversation that includes a flight search leads to a completed booking. This is our primary conversion metric. A drop in booking completion rate might mean the agent is presenting options poorly, failing to guide the traveler through the booking flow, or generating responses that confuse rather than help.

Conversation length is measured in message count. Trip planning conversations typically span 5 to 50 messages depending on complexity. Unusually long conversations (80+ messages for a simple one-way flight) suggest the agent is struggling. Unusually short conversations that end without a booking might mean the agent failed to engage.

Tool call success rate measures what percentage of tool calls return useful results. A tool call that times out, returns an error, or returns empty results is a failure. Our target is high, and a drop below threshold triggers investigation.

Explicit feedback from travelers who rate their experience. Not everyone rates, but the ratings we get are directionally useful. A cluster of low ratings on a specific query type points us to a quality problem.

Tool call monitoring

The agent has 70+ tools. Each tool has its own performance profile. The flight search tool hits an external API and takes 1-3 seconds. The airport info tool hits our cache and returns in milliseconds. The booking tool involves payment processing and takes several seconds.

We monitor every tool along three dimensions.

Frequency tells us how often the agent chooses each tool. A sudden change in frequency patterns indicates a behavioral shift. If the agent stops calling the hotel search tool, maybe a prompt change accidentally de-prioritized hotels. If it starts calling a rarely-used tool frequently, maybe it's misinterpreting queries.

Latency by tool tells us whether external dependencies are degrading. If flight search latency doubles, we investigate the upstream API. If it's just one tool, the problem is specific. If all tools are slower, the problem is systemic.

Failure rate by tool catches tools that are broken. A tool returning errors 20% of the time might be intermittent enough that individual travelers don't notice, but across all conversations, 20% failure rate on a key tool degrades the overall experience measurably.

We visualize this as a tool call heatmap: tools on one axis, metrics on the other, color-coded by health. A glance at the heatmap shows us which tools are working well and which need attention.

Hallucination detection

This is the hardest monitoring problem in AI. The agent generates natural language responses about flights, hotels, and destinations. Sometimes it generates information that sounds authoritative but is wrong. A flight number that doesn't exist. A hotel that closed last year. A visa requirement that changed.

We can't fully detect hallucinations in real time, but we use several heuristics.

Tool result correlation. When the agent mentions a specific flight (airline, time, price), we check whether that data came from a tool call result. If the agent references flight details that don't match any tool call in the current session, it's likely hallucinating. This catches the most dangerous hallucinations: fabricated flight information that a traveler might try to book.

Factual consistency checks. For destination information (visa requirements, safety advisories, weather), we compare the agent's statements against our knowledge base. Significant discrepancies get flagged for review.

Traveler reports. Some hallucinations only get caught when a traveler says "That flight doesn't exist." We track these reports and use them to improve our detection heuristics.

Full hallucination detection would require a separate model evaluating every response, which is expensive. We focus our detection on the high-stakes information (flight details, prices, booking confirmations) and accept that low-stakes conversational content might occasionally be imprecise.

Evaluation datasets

Beyond production monitoring, we run evaluation suites before every deployment. These are curated datasets of queries with expected behaviors.

Flight evaluations test the agent's ability to search flights, compare options, and guide booking decisions. Each test case has a query and expected tool call patterns.

Safety evaluations test resistance to prompt injection, information leakage, and unauthorized actions.

Verbose evaluations test the quality of the agent's explanations and recommendations.

These evaluations run as part of our deployment pipeline (`npm run eval`). A regression in any evaluation metric blocks the deploy. This catches quality degradation before it reaches production.

Connecting system health to business outcomes

The dashboard that matters most to leadership isn't the one showing CPU utilization and request latency. It's the one connecting system metrics to business metrics.

We correlate: when AI response latency increases, does booking completion rate decrease? When tool call failure rates rise, does conversation length increase (because travelers have to repeat themselves)? When we ship a new agent feature, does traveler satisfaction improve?

These correlations help us prioritize infrastructure investments. If reducing flight search latency by 500 milliseconds increases booking completion rate by 2%, that's a clear ROI calculation for infrastructure investment. Without the quality metrics connected to business outcomes, infrastructure improvements are abstract. With them, they're concrete.

Building an AI quality monitoring stack

If you're running an AI agent in production, here's the minimum monitoring to add beyond traditional APM.

Track tool call frequency, latency, and success rate per tool. This is your agent's behavioral fingerprint. Changes in this fingerprint correspond to quality changes.

Implement at least basic hallucination detection for high-stakes outputs. If your agent recommends products, verify the products exist. If it quotes prices, verify the prices are real.

Run evaluation datasets before every deploy. Automated quality gates catch regressions that system health metrics miss.

Connect agent quality metrics to business metrics. If you can't show that agent quality improvements lead to better business outcomes, it's hard to justify the investment. If you can, it's easy.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.