Conversation Analytics: Measuring AI Agent Performance

Every conversation with our AI agent produces data. Not just the messages themselves, but metadata about how the conversation went: how many turns it took, which tools the agent used, whether the user got what they wanted, and where things went wrong if they did.

This data is our most valuable feedback loop. It tells us what the agent does well, where it struggles, and what to fix next. Here is how we analyze it.

Conversation length and resolution

The first thing people ask about our agent is "how many messages does it take to book a flight?" The answer is complicated because conversation length alone does not tell you much.

A 4-message conversation might be great: "Find flights to Rome in April." Agent searches, presents options. "Book the second one." Agent books it. Quick, efficient, successful.

A 4-message conversation might also be terrible: "Find flights to Rome in April." Agent misunderstands and searches for Athens. "No, ROME." Agent searches correctly this time. "Book the cheapest one." Done, but only after a frustrating misunderstanding.

We measure conversation length alongside resolution. A resolved conversation is one where the user's original intent was fulfilled: booking completed, question answered, trip information provided. An unresolved conversation is one where the user stops responding or explicitly expresses frustration without getting what they wanted.

Short resolved conversations are ideal. Long resolved conversations are acceptable but suggest optimization opportunities. Short unresolved conversations are concerning. Long unresolved conversations are failures we need to understand.

The agent success rate, the percentage of conversations with successful outcomes, is our headline metric. We track it daily and investigate any meaningful decline.

Failure mode identification

Not all conversation failures are the same, and fixing them requires understanding why they failed.

We categorize failures into several modes:

Misunderstanding failures. The agent did not correctly parse the user's intent. "I need to fly from Houston" gets interpreted as a question about Houston rather than a flight search request. These are addressed by improving the agent's intent recognition.

Tool failures. The agent correctly understood the request but the tool it called failed. The flight search returned an error, or the hotel search timed out. These are infrastructure issues, not AI issues.

Ranking failures. The agent searched correctly and found results, but the options it presented did not match what the user wanted. The user asked for "boutique hotels" and got chain hotels. These are addressed by improving the ranking algorithm.

Scope failures. The user asked for something outside the agent's capabilities. "Book me a rental car" when we do not support car rentals yet. These are product gaps.

Conversation management failures. The agent lost track of context in a long conversation. The user said "change it to a window seat" and the agent did not know which booking "it" referred to. These are context management issues.

Each failure mode has a different fix. Categorizing failures lets us prioritize engineering work based on which failure modes affect the most users.

User satisfaction signals

We do not have a "rate your experience" button in the chat. Asking users to rate every conversation would be annoying and would produce biased data (only frustrated or delighted users would bother).

Instead, we infer satisfaction from conversational signals.

Positive signals: User completes a booking. User says "thanks" or "perfect." User returns for another conversation within a week. User shares the trip details (suggests they are acting on the agent's help).

Negative signals: User stops responding mid-conversation. User says "never mind" or expresses frustration. User restates the same request multiple times (suggesting the agent is not understanding). User contacts support after a conversation.

Neutral signals: User asks clarifying questions (normal, not negative). Conversation ends after information is provided without booking (user might be in the research phase).

We weight these signals into a satisfaction score per conversation. The score is not as precise as a direct survey, but it covers every conversation rather than just the ones where users choose to give feedback. At scale, the aggregate satisfaction score is a reliable indicator of agent quality.

Using analytics to improve tools

Our per-tool analytics reveal which agent capabilities work well and which need improvement.

Flight search has a high success rate and high user satisfaction. Hotel search has a slightly lower satisfaction rate because hotel preferences are more subjective and harder to encode than flight preferences. The booking tool has a high success rate but occasional timeout issues during peak hours. The currency conversion tool has a near-perfect success rate because it is simple and deterministic.

By tracking success and satisfaction per tool, we can allocate engineering effort where it has the most impact. Right now, hotel ranking improvement is our highest-priority agent enhancement because the tool analytics show it is our weakest area.

We also track tool usage patterns. Which tools are called most frequently? Which tools are called but their results are not used (agent searched but did not present results, suggesting it realized the search was unnecessary)? Which tool calls take the longest? These patterns inform both performance optimization and product development.

Privacy-preserving analysis

Conversation data contains personal information: names, dates of travel, destinations, budget details. We cannot casually read individual conversations for analytics purposes.

Our analytics pipeline operates on aggregated, anonymized data. We measure conversation lengths, failure rates, tool usage, and satisfaction signals without accessing the content of individual messages. The pipeline processes conversations into metrics and discards the raw content.

When we need to investigate a specific failure mode, we use a privacy-preserving review process. Conversations are reviewed only when flagged by automated quality checks, and the review focuses on the agent's behavior (tool calls, response patterns) rather than the user's personal information. Reviewer access is logged and audited.

This matters not just for compliance but for user trust. Our users share travel plans, passport details, and financial information with the agent. They need to trust that this information is not being casually browsed by our engineering team.

The feedback loop

Conversation analytics feed directly into agent improvement. The cycle works like this:

Analytics identify a failure pattern (e.g., hotel ranking satisfaction drops when users describe style preferences like "modern" or "boutique"). Engineering investigates the specific conversations that triggered the pattern. The team identifies the root cause (the ranking algorithm does not weight style preferences heavily enough). A fix is developed and tested. The fix is deployed. Analytics confirm whether the fix improved the metric.

This cycle runs continuously. Every week, we review the top failure patterns and satisfaction trends. Every sprint includes at least one agent improvement driven by conversation analytics. The agent gets measurably better every month because the data tells us exactly where to focus.

Traditional OTA analytics cannot capture any of this because traditional OTAs do not have conversations to analyze. They see page views, clicks, and conversions. We see intent, understanding, satisfaction, and failure modes. That depth of insight is what lets us improve the product at a pace that page-based analytics cannot match.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.