--- title: "Monitoring AI in Production: Beyond Uptime and Latency" description: Traditional monitoring checks if servers respond. AI monitoring must also check if the agent makes good decisions. Here is what we watch and why. canonical: https://nowah.xyz/blog/monitoring-ai-production-agent-health lastModified: "2026-08-07T03:48:40.491Z" --- # Monitoring AI in Production: Beyond Uptime and Latency Traditional monitoring checks if servers respond. AI monitoring must also check if the agent makes good decisions. Here is what we watch and why. Our first [production incident](/blog/worst-production-incident) with the AI agent was invisible to our traditional monitoring. The servers were healthy. Latency was normal. Error rates were zero. Response codes were all 200s. And the agent was confidently recommending flights that did not exist. A stale cache issue was serving outdated search results. The agent received data, processed it normally, and presented options that looked real. Users tried to book and the booking failed because the flights were no longer available at those prices. From a systems perspective, everything was green. From a user perspective, the product was broken. That incident taught us that monitoring an AI product requires a fundamentally different approach than monitoring a traditional web application. ## AI-specific metrics ![Illustration for this section](https://pics.nowah.xyz/website-media/engineering-039-img-1.webp) Traditional dashboards track uptime, latency, throughput, error rates, and resource utilization. These are necessary but not sufficient for an AI system. We added an entire category of AI-specific metrics. **Response quality score.** We sample agent responses and score them on a scale that evaluates relevance (did the response address what the user asked?), accuracy (are the facts correct?), and helpfulness (did it move the user toward their goal?). This runs as an automated evaluation that checks sampled conversations against quality criteria. If the quality score drops below a threshold, we get alerted before users start complaining. **Tool call success rate.** Our AI agent has access to dozens of tools: flight search, hotel search, booking execution, [currency conversion](/blog/launching-currency-conversion-global-travelers), and more. We track the success rate of each tool. If the flight search tool starts failing at a higher rate than normal, it could indicate a provider API issue, a data format change, or a prompt regression. **Hallucination rate.** We monitor for responses where the agent generates plausible-sounding information that is not grounded in actual data. For a travel agent, this might be inventing a flight route that does not exist or quoting a hotel price without actually searching. We detect this by comparing agent claims against the data returned by tool calls. **Streaming latency.** Time from user message to the first streamed token. This is the most visible performance metric because it is what the user experiences as "responsiveness." We track P50, P95, and P99 for this metric. P95 under 1 second is our target. **Conversation completion rate.** What percentage of conversations result in a successful outcome, whether that is a completed booking, a answered question, or a trip plan? Conversations that end abruptly or with the user expressing frustration indicate agent failures that error rates alone would not capture. ## Traditional metrics still matter We did not throw out traditional monitoring. We layered AI metrics on top of it. **Latency** across the full request lifecycle: time in queue, AI inference time, tool execution time, and response formatting time. Each segment is tracked independently so we can pinpoint where slowdowns originate. **Throughput** measured in active conversations, messages per minute, and bookings per hour. These are our capacity planning indicators. **Error rates** by type: 4xx errors (user-side issues), 5xx errors (server-side issues), timeout errors, and upstream API errors. Each type has different implications and different response playbooks. **Resource utilization** for CPU, memory, database connections, and an in-memory data store memory. AI workloads have different resource profiles than traditional web workloads. A spike in memory usage during peak hours might indicate that conversation contexts are growing faster than expected. ## Alerting for AI systems ![Supporting diagram](https://pics.nowah.xyz/website-media/engineering-039-img-2.webp) The hardest part of AI monitoring is deciding when to alert. Traditional alerting is straightforward: if error rate exceeds 5%, page someone. If latency exceeds 3 seconds, send a warning. AI quality degradation is more subtle. The quality score might drop from 0.92 to 0.85 over a few hours. Is that a real problem or normal variation? What if it is 0.85 at 2 AM when fewer people use the product and the sample size is small? We use a combination of threshold alerts and trend alerts. Threshold alerts fire immediately for clear failures: quality score below 0.7, tool success rate below 80%, streaming latency P95 above 3 seconds. These indicate something is clearly wrong. Trend alerts fire when a metric moves in the wrong direction for a sustained period. If quality score decreases for three consecutive hours, even if it is still above the absolute threshold, we want to know. These catch slow degradations that threshold alerts miss. We also have anomaly detection that flags statistical outliers. If Tuesday's booking success rate is 2 standard deviations below the previous four Tuesdays, something might have changed even if the absolute number looks acceptable. ## The daily dashboard Every morning, our engineering team reviews a dashboard that summarizes the previous 24 hours. The key sections: **AI agent health.** Quality score, tool success rate, average conversation length, and booking conversion rate. These tell us if the agent is performing well as an AI system. **System health.** Latency, error rate, resource utilization, and queue depth. These tell us if the infrastructure is performing well as a computing system. **Cost metrics.** AI inference cost per conversation, travel API cost per search, and total cost per booking. These tell us if we are staying within budget. **Anomalies.** Any metric that deviated significantly from its historical pattern, flagged by our anomaly detection. This daily review catches problems that automated alerting misses. Sometimes a metric is technically within bounds but "feels wrong" to an engineer who looks at these numbers every day. We have caught issues this way that automated systems did not flag. ## Incident response for AI misbehavior When the AI misbehaves, the response playbook is different from a traditional outage. For a server outage, you restart services, check logs, find the bug, fix it, deploy, post-mortem. The timeline is hours. For AI misbehavior, the first step is understanding what is happening. Is the agent giving wrong answers? Wrong to whom? About what topics? Is it a systematic problem or a specific conversation? Is it caused by our code, our prompts, our data, or the AI model itself? We built tools for rapid investigation of AI incidents. We can pull up conversations where quality scores dropped, see exactly what the agent said and what tools it called, compare the tool results with what the agent presented to the user, and identify patterns. Sometimes the fix is a prompt adjustment. Sometimes it is a data issue (stale cache, missing reference data). Sometimes it is a model-side regression that we need to escalate to our AI provider. Each cause has a different remediation path, and our monitoring system helps us triage quickly. The contrast with traditional OTA search quality monitoring is worth noting. OTAs monitor search result relevance through click-through rates and booking conversion. These are lagging indicators; by the time the metrics move, thousands of users have already had a bad experience. Our real-time quality scoring catches problems within minutes, often before any user reports an issue. --- Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. [Plan your next trip](https://app.nowah.xyz).