On-Call for AI: How We Handle AI Incidents Differently
When the AI gives a wrong answer, that is a different incident than when the server goes down. Here is our AI-specific incident response playbook.
At 11:43 PM on a Thursday, our monitoring system fires an alert. On a traditional platform, the alert says "5xx error rate above threshold" or "database connection pool exhausted" or "API latency P99 above 2 seconds." You pull up the dashboard, identify the failing component, fix it, and go back to sleep.
Our monitoring system fires a different kind of alert too: "AI agent recommending hotels in the wrong city for 3% of conversations in the last hour." The server is up. The database is healthy. The APIs are responding normally. The AI is just making bad decisions, and the standard incident response playbook does not cover this.
AI incidents are a different category of problem, and they require a different response framework. Here is ours.
AI-specific incident categories
We classify incidents into two broad categories: infrastructure incidents and AI judgment incidents.
Infrastructure incidents are familiar. A server goes down. A database connection pool exhausts. An external API times out. The response is the same as any other software system: identify the failing component, apply the fix (restart, scale up, fail over), verify recovery. Our runbooks for these are standard.
AI judgment incidents are new. The AI agent does something wrong not because the code crashed but because the model made a bad decision. Bad recommendations. Incorrect information. Inappropriate responses. Failure to understand a user's request. These incidents present fundamentally differently in monitoring (no errors in the logs) and require fundamentally different debugging (no stack trace to follow).
We break judgment incidents into subcategories:
[Recommendation quality](/blog/how-we-measure-recommendation-quality) degradation. The agent is recommending options that do not match user preferences as well as they should. This might be subtle and show up as a gradual decrease in booking completion rates or user satisfaction scores.
Factual errors. The agent states something incorrect. Wrong airport codes. Incorrect visa information. Stale pricing information. These are dangerous because users trust the agent's factual claims.
Behavioral regression. The agent starts behaving differently after a change. Maybe it is asking too many clarifying questions. Maybe it is not asking enough. Maybe it is calling tools in an inefficient order that increases response time.
Safety violations. The agent responds to a request in a way that violates our safety guidelines. Sharing personal data inappropriately. Providing advice outside its domain. Failing to refuse a harmful request.
Each subcategory has a different severity level and a different response procedure.
Rollback strategies for AI systems
You cannot rollback a bad AI answer. Once the agent has told a user that a flight costs $200 when it actually costs $800, that information is out there. But you can prevent it from happening again.
For prompt-level issues, rollback means reverting to the previous prompt version. We version our prompts with the same rigor as our code. Every prompt change is committed, reviewed, and deployable. Reverting a prompt is a deployment, not a code change.
For tool definition issues, rollback means reverting the tool configuration. If a tool parameter change caused the agent to send incorrect search queries, we revert the parameter definition and redeploy.
For model-level issues, the options are more limited. If the AI model provider has degraded quality (it happens), we cannot "rollback" their model. We can switch to a different model version if one is available, or we can tighten our guardrails to compensate for the degradation.
We maintain a "known good" configuration for every component of the AI system: prompt version, tool definitions, model version, and guardrail settings. At any point, we can revert the entire AI configuration to the last known good state. This is our "break glass" option for severe AI incidents.
Post-incident analysis for AI failures
Post-incident analysis for a traditional software failure asks: "What was the root cause?" The answer is usually a specific code bug, configuration error, or infrastructure failure. The fix is deterministic: change the code, fix the config, resize the instance.
Post-incident analysis for an AI failure often produces a different kind of answer: "The model made a bad judgment." That is not a root cause you can fix with a code change. The model was presented with a user request and context, and it produced an output that was wrong.
Our analysis framework for AI incidents has three steps.
Step 1: Conversation replay. We replay the conversation that triggered the incident, loading the exact context the agent had at the time. We examine the memory state, the tool call sequence, and the model's reasoning at each step.
Step 2: Pattern identification. Is this a one-off bad output or a systemic pattern? We search for similar conversations in recent history. If we find multiple instances of the same failure pattern, it is systemic and requires a structural fix. If it is a one-off, we add it to our eval suite and monitor for recurrence.
Step 3: Mitigation design. For systemic issues, we design mitigations. Better guardrails. More specific tool definitions. Additional eval test cases. Prompt adjustments that steer the model away from the failure mode. For one-off issues, we add the case to our eval suite so we will catch it if it recurs.
This analysis is slower and more ambiguous than traditional incident analysis. There is not always a clean "root cause." Sometimes the best we can say is "the model was on the edge of a decision boundary, and it fell on the wrong side in this case." The mitigation for that is to move the decision boundary through prompt and guardrail changes, then verify with evals.
Runbooks for non-deterministic systems
Traditional runbooks are step-by-step procedures: "If X happens, do Y." They work because traditional systems are deterministic. The same input produces the same output, so the same fix works every time.
AI incident runbooks need to be more flexible. "If the agent is making bad recommendations" is not specific enough to have a single fix. The runbook needs a diagnostic tree:
- Check if the issue correlates with a recent deployment. If yes, rollback.
- Check if the issue correlates with a model provider change. If yes, test with alternative model version.
- Check if the issue is specific to a conversation pattern. If yes, analyze those conversations for a common trigger.
- Check if eval scores have degraded. If yes, the issue may be in prompt or tool definitions.
- If none of the above, engage the full team for investigation.
Each branch in the diagnostic tree has its own sub-procedures. But unlike traditional runbooks, these procedures are diagnostic rather than prescriptive. They help the on-call engineer narrow down the problem, not execute a predetermined fix.
When to pull the AI out of the loop
We have a hard rule: if an AI judgment incident affects financial transactions, we pause AI-initiated bookings immediately and route to human review. The pause stays in place until we understand the root cause and verify the fix with our eval suite.
This has happened twice. Both times, the root cause was identified and fixed within hours. Both times, the pause prevented potential financial harm to users. The lost revenue from paused bookings was trivially small compared to the trust damage that would have resulted from completing bad bookings.
The trigger for pulling AI out of the loop is not a technical threshold. It is a judgment call by the on-call engineer. We give the on-call engineer explicit authority to pause AI-initiated actions without approval from anyone else. Speed matters in incident response, and a 15-minute delay to get approval could mean 15 minutes of bad bookings.
Real-time monitoring and alerting for all services is the foundation that makes incident response possible. But for AI systems, the monitoring needs to watch for different things than traditional systems. Not just error rates and latency. Conversation quality scores. Recommendation relevance metrics. Safety violation flags. Eval score trends.
The tools are different. The runbooks are different. The post-incident analysis is different. On-call for AI is still on-call, with the same late nights and the same adrenaline. But the incidents feel different because you are not debugging a machine. You are debugging a decision-maker.
Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.