--- title: "Staged Rollouts for AI Features: Why We Will Ship to a Small Cohort First" description: "Progressive rollouts for AI capabilities — feature flags, canary deployments, traffic splitting, and the gate criteria that decide when to widen the audience." canonical: https://nowah.xyz/blog/staged-rollouts-ai-features-ship-one-percent lastModified: "2026-08-07T08:24:30.975Z" --- # Staged Rollouts for AI Features: Why We Will Ship to a Small Cohort First Progressive rollouts for AI capabilities — feature flags, canary deployments, traffic splitting, and the gate criteria that decide when to widen the audience. We once shipped a booking feature to 100 percent of our users and regretted it within an hour. The feature worked perfectly in testing, passed all our evaluations, and survived a week of internal use. Then it met [real travelers](/blog/beta-testing-real-travelers-synthetic-data-misses) with real urgency and real edge cases, and the cracks appeared immediately. A traveler booked a flight for the wrong date because the agent misinterpreted "this Friday" across a timezone boundary. Another traveler hit a rate limit on a search-heavy conversation and got a terse error message instead of a helpful explanation. A third found that the new feature conflicted with an existing workflow in a way nobody anticipated. None of these were catastrophic individually. Together, they created a terrible first impression for every new user who encountered the feature on its first day. We could not un-ship it, and the damage to trust took weeks to repair. That experience taught us a lesson we have not forgotten: AI features ship to 1 percent first. Always. ## Feature flags for non-deterministic systems ![Illustration for this section](https://pics.nowah.xyz/website-media/launch-003-img-1.webp) [Feature flags](/blog/feature-flags-high-stakes) in traditional software are relatively simple. The flag is on or off. The feature works or it does not. You test it with the flag on, confirm it works, and roll it out. Feature flags for AI agent capabilities need richer gate logic. The same agent processing the same query can produce different results each time because of the non-deterministic nature of language models. This means a feature flag that gates an AI capability needs to account for more than just on/off state. Our feature flag configuration for AI capabilities includes the rollout percentage, but also monitoring thresholds that automatically halt the rollout if quality degrades. The flag knows its own success criteria: what error rate is acceptable, what quality score is required, and what latency budget applies. If any threshold is breached, the flag reverts without human intervention. This automated safety net has saved us multiple times. It caught a quality degradation at 3 percent error rate during a 10 percent rollout, halting the expansion before the issue affected more travelers. Without automated gates, we might not have noticed until the rollout reached 50 percent and the support queue started filling. ## The 1% canary The first stage of every rollout is the canary: 1 percent of traffic routes to the new capability while 99 percent continues with the existing experience. This is not just a smaller rollout. It is a fundamentally different monitoring posture. At 1 percent, we are not measuring success. We are looking for failure modes that evaluations and private testing missed. Every conversation that touches the new capability is reviewed, either manually or through automated quality checks. We are looking for: **Error patterns.** Are certain query types failing more than others? A feature that works for domestic flights but fails on international routes needs investigation before wider rollout. **Unexpected interactions.** Does the new capability interfere with existing features? When we added hotel booking, early canary traffic revealed that the agent sometimes confused hotel check-in dates with flight departure dates because both were "dates" in the conversation context. **Performance under load.** Even at 1 percent, we can extrapolate how the feature will perform at full traffic. If each invocation of the new capability generates five API calls to external providers, we can calculate whether full rollout will exceed our provider rate limits. **Traveler reception.** Do travelers engage with the new capability as expected? Sometimes a feature that tests well is confusing in practice because the agent does not explain it well, or the traveler does not realize it is available. The canary stage has a minimum dwell time of 24 hours. This is non-negotiable because travel behavior varies significantly between weekdays and weekends, and a feature that works perfectly on a Monday morning may [struggle with](/blog/legacy-otas-struggle-with-ai-architecture) the different query patterns of a Saturday evening. ## Promotion gates ![Supporting diagram](https://pics.nowah.xyz/website-media/launch-003-img-2.webp) Promotion from one stage to the next requires passing explicit criteria. These are not suggestions. They are gates that the team cannot override without a documented exception approved by engineering and product leadership. **1 percent to 10 percent.** Error rate below 5 percent. No P0 or P1 incidents during the canary period. Latency within budget. Manual review of at least 100 conversations touching the new capability shows no systematic issues. Monitoring dashboards confirmed to be capturing the right signals. **10 percent to 50 percent.** All 1-to-10 criteria sustained. Support ticket volume for the new capability at or below baseline. Booking completion rate for affected travelers is not lower than the control group. Cost per conversation within the budgeted range. **50 percent to 100 percent.** All previous criteria sustained over at least 48 hours at 50 percent. No degradation in overall platform metrics. The team has high confidence that the remaining 50 percent of traffic will not surface new failure modes. Rollback procedure rehearsed and ready. Each stage has its own 24-hour minimum dwell time. The total time from canary to full rollout is at minimum four days, and often longer if we encounter issues that require investigation without triggering a full halt. ## Traffic splitting for AI Traffic splitting for AI features is more nuanced than traditional A/B testing. The control group cannot just see "the old experience" if the old experience lacks a capability the new experience provides. We need to ensure that travelers in the control group still get a working agent, just without the new specific capability. This means the traffic split happens at the capability level, not the user experience level. A traveler in the control group who asks about hotels when hotel booking is the new feature gets a graceful response explaining that hotel booking is not yet available for their account, with a timeline for when it will be. They do not get a broken experience or silence. We also ensure that our background job processing respects the split. If the new capability triggers [background jobs](/blog/background-jobs-ai-travel-booking), those jobs only run for travelers in the treatment group. Orphaned background jobs from a rolled-back feature caused us problems once when a status polling worker continued running for travelers who had been moved back to the control group. ## Rollback at each stage Rollback procedures differ by stage and by the nature of the feature being rolled out. For pure agent capability additions like a new tool, rollback means removing the tool from the agent's available set. The agent continues operating with its remaining capabilities. This is fast and low-risk because the agent was working fine before the tool was added. For changes to existing capabilities like an improved [ranking algorithm](/blog/how-ai-ranks-your-flights), rollback means reverting to the previous version. This requires maintaining the old version in a deployable state for the duration of the rollout, which adds operational overhead but is essential for safe rollouts. For features that create persistent state like new data stored in the database or new preferences saved to memory, rollback is more complex. You cannot simply remove the code because the data it created still exists. We handle this by ensuring that new features are backward-compatible: the old code can coexist with the new data without breaking. Our target is code revert in under five minutes and full stabilization in under thirty minutes. We rehearse rollback procedures before every staged rollout, not after a problem occurs. The time to learn how to revert is not during an incident. ## The template For teams building their own AI feature rollouts, here is the configuration template we use. Define the capability and its scope. Set the evaluation pass rate threshold. Establish the monitoring signals and alert thresholds. Configure the rollout stages with explicit percentages. Set minimum dwell times at each stage. Define promotion gate criteria for each transition. Document the rollback procedure for each stage. Assign ownership for monitoring at each stage. Prepare communication templates for internal and external audiences. Then run it. Every time. No shortcuts. The day you skip a stage is the day you ship a problem that could have been caught. --- Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. [Plan your next trip](https://app.nowah.xyz).