--- title: "The AI Model Decision: How We Chose Our Brain" description: Choosing an AI model for a product where wrong answers cost thousands of dollars is not casual. Here is our evaluation process and the trade-offs we weighed. canonical: https://nowah.xyz/blog/ai-model-decision-choosing-brain lastModified: "2026-08-07T08:32:21.845Z" --- # The AI Model Decision: How We Chose Our Brain Choosing an AI model for a product where wrong answers cost thousands of dollars is not casual. Here is our evaluation process and the trade-offs we weighed. When a wrong recommendation costs three thousand dollars and a ruined vacation, choosing which AI model powers your product is not a casual decision. It is one of the most consequential technical choices we make, and we revisit it regularly as the landscape evolves. ## What we evaluate ![Illustration for this section](https://pics.nowah.xyz/website-media/founder-024-img-1.webp) We evaluate models across five axes, each weighted for travel-specific requirements. **Accuracy.** Does the model correctly interpret travel requests and produce factually reliable responses? For travel, this means understanding dates, destinations, airlines, airport codes, and the nuances of complex itineraries. Benchmark scores are a starting point. Travel-specific evaluations are what matter. **Latency.** How quickly does the model respond? Users expect visible progress within one to two seconds. The full response can take longer, but initial acknowledgment must be near-instant. A model that is ten percent more accurate but fifty percent slower is a worse choice for our use case. **Cost per query.** Every conversation has a real cost. The model must be efficient enough that our [unit economics](/blog/cost-first-thousand-ai-booked-flights) work at scale. A model that produces perfect responses but costs ten times more per query is not viable for a consumer product. **Tool use reliability.** This is the [most important](/blog/why-speed-is-most-important-feature) axis for our use case. The agent has over a large set of tools it can call: flight searches, hotel lookups, [payment processing](/blog/launching-payment-processing-ai-handles-money), [document management](/blog/launching-document-management-boarding-passes). The model must reliably decide which tools to call, in what order, with what parameters. Poor tool use is worse than poor text generation because it produces wrong results rather than just awkward phrasing. **Reasoning depth.** Complex travel queries require multi-step reasoning. "Find me the best option considering budget, timing, and that I need to be near the conference venue" requires the model to weigh multiple constraints simultaneously. Shallow reasoning produces shallow recommendations. ## The evaluation process We do not pick models based on public benchmarks alone. Our evaluation process has three layers. First, benchmark evaluation against our custom test suite. Hundreds of travel-specific scenarios, from simple one-way flights to complex multi-leg itineraries with special requirements. Each scenario has evaluation criteria that account for the non-determinism of AI outputs. Second, real-world evaluation with actual users. We run comparisons where different models power the agent for different conversation segments, and we measure [user satisfaction](/blog/measuring-user-satisfaction-ai-products), booking completion, and error rates. Third, edge case stress testing. We throw the hardest possible scenarios at each model: last-minute bookings, unusual routing, conflicting constraints, ambiguous requests. The model that handles edge cases gracefully wins our trust even if its average-case performance is slightly lower. ## Trade-offs between models ![Supporting diagram](https://pics.nowah.xyz/website-media/founder-024-img-2.webp) No model wins on every axis. The frontier is a series of trade-offs. Some models are faster but less accurate on complex reasoning. Some are more accurate but prohibitively expensive at scale. Some excel at text generation but [struggle with](/blog/legacy-otas-struggle-with-ai-architecture) reliable tool use. Some handle simple queries perfectly but fall apart on multi-step tool chains. We optimize for the combination that best serves travel: reliable tool use, strong reasoning, acceptable latency, and sustainable cost. Text generation quality is important but secondary to these concerns because the agent's value comes from what it does, not how eloquently it describes what it is doing. ## Staying current without chasing The AI model landscape changes monthly. New releases, new capabilities, new benchmarks. It is tempting to switch to the latest model every time something new drops. We resist this temptation. Model migrations are expensive. They require re-evaluation, re-tuning, and regression testing. A migration that introduces a subtle degradation in tool use reliability can cause more harm than the improvement in text quality. Instead, we evaluate new models against our existing test suite before considering any migration. If a new model meaningfully improves on our most important axes without regressing on others, we migrate. If it is a marginal improvement, we wait. The goal is to stay current, not to be first. The best model for our product is the one that reliably serves travelers, not the one that won the latest benchmark contest. Our evaluation framework is available for any AI team that wants to assess models for their own travel or high-stakes use case. The methodology matters more than the specific model, because the right answer changes, but the right way to find the answer does not. --- Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. [Plan your next trip](https://app.nowah.xyz).