Cost Engineering for AI Products: LLM, API, and Infra

The economics of running an AI product are different from running a traditional web product in ways that surprised us early on. A traditional SaaS product has infrastructure costs that scale roughly with users. An AI product has infrastructure costs that scale with conversation complexity, which is a much less predictable dimension.

This post breaks down our cost structure, the optimization strategies that have worked, and the metric we obsess over: cost per booking.

The unique cost structure

AI-native companies spend 15-30% of their infrastructure budget on AI model inference. That number was abstract to us until we got our first production invoice. Our cost structure breaks down roughly like this:

AI inference: ~30% of cost per booking. Every message the user sends triggers an inference call. The AI model reads the conversation context, the model instructions, available tool definitions, and the user's message, then generates a response. The cost scales with the total tokens processed (input context plus output response). A simple "thanks!" costs less than a complex "find me the cheapest flight from any New York airport to any London airport, any day in March, direct only, and also search hotels near the West End for the first week."

Travel API costs: ~25%. Our travel data providers charge per search. Each flight search or hotel search is a billable event. A single user conversation might trigger 1-2 searches (simple trips) or 8-10 searches (complex multi-city planning with refinements). These costs are relatively fixed per search, but the number of searches per booking varies dramatically.

Infrastructure: ~25%. Compute, database, an in-memory data store, storage, networking. These are more predictable and scale more linearly than AI costs. They are also the area where traditional cost optimization techniques (right-sizing instances, reserved capacity, auto-scaling) work as expected.

Operations: ~20%. Monitoring, logging, third-party services (authentication, email, push notifications, analytics), and the human infrastructure around keeping the product running.

LLM inference optimization

LLM inference costs have dropped roughly 10x in the past 18 months thanks to competition and efficiency improvements among AI model providers. This trend is our friend, but we do not rely on it. We optimize aggressively because usage grows faster than prices fall.

Context window management. The biggest driver of inference cost is the size of the context window. If the agent reads 4,000 tokens of conversation history and tool definitions for every message, that is the base cost before it generates a single word of response. We manage context carefully: only include the most recent relevant messages, summarize older history, and exclude tool definitions for tools the agent is unlikely to need for the current query.

Prompt engineering for efficiency. Shorter model instructions that achieve the same quality reduce per-request cost. We continuously refine our prompts to be more concise without losing capability. Shorter instructions that keep quality reduce input-token cost on every request.

Response caching. Some agent responses are effectively deterministic. "What is the currency in Japan?" always produces the same answer. We cache these common responses and skip inference entirely. This handles roughly 5-10% of messages.

Model selection. Not every message needs the most powerful model. Simple acknowledgments, clarifying questions, and reference lookups can use a smaller, cheaper model. We route messages to the appropriate model tier based on estimated complexity.

Travel API cost management

Travel API costs scale directly with the number of searches, and our AI agent has a natural tendency to search eagerly. Ask it "what about a slightly different date?" and it might run an entirely new search when the previous results could answer the question.

Intelligent search reuse. When the agent has already fetched flight results for a route, we teach it to reference those results for follow-up questions. "What about a day earlier?" triggers a check of the cached results first. If the cache covers the requested dates, no new API call is needed.

Search deduplication. Multiple users searching similar routes around the same time get deduplicated. If three users search for NYC to London next Tuesday within a 5-minute window, only one API call goes to the provider.

Batch and prefetch. For popular routes, we prefetch search results during off-peak hours and cache them. When a user searches for a popular route, the results are already warm.

These strategies together cut our travel API costs by a large share compared to naive search-on-every-request behavior.

Cost per booking as the north star

We track dozens of cost metrics, but cost per booking is the one that ties everything together. It includes all costs: AI inference, travel API searches, infrastructure allocated to that user, email delivery, notification delivery, and a share of fixed operational costs.

The goal is to get the cost of an AI-powered booking interaction below the cost of a human-handled interaction. Traditional travel agents cost $20-50 per booking in labor. OTAs cost $5-15 per booking in customer acquisition and infrastructure. Our AI-powered booking needs to come in at or below the OTA benchmark while providing a service quality closer to the human agent.

We are approaching that threshold. The main drivers of improvement are decreasing AI inference costs (external trend), better prompt optimization (internal effort), and smarter caching (internal effort). Each month, our cost per booking trends down.

We review cost per booking weekly, broken down by conversation complexity. Simple one-way flight bookings are well below target. Complex multi-city itineraries with hotel bundles are still above. The optimization work focuses on the expensive tail: reducing the cost of complex conversations without reducing their quality.

The trajectory

The economics of AI products are on a favorable trajectory. Inference costs are falling. Model efficiency is improving. Our optimization techniques compound over time. The cost of an AI-powered booking interaction is approaching parity with human-handled interactions, and once it crosses that line, the unit economics become very attractive because AI scales without proportional headcount growth.

Cost engineering for AI products is a discipline that did not exist three years ago. We are learning it in real time, and we suspect the patterns we are developing will be applicable to any AI-native product, not just travel.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.