Running LLM Inference at Scale Without Going Broke

The monthly language model bill just tripled. We shipped a new feature that added contextual recommendations to every flight search result. Each recommendation added 200 tokens of context and 100 tokens of response. Multiply by thousands of searches per day, and the cost increase was predictable in retrospect but surprising in the moment.

This is the core tension of building on language models: the thing that makes the product good (more context, richer reasoning, better personalization) is also the thing that makes it expensive. Every improvement that adds tokens adds cost. Managing that trade-off is what I'd call AI cost engineering, and it's one of the most important disciplines for any team building AI products.

Token consumption varies wildly

The most surprising thing about our cost structure is the variance. A simple query like "What's the weather in Tokyo?" costs almost nothing. The context is small, the reasoning is trivial, the response is short. Maybe a few hundred tokens total.

A complex query like "Plan a two-week trip through Southeast Asia for a family of four with a teenager who's vegetarian, staying under $5,000, with at least three beach days" is a different story entirely. The agent reasons through multiple destinations, calls the flight search tool for several routes, calls the hotel search tool for multiple cities, considers dietary requirements, checks visa requirements, looks at weather patterns, and composes a detailed response. That's thousands of tokens of context, multiple rounds of tool calls (each adding to the context), and a long response. The cost difference between the simple and complex query can be 10-100x.

This variance makes flat-rate pricing difficult and makes per-query cost monitoring essential.

Prompt engineering for cost efficiency

The model instructions is the biggest fixed cost per query. It includes the agent's role, capabilities, available tools, behavioral guidelines, and safety instructions. Every query pays this cost.

We've spent significant effort reducing the model instructions's token count without reducing its effectiveness. Concise tool descriptions, compressed guidelines, and carefully curated few-shot examples. A 20% reduction in model instructions length is a 20% reduction in the fixed cost of every single query.

Tool results are the biggest variable cost. A flight search returns structured data about multiple options. We control how much of that data enters the model's context. Raw API responses might be 5,000 tokens per search. Our cleaned and summarized results are 500-1,000 tokens. The model reasons just as well on the summary. The 4x token savings pays for itself immediately.

Memory context is the hidden cost. Every query that benefits from personalization injects preference data into the context. We rank memories by relevance and include only the top matches rather than everything we know about the traveler. Injecting 200 relevant tokens is worth far more than injecting 2,000 tokens where most are irrelevant.

Model selection by query complexity

Not every query needs the most capable model. Simple factual questions, greetings, and clarification requests can be handled by lighter models at a fraction of the cost. Complex multi-step trip planning needs the most capable model for good results.

We're exploring query routing: a lightweight classifier that examines the incoming query and routes it to the appropriate model. Simple queries go to a fast, cheap model. Complex queries go to the full-capability model. The classifier itself is cheap to run and can reduce overall inference cost substantially by keeping simple queries off the expensive model.

The challenge is accuracy. If the classifier routes a complex query to the simple model, the traveler gets a bad response. If it routes a simple query to the complex model, we waste money but the traveler doesn't notice. We optimize for conservative classification: when in doubt, use the more capable model. The cost of a bad response is higher than the cost of a few extra tokens.

What you can and cannot cache

Language model responses are tempting to cache. If ten travelers ask about the weather in Tokyo, why compute the answer ten times?

In practice, caching AI responses is dangerous for our use case. Each response should be personalized. The weather in Tokyo means different things to different travelers (one might need beach weather advice, another might be asking because they're packing). The agent's tone, level of detail, and supplementary information should vary by traveler context.

What we do cache: tool call results for identical queries within a short window (request coalescing, not response caching). If ten concurrent users search for JFK to NRT on the same date, one API call serves all ten. The model still generates a unique response for each, but the expensive external API call happens once.

We also pre-compute some context that multiple queries need. Airport information, visa requirements, and safety advisories are relatively stable. We can retrieve and prepare these outside the inference loop so the model doesn't need to reason about fetching them.

Cost attribution

We track inference cost per feature, per user segment, and per time period. This lets us answer questions like: "How much does the flight search recommendation feature cost per user per month?" and "Which user segment generates the highest inference cost?"

Cost attribution requires tagging each inference call with the feature context. A query triggered by the search recommendations feature gets tagged differently than a query from a direct user message. The tags flow through to the billing data, and we can slice costs by any dimension.

This data drives product decisions. If a feature costs $0.15 per user per month and increases booking conversion by 0.5%, is it worth it? That's a question we can answer with cost attribution data. Without it, the feature's cost is invisible, buried in the aggregate language model bill.

Building your cost management framework

If you're running language model inference at scale, here's the framework.

Monitor token consumption per query, broken down by input tokens, tool call tokens, and output tokens. This tells you where the cost is concentrated.

Reduce the model instructions. Every token saved is saved on every query. This is the highest-leverage optimization.

Summarize tool results before injecting them into context. Don't send raw API responses to the model. Send clean, relevant summaries.

Implement request coalescing for identical external API calls. This reduces external API costs and latency without affecting personalization.

Track cost per feature. If you can't attribute costs to features, you can't make informed product decisions about which features are worth their inference cost.

Set rate limits as cost guardrails. Our 30 messages per minute limit on the chat endpoint isn't just about abuse prevention. It's about bounding the maximum inference cost any single user can generate in a given time window.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.