Rate Limiting and AI Agent Resource Management
Each user query triggers 3-7 API calls to external providers. Without cost-aware rate limiting, a single conversation can exhaust your budget.
A traditional web application serves a page. The cost per request is negligible: some compute, a database query, a template render. You can handle millions of requests cheaply.
An AI travel agent is different. Each user message triggers a chain of expensive operations: LLM inference (paying per token), external API calls to flight and hotel data providers (paying per search), memory retrieval (compute-intensive embedding lookups), and potentially payment processing. A single complex query can trigger 3-7 external API calls.
Without rate limiting, a single abusive or even enthusiastic user can exhaust your daily API budget in a few hours. And without cost-aware resource management, you spend more on a user's casual browsing than you earn from their eventual booking.
The cost multiplication problem
When a user types "find me flights to Tokyo," here is what happens behind the scenes:
- LLM inference to interpret the query (~3,500 input tokens + ~100 output tokens)
- Flight search API call to the data provider ($0.01-0.05 per search)
- LLM inference to rank results and generate response (~5,500 input tokens + ~800 output tokens)
- Memory write to update user context (embedding computation)
That is a minimum of two LLM inference calls and one external API call for a simple flight search. A complex query that searches flights and hotels, checks visa requirements, and looks up weather triggers 5-7 external operations.
At consumer scale with thousands of daily active users, each having multi-turn conversations, the total daily API spend is significant. And the spend is per interaction, not per user. A user who sends 50 messages in a day costs 50x more than a user who sends one.
Per-user rate limiting
Our rate limiting operates at multiple levels:
Chat messages: 30 per minute. This prevents a user from flooding the agent with rapid-fire messages. In practice, even the most active user rarely exceeds 10 messages per minute during normal usage. The 30/minute limit catches automation and abuse without affecting legitimate users.
Booking attempts: 5 per hour. This prevents accidental or malicious multiple bookings. Combined with idempotency, it is a belt-and-suspenders approach to booking safety.
Payment operations: 10 per minute. This prevents payment fraud attempts. Legitimate users rarely need more than 1-2 payment operations per booking session.
The rate limits are per user, not global. A busy user hitting their limit does not affect other users. The limits are generous enough for normal usage and tight enough to prevent abuse.
When a limit is hit, the agent responds helpfully rather than with a generic error: "I am handling a lot of requests from you right now. Give me a moment to catch up, and I will be ready for your next question."
Cost-aware tool selection
Not all tools cost the same. A flight search API call might cost 5-10x more than a weather lookup. A hotel search with detailed property information costs more than a basic availability check.
We implement cost-aware tool selection: the agent queries cheaper APIs first and escalates to expensive ones only when needed.
For example, when a user asks about a destination, the agent might first check its cached knowledge (free), then call a weather API (cheap), and only call the flight search API (expensive) when the user expresses genuine booking interest. This prevents expensive API calls for casual browsing queries.
The model's model instructions includes cost awareness: "If the user is exploring and not ready to book, provide destination information without triggering flight or hotel searches. Only search live inventory when the user has expressed specific travel intent."
Caching strategies
Caching is the single most effective cost optimization for an AI travel agent. Repeated searches for the same route and dates are common (multiple users searching the same popular routes) and the results are valid for a reasonable time window.
Our caching architecture:
Route-level cache. Flight search results for SFO-NRT in April are cached for 60 minutes. Any user searching the same route within that window gets cached results instantly. The cached results are marked with a freshness timestamp: "Prices checked 23 minutes ago."
User preference cache. A user's preference profile is cached in memory for the duration of their session. This prevents repeated database and vector store lookups for the same preference data across multiple turns.
Destination information cache. Weather, visa requirements, general destination facts, and safety information are cached for 24 hours. This data changes slowly and does not need real-time freshness.
Caching repeated searches reduces API costs by 30-40%. For popular routes (the top 100 city pairs account for a large fraction of all searches), the cache hit rate is even higher.
Quota management across providers
We integrate with multiple external data providers, each with their own rate limits and pricing tiers. Managing quotas across providers requires a coordination layer.
The coordination layer tracks:
- Current usage against each provider's rate limit
- Cost accumulation against budget thresholds
- Response quality across providers (if one provider is returning degraded results, route traffic to others)
When one provider approaches its rate limit, traffic is automatically routed to an alternative. The user sees no difference. The agent continues to search and recommend without interruption.
Budget thresholds add another control dimension. We set daily and monthly spending limits per provider. If the daily limit is reached (which might happen during a traffic spike), the system degrades gracefully: it serves from cache more aggressively, routes to cheaper providers, and if necessary, queues non-urgent queries for processing when budget replenishes.
The economics at scale
At scale, the economics of AI agent API consumption follow a power law. A small percentage of users generate a large percentage of costs. Heavy users who search dozens of routes, compare extensively, and plan complex multi-city trips consume 10-50x the resources of a user who asks a single question and books.
The question is whether these heavy users generate proportionally more revenue. Usually, yes. A user who searches extensively is likely planning a high-value trip and is more likely to book (and to book a more expensive trip) than a casual browser.
Cost per interaction is the critical economic metric. We track it across user segments, query types, and time periods. When cost per interaction rises without a corresponding increase in booking rate or booking value, something is wrong. Either the agent is making unnecessary API calls, caching is underperforming, or users are engaging without converting.
The goal is not to minimize cost per interaction. It is to maximize the ratio of revenue to cost. An expensive interaction that leads to a $2,000 booking is far more valuable than a cheap interaction that leads nowhere.
Rate limiting and resource management are not glamorous engineering work. But they are the difference between an AI travel agent that scales sustainably and one that burns cash with every conversation.
Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.