Designing for Scale: Infrastructure Risks Between Early Users and Growth

Scale failures are predictable if you have seen them before: connection pools exhaust, provider rate limits trip, job queues back up, realtime fan-out multiplies cost. We design for those cliffs while traffic is still small — so growth does not become a blank-screen incident.

The amplification problem

A traditional web application serves a request by querying a database and returning a response. One request, one query, one response. The relationship between user count and infrastructure load is roughly linear.

An AI travel application amplifies every user action. A single message from a traveler can trigger the agent to make five to ten tool calls. Each tool call might hit an external API, query the database, or process data. A flight search tool call generates an API request to the travel data provider, which returns results that are processed, ranked, and formatted. A booking tool call triggers payment processing, provider confirmation, notification dispatch, and record creation.

One traveler message can generate fifty backend operations. At a hundred users, that is manageable. At much larger users with concurrent conversations, the amplification turns a modest user count into an enormous operational load.

We did not fully account for this amplification in our capacity planning. Our load tests simulated user counts and request rates, but they used simplified request patterns rather than realistic AI agent behavior. A realistic simulation would have shown that much larger users generating AI conversations produce backend load equivalent to a hundred thousand users on a traditional platform.

Database connection exhaustion

The first bottleneck was the database connection pool. Our a relational database database, managed through an ORM, had a connection pool sized for our expected load. At a hundred users with AI-amplified queries, the pool was comfortably utilized at around 40 percent. At much larger users, it hit 100 percent and new connections were queued.

The queue grew faster than connections were freed because AI agent conversations hold connections longer than typical web requests. A traditional request takes milliseconds: query, respond, release. An agent conversation might hold a connection for seconds while the agent reasons, calls tools, and generates a response. The combination of longer hold times and more concurrent sessions exhausted the pool.

The fix was multi-layered. We increased the connection pool size, which bought immediate headroom. We added a connection pooler between the application and the database, which manages connections more efficiently than the application-level pool. We optimized the longest-running queries by adding database indexes that reduced query time by an order of magnitude for the most frequent access patterns. And we restructured the agent's database access to acquire connections only when needed and release them immediately rather than holding them for the duration of a conversation.

Upstream rate limits

Our infrastructure was not the only bottleneck. The travel data provider that supplies flight and hotel inventory has its own rate limits. At a hundred users, our API call volume was comfortably within the provider's quotas. At much larger users generating concurrent searches, we exceeded the provider's rate limits during peak hours.

When the provider throttled us, search requests returned errors instead of results. The agent interpreted these errors and told travelers that no flights were available, which was incorrect. Flights were available. We just could not access them because we were being throttled.

The solution involved multiple strategies. Request coalescing combines identical or near-identical search requests into a single upstream call. If three travelers simultaneously search for flights from New York to London on the same date, we make one API call instead of three. Result caching stores search results for a short duration so that subsequent requests for similar routes and dates can be served from cache without hitting the upstream API. And graceful degradation distinguishes between "no results found" and "results temporarily unavailable" so the agent can accurately communicate the situation to the traveler.

We also negotiated higher rate limits with the provider based on our growth trajectory. But rate limit increases are temporary solutions. The long-term solution is architectural: reducing upstream API dependency through caching, coalescing, and efficient request management.

Real-time connection management

The platform uses a bidirectional socket connections for real-time features: live flight status updates, booking progress notifications, and chat message delivery. Each connected user maintains a persistent a bidirectional socket connection.

At a hundred users, a bidirectional socket connections consumed minimal resources. At much larger, memory usage for connection state grew faster than linearly because each connection maintains buffers, event listeners, and associated state. The memory footprint per connection was small, but multiplied by much larger with associated overhead, it consumed a significant portion of available memory.

The fix involved optimizing connection state management: reducing the per-connection memory footprint, implementing connection timeouts for idle sessions, and adding connection limits with graceful queuing for periods of extreme demand. We also restructured how real-time updates are dispatched, batching multiple small updates into fewer, larger payloads to reduce per-message overhead.

Job queue saturation

Background job processing handles asynchronous tasks: booking confirmations, email delivery, notification dispatch, status polling, and analytics processing. The system uses seven worker queues, each dedicated to a category of work.

At peak traffic with much larger users, all seven queues were hitting capacity simultaneously. Booking confirmation jobs backed up, causing travelers to wait longer for their confirmation emails. Notification jobs backed up, causing delayed delivery of flight status updates. Analytics jobs backed up, causing gaps in our monitoring dashboards at exactly the moment we needed them most.

The job queue system uses an in-memory data store for queue management, and its memory usage scaled faster than expected because job retention settings kept completed jobs in memory for debugging purposes. At volume, the completed job history consumed more memory than the active queue.

We fixed the immediate issue by reducing job retention duration and increasing worker concurrency for critical queues. Booking confirmation and notification workers received priority over analytics workers during peak periods. The longer-term fix was implementing job prioritization logic that automatically scales worker allocation based on queue depth, ensuring that user-facing jobs are never delayed by backend processing tasks.

The "works at 100 users" trap

Every system works at small scale. The architecture that handles a hundred users gracefully may collapse at much larger because small-scale success masks structural limitations.

AI travel products are particularly susceptible to this trap because of the amplification factor. A load test that simulates much larger concurrent HTTP requests does not capture the reality of much larger concurrent AI conversations, each generating cascading tool calls and database operations.

Our improved load testing now simulates realistic agent behavior: multi-turn conversations with tool calls, search requests, booking flows, and notification delivery. The simulation includes the amplification factor so that infrastructure capacity is measured against actual operational load, not theoretical request rates.

The scaling playbook

After stabilizing at much larger users, we built a playbook for the next ten-times growth.

Database scaling: connection pooling is now a first-class infrastructure component with automated scaling based on connection utilization. Query performance is continuously monitored with automatic alerting when any query exceeds latency thresholds.

Upstream API management: request coalescing and result caching are now default behaviors for all external API calls, not just the flight search endpoint. Rate limit consumption is monitored in real time with automatic throttling of our own traffic before the provider's limits are reached.

Real-time connections: a bidirectional socket infrastructure is designed for horizontal scaling with connection routing that distributes load across multiple server instances. Connection state is externalized so that any server can handle any connection.

Job queue scaling: worker allocation is dynamic, with automatic scaling based on queue depth and priority. User-facing job queues have guaranteed minimum workers that are not shared with background processing queues.

The infrastructure that serves much larger users today is fundamentally different from the infrastructure that served a hundred users six months ago. Every component has been rebuilt, optimized, or replaced. The architecture that works at much larger is designed with the awareness that a hundred thousand is coming, and the systems must scale smoothly rather than hitting walls that require emergency intervention.

Scaling is not a problem you solve once. It is a practice you maintain continuously. The walls are always ahead. The goal is to see them before you hit them.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.