Real-Time Data Pipelines for AI Travel Search

Flight prices are among the most volatile consumer prices in existence. A fare can change five times in a single day. A seat that's available at 10 AM might be gone by 10:15. Two people searching for the same flight at the same time might see different prices depending on which provider responds first.

This volatility is fine for a traditional search engine that shows results once and lets the user browse. It's a nightmare for a conversational AI that might reference a price mentioned three minutes ago, recommend an option the user takes two minutes to think about, and then try to book it.

Serving travel data fast enough for conversational AI, while keeping it fresh enough to be trustworthy, is a real engineering challenge. Here's how we handle it.

The freshness problem

When a user says "Find me flights to Tokyo in April," our agent searches live travel data providers and returns results with current prices. Those prices are accurate at the moment of search. But the user might spend several minutes discussing the options, asking follow-up questions, comparing with hotels, and then deciding to book.

During those minutes, prices can change. Seats can sell out. Availability can shift. If we quote $780 at search time and the price is $820 at booking time, the user is surprised and annoyed. If a seat sells out between search and booking, the user is frustrated.

The challenge is balancing speed (return results fast) with freshness (results should be accurate when acted upon).

Caching strategies for volatile data

We cache travel data, but with nuance. Not all data is equally volatile, and caching everything the same way leads to either stale prices (cache too long) or wasted API calls (cache too short).

Search results get short-lived caches. If the same user searches the same route within a few minutes, we return cached results instead of hitting the provider API again. This handles the common case where the user refines their search multiple times in quick succession ("Show me those Tokyo flights again, but for the second week instead").

Price data gets even shorter caches with explicit freshness tracking. Every cached price has a timestamp. When presenting prices to the user, we include a relative freshness indicator. At booking time, we always re-verify the price with a fresh API call, regardless of cache state.

Reference data gets longer caches. Airport codes, airline names, aircraft configurations, route availability. This data changes infrequently (daily at most) and can be cached aggressively.

Availability data is the most volatile and the least cacheable. We cache availability for very short windows to handle rapid re-requests, but for anything older than a few minutes, we do a fresh check.

Caching and request reduction cut our travel API costs by a large share. Travel API providers charge per search, so every unnecessary API call is wasted money. Smart caching is both a performance optimization and a cost optimization.

Data normalization across providers

Different travel data providers return the same information in different formats. Flight number conventions differ. Price currencies vary. Fare class codes aren't standardized. Baggage policies are described differently. Layover airports might use IATA codes, full names, or a mix.

We normalize all provider responses into a canonical format before they enter our system. This normalization layer:

Standardizes identifiers. All airports use IATA codes. All airlines use standard two-letter codes. All currencies use ISO codes.

Unifies pricing. All prices are stored in their original currency plus a converted common currency for comparison. Exchange rates are updated frequently.

Normalizes time. All times are stored in UTC with the local timezone noted. This avoids the common bug where a departure time in local time gets compared to an arrival time in a different timezone.

Consolidates duplicate results. The same flight from two different providers gets merged, with the best price and fullest data set retained.

This normalization is invisible to the user but essential for the AI agent. The ranking algorithm needs comparable data across options. The agent's reasoning needs consistent formats. A mismatch between how two providers express baggage allowance would produce confusing recommendations.

Handling stale prices gracefully

Despite our best caching and freshness efforts, prices will sometimes change between search and booking. We handle this transparently.

At booking time, we do a fresh price check. If the price has changed, the agent tells the user: "The price went up by $35 since your search. The current fare is $815. Want to proceed, or should I search for other options?"

This transparency is a product decision backed by engineering. We could silently process at the new price. Many OTAs do. But that erodes trust. The user saw $780 and is being charged $815 without explanation. Even if the explanation is simply "prices changed in the last few minutes," saying it out loud maintains the relationship.

If a seat sells out between search and booking, the agent handles that too: "That flight is no longer available. Your other two options are still bookable. Want one of those, or should I search again?"

The engineering here is in the detection and conversational recovery, not in the pricing itself. Our booking pipeline is designed to verify, explain, and offer alternatives rather than silently proceeding or silently failing.

Performance optimization for sub-second responses

Our agent responds with streamed content within a couple of seconds. Meeting that target while fetching data from external providers in real time requires aggressive parallelism and smart latency management.

Parallel provider queries. When searching flights, we query multiple providers simultaneously and merge results. The total search time is determined by the slowest provider, not the sum of all providers. We set timeouts so that a slow provider doesn't hold up the entire search.

[Connection pooling](/blog/connection-pooling-under-pressure). We maintain persistent connections to provider APIs rather than establishing new connections for each request. Connection setup time is surprisingly significant when you're measuring in milliseconds.

Pre-warming. For common routes and dates, we pre-fetch data before the user asks. If a user is planning a trip to Barcelona and mentions April dates, we might pre-fetch flight data for several date combinations so that follow-up searches return from cache.

Request batching. When the agent needs to check availability for multiple date combinations, we batch these into a single provider request where the API supports it, rather than making separate calls for each date.

The global online travel market exceeds $800 billion annually. The companies that win in this market will be the ones that serve accurate data fastest. Speed and freshness are direct competitive advantages when your interface is a conversation that doesn't tolerate loading spinners.

Where data freshness beats data volume

The largest travel search companies have a data advantage: they process billions of searches and can build comprehensive pricing databases. But that volume advantage matters less than you'd think.

A price database that's six hours old is less useful than a fresh API call that returns current prices. A historical trend ("this route is usually cheapest on Tuesdays") is valuable for advice but useless for a quote that the user will act on in the next five minutes.

We focus on data freshness over data volume. Our prices come from live provider APIs at search time. Our recommendations are based on what's actually bookable right now, not what was bookable yesterday. Our booking confirmation reflects the price at the moment of booking, verified in real time.

This approach means we don't need a massive data warehouse of historical prices (though we do track trends for the agent's advisory capabilities). We need fast, reliable connections to live data sources and smart caching to minimize the cost and latency of real-time data fetching.

For AI travel search specifically, real-time data is the foundation. Every recommendation, every price quote, every availability check needs to be grounded in what's actually true right now. The data pipeline's job is to make that real-time grounding fast enough for conversation.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.