Travel Data at Scale: Ingesting, Normalizing, and Serving

Travel data is a mess. I say this with no disrespect to the industry. When you have thousands of airlines, tens of thousands of airports, millions of routes, and billions of price points that change every minute, any human-readable structure is an achievement. But "human-readable" and "AI-ready" are different standards, and the gap between them is where our data engineering work lives.

The messy reality

Here is a concrete example. A flight from San Francisco to London might appear in our travel data provider's API with the airport code SFO or with the city code QSF (the multi-airport code for the San Francisco Bay Area). The airline might be identified by its IATA code (BA for British Airways), its ICAO code (BAW), or its full name (which itself varies: "British Airways," "BRITISH AIRWAYS PLC," "British Airways Plc"). The departure time might be in the airport's local timezone, UTC, or an offset notation. The price might be in the booking currency, the fare currency, or both.

And that is one flight from one provider. Different providers return the same flight with different formatting for every one of these fields. Airport codes might be padded or trimmed differently. Time formats might use different ISO 8601 variants. Price decimals might use commas or periods depending on the provider's locale.

If we pass this raw data to the AI agent, the agent has to figure out that SFO and QSF both mean San Francisco, that BA and BAW and British Airways are the same airline, and that "14:30" and "2:30 PM" are the same time. The agent can probably handle this most of the time, but "most of the time" is not good enough when you are booking real flights with real money.

Reference data management

We maintain our own reference data for airports and airlines. This is the canonical source of truth that all other data maps against.

Our airport dataset covers over 10,000 airports worldwide. Each entry has the IATA code, ICAO code, full name, city, country, timezone, latitude and longitude, and various aliases and alternative names. We have dedicated sync scripts that update this data from authoritative sources.

The airline dataset covers over 5,000 airlines. Each entry includes IATA code, ICAO code, full name, country of origin, alliance membership, and operational status (active, defunct, charter-only). Airlines merge, rebrand, and go bankrupt regularly, so this data requires ongoing maintenance.

This reference data serves multiple purposes. It normalizes incoming search results (map "BA" to our canonical British Airways record). It provides the AI agent with knowledge about the aviation network (which airports are hubs, which airlines are in the same alliance). And it powers features like flexible airport search ("fly from the Bay Area" means consider SFO, OAK, and SJC).

Real-time pricing challenges

Flight prices change constantly. A seat that costs $450 right now might cost $480 in ten minutes because demand shifted or a fare bucket sold out. Hotels are slightly less volatile but still change frequently, especially as occupancy thresholds trigger rate adjustments.

This volatility creates a tension between data freshness and API cost. We could search for live prices on every user request, but that would be expensive and slow. We could aggressively cache and serve slightly stale data, but that risks showing users prices they cannot actually book at.

Our strategy is layered. For a new search (user's first request for a specific route and dates), we always hit the live API. For refinements of the same search (user asks "what about one day later"), we check whether cached results can answer the question first. For repeated searches by different users (popular routes), we cache results with short TTLs (3-5 minutes for flights, 10-15 minutes for hotels).

We always verify the price at booking time. When a user selects an option and initiates booking, we re-validate the price with the provider. If the price has changed, the agent tells the user: "The price for this flight has changed from $450 to $465 since I showed it to you. Want to continue?" This is honest and builds trust.

Normalization pipeline

Raw API responses from our travel data providers go through a normalization pipeline before reaching the AI agent.

Step 1: Schema mapping. Each provider's response format is mapped to our internal schema. Airport codes are resolved against our reference data. Airline codes are canonicalized. Times are converted to a consistent format with explicit timezone.

Step 2: Enrichment. We add data that the provider does not include. Alliance membership for airlines. Terminal information for airports. Historical on-time performance for routes. This enrichment data comes from our reference datasets and helps the AI agent make better recommendations.

Step 3: Validation. We check for obvious errors: departure after arrival, negative prices, unknown airport codes, duplicate segments. Bad data is flagged and excluded rather than passed to the agent. Better to show fewer results than to show wrong results.

Step 4: Deduplication. The same flight might appear from multiple providers with slightly different formatting. We deduplicate based on airline, flight number, and departure time, keeping the result with the best price or most complete data.

After normalization, the AI agent receives clean, consistent data that it can reason about confidently. The agent does not need to know which provider the data came from or what format it was originally in.

Data quality for AI consumption

Data quality matters more when AI is the consumer than when humans are. A human looking at a search results page can mentally correct minor inconsistencies. "Oh, that price is probably in euros, not dollars, because it's a European airline." The AI agent might not make that inference, or worse, it might make the wrong inference confidently.

We enforce data quality at the pipeline level. Every field in our internal schema has a type, a format specification, and a set of valid values. The normalization pipeline enforces these constraints, and any data that does not pass validation is logged and excluded.

Data quality directly impacts AI recommendation quality. If the agent sees conflicting information about a flight (different prices from different sources, inconsistent times), it cannot make a confident recommendation. Clean, consistent data enables confident, accurate agent responses.

Google Flights benefits from data scale, with billions of queries giving them pricing trend insights no one else has. But scale alone does not solve the personalized serving problem. Having the world's best data about average flight prices does not help you pick the right flight for a specific user with specific preferences. That is where AI-powered ranking on top of clean, normalized data creates the most value.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.