How Early Feedback Should Change a Roadmap: Signals We Watch

Roadmaps written in isolation rot on contact with real traveler behavior. We use private testing and waitlist conversations to kill features early and promote the ones people actually ask the agent for. This is how feedback is allowed to change the plan *before* public launch freezes priorities.

What we assumed versus what happened

We assumed the primary use case would be straightforward flight booking. A traveler opens the app, asks the agent to find a flight, reviews the options, and books. Our roadmap was organized around making this flow faster, richer, and more reliable. More airline integrations. Faster search results. Better price comparison displays.

Travelers did book flights. But they spent far more time on activities we had not prioritized: multi-step trip planning conversations where they explored destinations before committing, itinerary management for trips they had already booked, and informational queries about destinations, visa requirements, and travel logistics.

The agent has over seventy tools, and the usage distribution was not what we expected. Some tools were used a hundred times more than projected. Others were used near zero. The tools that travelers used most heavily were often the ones we had built as supporting utilities, not as primary features.

The tools that we plan for

The tool usage data revealed that travelers valued the agent's ability to handle complex, multi-turn planning conversations more than simple search-and-book transactions. A traveler would start with "I want to go somewhere warm in April for about a week" and spend twenty minutes in a conversation that explored destinations, compared flight options, checked hotel availability, and considered logistics. The conversation was the product, not just a means to reach a booking.

Trip management tools, which let travelers view, modify, and organize their existing bookings, had engagement rates far above what we projected. We had built these as a complement to the booking flow. Travelers treated them as a primary reason to open the app. The daily active usage pattern showed peaks not around searching and booking, which happen infrequently, but around managing and reviewing existing trips, which happen regularly.

Informational queries about destinations, customs, safety, and logistics accounted for a surprisingly large share of conversations. Travelers were using the agent as a travel knowledge resource, not just a booking tool. These conversations did not directly generate revenue, but they drove engagement and retention.

The features no one used

We had invested significant engineering time in a feature that displayed detailed fare class breakdowns and comparison matrices. We assumed power travelers would value understanding the differences between fare classes across airlines. Usage data showed that almost nobody engaged with the detailed comparison. Travelers looked at the price, the schedule, and the airline name. The fare class details were ignored.

We had also built a social sharing feature that let travelers share trip details with friends. The feature worked well technically. Nobody used it. Travelers shared trip information through their existing messaging apps by taking screenshots, not through our built-in sharing flow.

The gap between stated preference and revealed preference was significant. In pre-launch surveys, travelers said they wanted detailed fare comparisons and social sharing. In practice, they ignored both. Surveys capture what people think they want. Behavior reveals what they actually value.

Reading the data correctly

Not every underused feature is unwanted. Some features are undiscovered. Distinguishing between "not wanted" and "not found" is critical because the response to each is different. An unwanted feature should be deprioritized or removed. An undiscovered feature needs better surfacing.

We used multiple signals to make this distinction. Features that travelers discovered but did not use repeatedly were genuinely unwanted. Features that had low discovery rates but high engagement among those who found them were undiscovered. The fare comparison fell into the first category: travelers saw it and did not care. Other features fell into the second: travelers who stumbled upon them loved them, but most travelers never discovered them.

For undiscovered features, the fix was not engineering. It was design. We redesigned the agent's conversational patterns to naturally introduce capabilities that travelers were not finding on their own. Instead of waiting for a traveler to ask about trip management, the agent proactively offered trip updates after a booking was confirmed.

The roadmap rewrite

The reprioritization happened within two weeks of launch data analysis. It was not a minor reshuffle. It was a fundamental reordering.

Multi-turn planning conversations moved from a background priority to the top of the roadmap. If travelers valued the exploration and planning conversation, we needed to invest in making that conversation dramatically better: more destination knowledge, more contextual recommendations, and better memory of preferences across planning sessions.

Trip management moved from a supporting feature to a core retention driver. If travelers opened the app daily to check on existing trips, we needed to make the trip management experience richer: real-time status updates, proactive notifications about changes, and tools for modifying bookings.

Informational query handling moved from a nice-to-have to a strategic priority. If travelers used the agent as a travel knowledge resource, that behavior was the top of our engagement funnel. Better informational responses would drive more conversations, more trust, and eventually more bookings.

The detailed fare comparison was deprioritized. The social sharing feature was removed entirely. The engineering resources freed up were redirected to the features that travelers were actually using.

Building a responsive roadmap process

The launch is why we design for a six-month roadmap based on pre-launch assumptions is a fiction. It feels productive to plan six months ahead. It is not productive if those plans are based on what you think travelers want rather than what they actually want.

Our roadmap process now operates on a shorter cycle with built-in adaptation points. We plan in detail for the next six weeks and in broad strokes for the next six months. Every two weeks, we review usage data and adjust priorities. Major reordering requires explicit justification, but minor adjustments happen continuously.

We also changed how we weight different data sources. Usage data from production now outweighs survey data from pre-launch research. Stated preferences are inputs to hypothesis generation. Revealed preferences are inputs to prioritization decisions. The two are not interchangeable.

The roadmap slide that showed our beautiful six-month plan is still saved somewhere. We keep it as a reminder that plans are hypotheses, not predictions. The travelers who used our product taught us more in two weeks than six months of planning had. Our job is to listen to what they do, not just what they say, and to have the humility to reorganize our plans when the data says we were wrong.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.