The AI Product Manager's Playbook: Lessons From Building Nowah

I have been a product manager in various roles for years. I thought I knew how to ship software. Then I started building an AI product and realized that about half of what I knew was wrong or irrelevant.

Not wrong in a "the fundamentals don't apply" way. User research still matters. Understanding the problem still matters. Shipping fast still matters. But the mechanics of how you spec, build, test, and ship an AI product are different enough that many traditional PM frameworks break down.

Here is what we learned building Nowah, an AI travel agent, from zero to a shipping product. These are lessons I wish someone had told me before we started.

Prompt engineering is product design

The single biggest mental shift for a PM working on AI products is understanding that prompt engineering is not an engineering task. It is product design.

The prompt that instructs the AI agent on how to behave is the most important product artifact you have. It is more important than the UI mockups. More important than the database schema. More important than the API design. Because the prompt determines the user experience more directly than any of those things.

When we write the prompt for our travel agent, we are making product decisions. Should the agent be formal or casual? Should it ask clarifying questions or make assumptions? Should it present three options or five? Should it explain its reasoning or just show results? How should it handle requests it cannot fulfill? What personality should it have?

These are not engineering questions. They are product questions. And they require the same rigor, user research, and iteration that any other product decision requires.

In practice, this means the PM writes the first version of every prompt. Not the engineer. The PM. Because the PM understands the user, the use case, and the desired experience. The engineer optimizes the prompt for performance, reliability, and cost. But the intent, the product intent, comes from the PM.

We have learned that a single word change in a prompt can shift the user experience more than a month of UI engineering. Changing "recommend" to "present" in how the agent introduces flight options changed the perceived tone from pushy to neutral. Adding "explain your reasoning briefly" to the prompt increased user confidence in the results without us changing a single UI component.

Prompt engineering is product design. If your PM is not deeply involved in prompt work, your AI product is being designed by accident.

You cannot fully spec AI behavior

Traditional product management runs on specs. You write a PRD that describes what the product should do in detail. You define acceptance criteria. QA tests against those criteria. If the behavior matches the spec, the feature ships.

AI products do not work this way. You can spec the general behavior: "The agent should search for flights based on user preferences and present three options." But you cannot spec every possible response. The AI's output varies every time. Two users asking the same question will get different responses. The same user asking the same question twice will get slightly different responses.

This drove me a little crazy at first. I would write detailed specs, and the AI would produce outputs that were "right" but not what I had imagined. The tone was off. The ordering was different. It included information I did not expect or omitted something I thought was obvious.

The lesson: stop trying to spec outputs. Spec the constraints and the intent.

Instead of "the agent should respond with: [exact text]," write "the agent should acknowledge the user's request, confirm the search parameters, and indicate that it is searching." This leaves room for the AI to generate natural, varied responses while staying within the product intent.

Instead of acceptance criteria that say "the response must contain X, Y, and Z in that order," write criteria like "the response should include the price, the number of stops, and the departure time" without specifying format or order. Then evaluate whether the response feels right, not whether it matches a template.

This is uncomfortable for PMs trained on deterministic software. But the alternative, trying to control every word the AI generates, leads to brittle systems that feel robotic and break whenever the model is updated.

Evaluation is the new QA

If you cannot spec exact outputs, how do you know the AI is working well? The answer is evaluation, and it replaces traditional QA in importance and practice.

In a traditional product, QA runs test cases: "Click button X, verify that page Y loads." Pass or fail. Binary. In an AI product, you run evaluation suites: "Send these 200 travel queries, score the responses on relevance, accuracy, completeness, and tone." The scores are continuous, not binary. An 85% relevance score is better than 80% but worse than 90%. There is no "pass" threshold that applies universally.

We run evaluation suites constantly. Every prompt change, every model update, every tool modification triggers an eval run. The results tell us whether the change made the product better or worse across hundreds of scenarios.

This means shipping decisions are driven by eval scores, not by calendar dates. We do not ship a new agent behavior on Tuesday because Tuesday was the planned release date. We ship when the eval scores for the new behavior exceed the scores for the current behavior across the scenarios we care about.

Building good evaluation suites is one of the hardest parts of AI product management. You need representative test cases that cover the range of real user queries. You need scoring rubrics that capture what "good" means for your product. You need automated scoring for the metrics that can be automated and human evaluation for the ones that cannot.

We evaluate on dimensions like: Did the agent find real flights? Did it present the right number of options? Did it explain its reasoning? Did it respect the user's stated preferences? Did the tone match our product voice? Did it handle ambiguity gracefully? Did it avoid making things up?

Some of these can be scored automatically. "Did the agent present three options?" is binary and testable. "Did the tone match our product voice?" requires human judgment. Building the right mix of automated and human evaluation is ongoing work that never stops.

The demo trap

AI products demo beautifully. You show someone the agent searching for flights, presenting options, completing a booking in a smooth conversational flow. The demo takes five minutes and looks magical.

Then you try to productionize it and discover that the demo was the easy 10%. The other 90% is handling edge cases, error recovery, inconsistent inputs, model hallucinations, API failures, timeout management, cost optimization, and the ten thousand things that go wrong when real users interact with real systems at scale.

The demo-to-production gap is roughly 10x in effort. We spent more time on edge case handling than on the core happy path. What happens when the user asks for flights to a city that does not have a commercial airport? What happens when the flight search API times out? What happens when the user changes their mind mid-booking? What happens when the AI misunderstands the destination?

Every one of these scenarios needs a graceful recovery path. And unlike traditional software where edge cases can be enumerated, AI edge cases are infinite because the input space is natural language. You cannot write a test case for every possible thing a user might say.

The lesson for PMs: do not let the demo set expectations. The demo shows what the product can do under ideal conditions. The product needs to work under all conditions. Budget 10x the time between "it works in a demo" and "it works in production." And be honest with stakeholders about that gap.

Feature flags and gradual rollouts are not optional for AI products. They are survival mechanisms. Every change to the agent's behavior goes behind a flag and rolls out to a small percentage of users first. We monitor eval scores and user feedback on the rollout population before expanding. This catches problems that evaluation suites miss because real users do things that test cases do not anticipate.

Shipping cadence for AI products

Traditional software ships features on a cadence. Quarterly planning, two-week sprints, monthly releases. The product stays stable between releases. Users get updates periodically.

AI products do not work on this cadence. The agent's behavior changes continuously. Prompt adjustments, tool improvements, model updates, memory refinements. These changes are often invisible to users but significantly affect the experience. The flight ranking might improve by 15% after a prompt tweak. The conversation tone might shift subtly after a model update. The agent might handle a new edge case that it previously failed on.

We deploy changes to the agent multiple times per day, sometimes more. Each deployment is small, tested by evaluation, monitored by metrics. There is no big quarterly release because the product evolves continuously.

This requires a different relationship with the PM discipline. You are not managing a backlog of features to ship on a schedule. You are managing a continuous optimization process where improvements are measured, deployed, and monitored in a flowing cycle.

The PM's job shifts from "what do we build next quarter" to "what improved this week, what degraded, and what should we optimize next." It is less like managing a construction project and more like tending a garden.

Prompt and tool improvements are often invisible to users but change the experience dramatically. A user does not see that we rewrote the hotel recommendation prompt. They just notice that the hotel suggestions seem better than they were a month ago. This invisibility is a challenge for communication but a feature for product quality. The product gets better without users needing to learn anything new.

How traditional PM frameworks break down

Let me be specific about which familiar PM tools stop working and what replaces them.

PRDs break down because you cannot fully specify AI behavior. Replace with behavior guidelines and evaluation rubrics. Instead of "the agent must say X," write "the agent should achieve Y outcome" and define how to measure it.

User stories break down in their standard form. "As a user, I want to search for flights so that I can find the best option" is fine as a goal but useless as a spec because the AI has infinite ways to accomplish it. Replace with scenario libraries: collections of representative user queries with expected behavior ranges.

Acceptance criteria break down because AI outputs are non-deterministic. Replace with evaluation score thresholds. "This feature ships when the eval suite shows 90%+ accuracy and 85%+ relevance across the test set."

Sprint planning breaks down because improvement is continuous, not chunked. Replace with a continuous optimization board where the team tracks active experiments, their eval results, and their deployment status.

Bug reports break down for the same reason. "The AI said something wrong" is not a bug in the traditional sense. It is an eval failure on a specific input. Replace with eval case additions: every user-reported issue becomes a new test case in the eval suite, ensuring that specific failure is caught in future evaluations.

The new PM skill set

AI product management requires skills that most traditional PMs do not have.

Prompt literacy. You need to understand how prompts affect AI behavior. Not at the level of building models, but at the level of knowing that "always" in a prompt is interpreted differently than "prefer," and that adding examples to a prompt changes output format dramatically.

Statistical thinking. You need to interpret eval results, which are probability distributions, not binary pass/fail. "The agent answers correctly 87% of the time" requires statistical thinking to evaluate whether that is good enough and how it compares to the previous version.

Ambiguity tolerance. You need to be comfortable shipping products whose behavior you do not fully control. The AI will say things you did not anticipate. Some of those things will be good. Some will be bad. You manage the distribution, not the individual outputs.

Rapid experimentation. You need to run experiments at a pace that traditional PM cycles do not support. A prompt change can be written, evaluated, and deployed in hours. The PM needs to operate at that speed.

User behavior analysis at the conversation level. Instead of analyzing page views and click rates, you analyze conversation transcripts. What did users ask? How did the agent respond? Where did the conversation go well? Where did it fail? This is a different analytical skill than traditional product analytics.

I have hired PMs who were excellent at traditional product management and struggled with AI products, and PMs with less traditional experience who excelled because they had the right instincts for probabilistic systems. The skills transfer somewhat, but the gaps are real.

The AI product manager role is new enough that there is no established playbook. We are writing ours as we go, and the lessons change with every deployment. The best AI PMs I have worked with share one trait: they are comfortable with ambiguity without being passive about it. They accept that the AI will surprise them, and they build systems to ensure the surprises are mostly good.

This is what building AI products feels like. You have more control over the intent than the output. You measure quality in distributions rather than absolutes. You ship continuously rather than periodically. And you learn that the product you are building has a mind of its own, which is simultaneously the best and most challenging thing about the work.

Nowah is an AI travel agent that searches and books real flights and hotels through conversation — no filters, no thirty open tabs. Plan your next trip.