Assumption Mapping for AI Products: Find and Test What Your Product Actually Depends On
TL;DR
Most AI products fail because the team got an assumption wrong, not because the model was bad or the engineering was sloppy. Standard assumption mapping from Lean and Teresa Torres works for software products. AI products need an extended version that captures four specific assumption types: technical feasibility (the model can do this), data readiness (we have the data to make it work), user behavior (users will use it this way), and threshold validity (the accuracy level we need is achievable). This guide walks through how to build an AI product assumption map, prioritize which assumptions to test first, and turn the map into a concrete experiment roadmap.
The AI PM Minute
One tactic to make you a sharper AI PM, twice a week. 60 seconds to read. Free.
No fluff. Unsubscribe anytime.
Why AI Products Need a Different Kind of Assumption Map
Assumption mapping, popularized by Jeff Patton and applied to product discovery by Teresa Torres, is a deceptively simple practice: before you build, list every belief your product depends on being true, then sort them by importance and current evidence. The ones that are both critical and poorly evidenced are where your product is most likely to fail. Test those first.
For traditional software, the assumptions that kill products tend to cluster around desirability (do users want this?), viability (will they pay?), and usability (can they use it?). These assumptions exist in AI products too. But AI products carry a second tier of assumptions that software products do not, and this second tier is responsible for a disproportionate share of AI product failures.
An AI product that perfectly solves a real user problem still fails if the model cannot produce accurate enough outputs, the training data does not exist or cannot be assembled, users reject outputs because they do not trust them at the right calibration, or the accuracy threshold you set turns out to be unachievable at a cost that makes business sense. Standard assumption maps miss all of this.
The AI product assumption problem in one example
A healthcare startup builds an AI triage tool. User research confirmed demand. Willingness to pay was validated. The model performed well on the benchmark dataset. At launch: clinicians trust the tool enough to review its recommendations but not to act on them without a second check, which doubles their workload instead of cutting it. The fatal assumption was about user trust calibration, not model accuracy. It never appeared on the assumption map because the team was not looking for it.
The Four AI Product Assumption Types
Every AI product sits on a stack of assumptions distributed across four categories. You need to surface assumptions from all four before you can prioritize which to test.
Type 1: Technical Feasibility Assumptions
Beliefs about what the model can actually do. These are the easiest to test quickly with a technical spike, and the most reliably overestimated in early product discovery.
Examples: The model can extract structured data from unstructured clinical notes with acceptable accuracy. The model can reliably identify when it does not know something. Multimodal inputs can be processed within the latency budget. Fine-tuning will improve accuracy on our specific document types.
Risk level: High: teams frequently underestimate the gap between benchmark performance and in-domain performance on real user data.
Type 2: Data Readiness Assumptions
Beliefs about the data the product needs and your ability to assemble it. Data readiness failures kill more AI products than model capability failures, and they take longer to discover.
Examples: We have enough labeled examples to fine-tune effectively. The data we have is representative of real production inputs. We can acquire the additional data we need within the timeline and at the cost we have budgeted. The data does not contain distribution shifts that will degrade performance post-launch.
Risk level: Very high: data problems are expensive to discover late in development and impossible to solve quickly.
Type 3: User Behavior Assumptions
Beliefs about how users will actually interact with AI outputs: how much they will trust them, when they will override them, how they will interpret uncertainty signals, and what they will do when the model is wrong.
Examples: Users will check AI recommendations before acting. Users will report errors when the model is wrong. Users who see confidence scores will calibrate their trust to the score. Users will adopt the AI workflow without a significant change management investment.
Risk level: High and systematically underestimated: user behavior with AI systems is consistently counterintuitive. Over-reliance and under-reliance are both failure modes.
Type 4: Threshold Validity Assumptions
Beliefs about the accuracy level required for the product to deliver its intended value, and your confidence that this level is achievable. This is the assumption type most specific to AI products and the one most commonly left implicit.
Examples: 85% accuracy is sufficient for the use case. The model will maintain accuracy as input distribution shifts in production. Accuracy will not degrade significantly as we scale to more users and more diverse inputs. The cost of achieving 95% accuracy is economically viable given our pricing model.
Risk level: Critical: a product built on the wrong accuracy threshold either under-delivers or is impossible to ship at the economics required.
How to Build Your AI Product Assumption Map
The goal of the assumption mapping session is to surface everything the product depends on being true. You want quantity over quality at this stage. Precision comes in the prioritization step.
Assemble the right room
The session needs at minimum: the product manager, an ML engineer or AI technical lead, and someone who has done user research on the target problem. Without the ML voice, you will miss technical feasibility and data readiness assumptions. Without the user research voice, you will miss behavior assumptions.
Write assumptions as beliefs, not questions
An assumption is a statement you believe is true: 'Users will flag incorrect outputs.' A question is 'Will users flag incorrect outputs?' Questions stay on the whiteboard. Belief statements can be tested. Write every assumption in the affirmative form.
Run one brainstorm per assumption type
Give each assumption type a dedicated 10 minutes. Technical feasibility first, then data readiness, then user behavior, then threshold validity. Focusing on one type at a time produces more assumptions than an open-ended brainstorm. Aim for at least five assumptions per category.
Include the standard product assumptions too
Do not skip desirability, viability, and usability assumptions. AI-specific assumptions sit on top of standard product assumptions, not in place of them. A product that passes all four AI assumption types can still fail because nobody wants it.
Write each assumption on its own card or sticky note
You will be sorting and prioritizing them. If two assumptions are on the same card, you cannot separate them when you need to.
Learn Frameworks That Apply on Day One
The AI PM Masterclass covers discovery, evaluation design, and the mental models that make AI PMs effective. Taught live by a Salesforce Sr. Director PM.
Prioritizing Which Assumptions to Test First
Once you have a full list of assumptions, you prioritize them on two dimensions: importance to the product succeeding, and current evidence strength. Plot each assumption on a two-by-two matrix.
High importance, low evidence
Test immediately
These are your product's kill shots. If any assumption in this quadrant is false, the product either cannot be built or cannot deliver value. These go into your first sprint of testing, before any further design or engineering investment.
High importance, high evidence
Monitor and document
You are confident these are true and they matter. Document the evidence explicitly so you can revisit them if the product evolves. If new information makes you less confident, they move left.
Low importance, low evidence
Deprioritize or drop
Even if these are false, the product can still succeed. Do not spend time testing them until higher-priority assumptions are resolved. Many assumptions in this quadrant can simply be dropped.
Low importance, high evidence
Background track
You know these are probably true and they are not critical. Log them and move on.
A practical heuristic for AI products: data readiness assumptions should almost always be treated as high importance unless you have direct evidence your data exists and is clean. Teams consistently overestimate data readiness and discover the real state of their data only after engineering investment is already sunk.
Threshold validity assumptions are frequently underestimated on importance. The question "what accuracy level do we actually need for this to work?" is one of the most consequential questions in AI product management and one of the least frequently asked explicitly.
Testing Techniques for Each Assumption Type
Different assumption types call for different testing approaches. Using the wrong test type wastes time and generates ambiguous results.
Technical feasibility assumptions
Best test: Technical spikes: build the smallest possible prototype that directly tests the model's ability on your specific inputs. Use your actual data, not benchmark datasets. Measure accuracy on a labeled holdout set before investing in any product wrapper.
Avoid: Relying on published benchmarks. Model performance on benchmark data and model performance on your data are not the same number.
Data readiness assumptions
Best test: Data audit: before any engineering, pull a representative sample of the data you plan to use. Label 100 to 200 examples and assess quality, completeness, and distribution. This takes two to three days and surfaces most data problems before they become engineering problems.
Avoid: Assuming the data that exists is the data you need. The question is not 'do we have data?' but 'do we have the right data in the right volume with the right quality?'
User behavior assumptions
Best test: Wizard of Oz studies: simulate the AI output with a human in the loop and observe real user behavior. This tests how users respond to AI-quality outputs without building the AI. Follow-on interviews to understand the reasoning behind observed behaviors.
Avoid: User surveys about hypothetical AI behavior. What users say they will do with AI outputs and what they actually do are reliably different.
Threshold validity assumptions
Best test: Threshold discovery interviews: show users outputs at different accuracy levels (using staged examples or early model outputs) and observe at what level they find the product useful. Ask directly: 'At what error rate would you stop using this?' The answer is often far more permissive than the team assumed.
Avoid: Setting accuracy thresholds based on gut feel or competitor benchmarks. Your use case has specific cost-of-error economics that determine your threshold.
From Assumption Map to Experiment Roadmap
An assumption map only generates value if it changes what you do next. The output of the mapping session should be an experiment roadmap: an ordered list of tests, one per critical assumption, with the specific question being tested, the method, the timeline, and the decision criteria.
Users will use the AI recommendation without requiring a second verification step
User behavior
Low: no direct observation, assumption based on PM intuition
Wizard of Oz study with 8 to 10 target users, 2-week run
If 7 or more of 10 users act on the recommendation without seeking confirmation, the assumption is validated. If fewer than 5 do, redesign the trust scaffolding before building further.
Product manager and UX researcher
Sprint 2, before any engineering investment in the recommendation engine
Run at most two or three tests in parallel. Testing more than that simultaneously makes it hard to isolate what each result means. The fastest AI product teams run one critical assumption test per sprint and update their map as results come in.
The most important rule: set your decision criteria before you run the test. "We will proceed if we see X" is a concrete standard that prevents the natural tendency to rationalize results that did not go the way you hoped. An assumption map that produces ambiguous tests that get interpreted as validation is worse than no assumption map at all.
Build AI Products That Actually Work
The AI PM Masterclass covers product discovery, evaluation design, and the judgment frameworks that separate AI PMs who ship from those who pilot endlessly. Join the next cohort.
Related Articles
Before you go: get the AI PM Minute
One tactic to make you a sharper AI PM, twice a week. 60 seconds to read. Free.
No fluff. Unsubscribe anytime.