Continuous Discovery for AI Products: Running Weekly User Research at Scale

Why Standard Discovery Breaks on AI Products

Teresa Torres's continuous discovery framework is built around a core habit: at least one conversation with a user or customer every week, by the team that is building the product. The goal is not research for its own sake but staying calibrated on the gap between what users are trying to do and what your product helps them actually do. Most PM frameworks assume your product has deterministic behavior: if you click this button, this thing happens. Continuous discovery was designed in that world.

AI products break this assumption in three specific ways:

Users cannot articulate AI failures

When a user encounters a bug in a traditional product, they can often describe it precisely: the button did not work, the page did not load, the form threw an error. When an AI product gives a wrong or unhelpful response, users typically say 'it just felt off' or 'it did not understand me.' They have the experience of the failure but not the vocabulary. Your discovery questions need to be designed for this.

The output varies even with the same input

A user who ran the same query twice and got two different answers will not know whether to attribute that to the product, the model, their phrasing, or some invisible context they cannot see. You need to help them reconstruct the context, not just report the experience. This requires different interview techniques than standard discovery.

Users calibrate to the AI's current capability

After a few sessions with your AI product, users stop trying the things it cannot do. Their feedback reflects what they now believe is possible, not what they originally needed. This is called accommodation, and it hides your product's most important gaps. In continuous discovery, you have to periodically probe the original job rather than just the current usage pattern.

The Weekly Session: What to Ask and How to Listen

The weekly discovery session is 30 to 45 minutes per user. It is a structured but conversational exploration of a specific task the user tried to accomplish. You are not evaluating your feature; you are understanding their actual work. The question is not "what did you think of the AI feature" but "walk me through the last time you tried to do X."

For AI products, two question types do the most work:

Reconstruction questions

Get the exact context of an AI failure

—What were you trying to accomplish before you asked it that?
—What had you already tried before this session?
—What did you do after you got that response?
—If you could replay that moment, what would you have asked instead?

Job clarity questions

Separate the underlying need from the AI interaction

—Before this product existed, how did you do this?
—What would a perfect outcome have looked like, ignoring what the tool can do?
—What decision were you trying to make with that information?
—Who else on your team needs to act on this output?

The goal of reconstruction questions is to arrive at a specific interaction you can pull from your observability logs. If you know the user used the product on Tuesday around 11am and got an answer that felt wrong about contract renewal terms, you can pull the exact trace, see what document was retrieved, what the model received, and what it returned. This connection between the user's experience and the system trace is the most powerful diagnostic tool in AI product discovery.

The Opportunity Solution Tree for AI Products

The Opportunity Solution Tree (OST) is the visual artifact at the center of continuous discovery. It maps the connection between your desired outcome (the thing your product is measured on), the opportunities your users face (unmet needs and pain points), your solution hypotheses, and your experiments. The OST keeps teams honest: every feature should trace back to an opportunity, and every opportunity should trace back to a real user insight.

For AI products, three adaptations make the OST more useful:

Add an 'AI assumption' branch under each solution

Standard OSTs track assumptions about user behavior: will users want this feature, will they pay for it, will they understand it? AI products have an additional category of assumption that is technical: can the model actually do this reliably at the quality level users expect? Add this as a first-class assumption branch. Many AI teams waste discovery cycles validating user interest in a solution that the model cannot reliably deliver.

How to do it: Under each solution, list the model capability assumptions alongside the user behavior assumptions. Rate each by known confidence and unknown risk. The intersection of high user desirability and uncertain model capability is your highest-priority evaluation work.

Use frequency and magnitude to weight opportunities

In standard discovery, you weight opportunities by how many users face them and how much pain they cause. For AI products, add a third dimension: how much of this opportunity would actually be addressed by a better AI capability versus a UX change, a workflow change, or additional context in the retrieval layer.

How to do it: When clustering interview insights on your OST, tag each opportunity with whether better AI capabilities, better AI context, or a non-AI change is the right solution. This prevents over-indexing on model quality when a UX change would address 80% of the problem.

Treat model evaluation results as discovery data

In a standard product, experiments produce behavioral data: users clicked or did not click, converted or did not convert. AI products produce both behavioral data and quality data from your evaluation suite. When a prompt change improves your eval scores on a specific failure type, that is discovery data. It tells you whether your solution hypothesis about the model's capability gap was correct.

How to do it: Include your offline evaluation results in your weekly discovery review alongside user interview insights. The combination of a user who says 'it gives wrong contract dates' and an eval result showing 28% error rate on date extraction from PDFs is a confirmed opportunity with a known magnitude.

Learn the Full AI PM Toolkit

The AI PM Masterclass covers continuous discovery, opportunity solution trees, AI evaluation design, and the full product development workflow for AI features.

Using AI Tools to Scale Discovery Without Losing Judgment

AI tools can now automate large portions of the discovery workflow: scheduling interviews, transcribing sessions, clustering themes, and generating opportunity statements. Teams that use these tools well run more interviews and synthesize faster. Teams that use them poorly generate high-volume, low-fidelity insight summaries that feel comprehensive but miss the specific moments that actually matter.

Interview scheduling and recruiting

Automate it: AI scheduling tools handle calendar coordination, screening surveys, and reminder sequences. Use them freely. The human judgment here is in your screener criteria, not the logistics.

Watch for: None significant.

Transcription and session summaries

Automate it: AI transcription is accurate enough for synthesis. AI session summaries are useful as a starting point but miss tone, hesitation, and the specific moment a user looked confused or excited.

Watch for: Do not substitute AI summaries for watching the session. Set aside 10 minutes to scan the transcript for moments the AI summary did not flag.

Theme clustering across sessions

Automate it: AI clustering tools are effective for finding common phrases and grouping similar quotes. They are weak at identifying which clusters represent genuine unmet needs versus which represent surface complaints.

Watch for: Use AI clustering to generate candidate opportunities, then validate each against the original interview moment before adding it to your OST.

Opportunity statement generation

Automate it: AI can draft 'When [user] is trying to [goal], they struggle with [obstacle] because [root cause]' statements from clusters. These drafts are often syntactically correct but semantically shallow.

Watch for: Edit every AI-generated opportunity statement to include the specific user context that makes it real. Generic opportunities generate generic solutions.

The Minimum Weekly Discovery Rhythm for an AI Product Team

Torres's benchmark is at least one conversation per week, per team. For AI product teams, the rhythm works best when it is integrated with your evaluation cycle rather than running separately. Here is the minimum viable cadence:

Monday

Review the week's AI quality signals. Pull evaluation results, look at error rate by category, check for user-reported issues from the previous week. Identify which failure patterns to focus on in discovery sessions this week.

Tuesday or Wednesday

Run at least one user interview, focused on a task that maps to a known quality gap. Bring a specific interaction from your observability logs to the session if you have user permission.

Thursday

Synthesize the session alongside any evaluation data from the week. Update the OST with new opportunities or evidence for existing ones. Generate one clear question for engineering or model evaluation.

Friday (biweekly)

Team discovery review: share OST updates, discuss whether opportunity priorities have shifted, and decide on experiments for the next sprint. This is where product, engineering, and design align on what you are learning.

The sign that your discovery is working

You know continuous discovery is embedded when engineers start asking "did we hear anything in interviews about X?" before committing to an implementation approach. When the team has opinions about opportunities rather than just requirements, discovery is shaping the product rather than documenting what the product already does.

When to Adjust the Cadence for Your AI Product Stage

The weekly cadence is the benchmark. There are legitimate reasons to adjust it based on your product stage, but each adjustment comes with a tradeoff.

Pre-launch / private beta

Adjust to: Increase to 2 to 3 sessions per week. You do not yet know which opportunities matter most, and early discovery is cheap relative to building the wrong thing. Focus on job clarity over failure analysis.

Tradeoff: High time cost. Acceptable when the alternative is months of building toward an unvalidated assumption.

Post-launch, high growth

Adjust to: Use AI synthesis tools aggressively to keep pace with volume. Add a user panel of 8 to 10 power users you talk to monthly. Do not reduce weekly sessions below 1.

Tradeoff: Risk of over-indexing on power users. Balance panel feedback with support ticket analysis from the broader user base.

Mature product with stable quality

Adjust to: Shift from broad discovery to targeted research on specific opportunity hypotheses. One weekly session focused on a single question rather than open-ended exploration.

Tradeoff: Reduced signal diversity. Offset by quarterly broader discovery sprints to avoid blind spots.

Model capability jump

Adjust to: Restart discovery from first principles immediately after a major model upgrade. User mental models and product possibilities both change. Old opportunities may be solved; new ones emerge.

Tradeoff: Disruptive to delivery cadence. Worth it: teams that skip discovery after model upgrades frequently miss new opportunities and keep building for old limitations.