AI User Testing Script Template: How to Run Usability Tests on AI Features

Why AI Features Need a Different Testing Script

Standard usability testing scripts assume deterministic software: the same input produces the same output every time. AI features break this assumption. The model may produce a confident but incorrect answer. Two participants running the same task may get meaningfully different responses. And users have no frame of reference for evaluating AI quality beyond their gut feeling.

This creates three failure modes that a standard script will not catch:

Trust miscalibration

Users either over-trust the AI (accepting hallucinated outputs without question) or under-trust it (ignoring correct outputs because they look uncertain). Standard UX testing does not probe trust formation.

Expectation gap

Users arrive with expectations shaped by ChatGPT, Alexa, or science fiction. When your AI behaves differently from their mental model, they experience it as a bug even when it is correct behavior. You need to surface these gaps before they become support tickets.

Output evaluation paralysis

When asked 'was that response good?' after an AI task, users often say yes because they cannot tell, not because the output was correct. A testing script designed for AI features must include verification tasks that reveal whether the user is actually evaluating the output or just accepting it.

The script below is structured to surface all three. It takes 45-60 minutes per participant and can be run remotely over video call with screen sharing. You need at least 5 participants to see patterns — 8 is better for AI features where output variation matters.

Pre-Session Setup: What to Get Right Before You Record

Preparation for an AI usability session differs from standard UX testing in two key ways: you need a seeded test environment (consistent starting state), and you need to decide in advance how you will handle output variation across participants.

Seed a consistent test account

Create a test account or environment with pre-populated data that gives every participant the same starting context. AI outputs may still vary, but the inputs should be controlled. Never test on participants' live data — it introduces too many variables and raises privacy issues.

Set temperature / model parameters

If your AI feature allows it, reduce temperature slightly (or pin a seed) during testing to reduce output variance across participants. This is not about hiding the non-determinism permanently — it is about isolating UX issues from output quality issues during initial research.

Prepare three output variants per task

For each core task, generate a good output, a mediocre output, and a subtly incorrect output in advance. You may choose to show a specific variant to specific participants to test how users respond when the AI is wrong. This is a deliberate research method, not deception.

Decide your observation focus

Do not try to observe everything. Pick two or three focal behaviors per session: Where do users pause? Do they read the output or just skim it? Do they take any verification action? Narrow focus produces better notes.

Recruiting note for AI feature testing

Screen for participants who have at least some familiarity with AI tools (have used ChatGPT or similar at least occasionally). Completely AI-naive users surface entirely different issues — important, but a separate research question. For most AI feature tests, you want participants who have formed mental models but are not expert users.

The Core Testing Script

Use this script verbatim for the opening and prompts. The bracketed placeholders are where you adapt for your specific feature. Do not improvise — consistent language across participants is what makes findings comparable.

OPENING (5 minutes)

Read this aloud, exactly as written:

"Thanks for joining today. Before we start, I want to explain what we're doing. We're testing [the product / this feature], not testing you. There are no wrong answers. If something is confusing, that's useful information for us.

As you use the product, I'd like you to think aloud — narrate what you're reading, what you're thinking, what you're deciding. It feels strange at first but it really helps us.

One thing to know about this product: it uses AI, so it may give you different answers than it gave someone else, and it may sometimes be imprecise. That's normal — and it's part of what we're trying to understand today.

Do you have any questions before we begin?"

BASELINE MENTAL MODEL (5 minutes)

Ask before they see the product. Record verbatim answers.

Q1: Before I show you anything, tell me what you'd expect [product name] to do when you [describe the core task in one sentence].

Q2: How confident would you expect to be in the answer it gives you?

Q3: What would you do to check whether the answer was right?

TASK 1: FIRST CONTACT (10 minutes)

Show them the product for the first time. Give the task in natural language, not UI instructions.

Task prompt: "Imagine you need to [describe a realistic user scenario that requires the AI feature]. Please go ahead and do that."

Observe silently. Note exactly when they pause, re-read, or hesitate. After they complete or give up:

Q1: Walk me through what just happened. What were you thinking when you saw the response?

Q2: On a scale of 1 to 5, how much do you trust that response? What makes you say that?

Q3: What, if anything, would you do before acting on what you just saw?

Q4: If you got a different answer tomorrow, how would that make you feel?

TASK 2: EDGE CASE / INCORRECT OUTPUT (10 minutes)

This is the most important task for AI features. Show a subtly incorrect output or an edge case the model handles poorly. Do not tell them it may be wrong.

Task prompt: "Now let's try a slightly different scenario: [describe the edge case task]. Go ahead."

After they interact with the output:

Q1: Does anything about that response give you pause?

Q2: If you were using this in real life, would you take any action based on that? Walk me through what you'd do next.

Q3: Is there anything about this response that seems off to you, or does it look right?

After they finish: "I want to let you know — that response actually contained [describe the error]. How do you feel about that?"

CLOSING (10 minutes)

Q1: If you were describing this to a colleague, what would you say it does and how well does it do it?

Q2: What would need to change about the AI part of this product for you to trust it more?

Q3: Is there a moment in the session where you felt most uncertain about what the AI had given you? Tell me about that.

Q4: If this product made a mistake that affected something important to you, what would you do?

Q5: Is there anything else about the AI part of this experience you want to tell me before we wrap up?

Learn to Run Research That Shapes AI Products

The AI PM Masterclass covers AI-specific user research methods, evaluation design, and the feedback loops that make AI products improve over time.

Handling AI-Specific Testing Challenges

Three challenges come up in every AI usability session that do not appear in standard testing. Here is how to handle each one without breaking the session.

The model gives a wrong or strange answer mid-session

How to handle it: Do not correct the output or explain why it happened. Observe the participant's reaction — that reaction is exactly what you are there to study. If the participant asks whether the answer is right, say: 'We're interested in your natural reaction. What do you think?' If they directly ask you to confirm or deny, say: 'I'll tell you more about that after the session.'

Two participants get very different outputs for the same task

How to handle it: This is expected with AI and is data, not a problem. Document both outputs and the corresponding participant reactions. The interesting question is not which output was better — it is whether each participant was able to evaluate the output they received and act appropriately on it.

The participant only gives surface-level reactions ('it looks fine', 'seems good')

How to handle it: Use the 'five whys' probe: follow every shallow answer with 'what makes you say that?' and 'what specifically looks fine to you?' If a participant says 'I trust it,' ask 'what about the response makes it feel trustworthy?' You are trying to surface the specific cues that drive trust or distrust — vague answers are not useful data.

Synthesizing AI User Testing Findings

Standard affinity mapping works for most usability findings. For AI features, add two additional synthesis layers that are specific to the AI component.

Layer 1: Standard affinity mapping

Group observations by theme: navigation issues, comprehension failures, task completion barriers. This is the same process you would use for any usability test. Do it first before adding AI-specific analysis.

Layer 2: Trust calibration map

Plot each participant's stated trust level (1-5 from your closing questions) against whether the output they received was actually correct. Your goal: most participants should be in the top-right quadrant (high trust, correct output) and the bottom-left quadrant (low trust, incorrect output). Participants in the top-left (high trust, wrong output) represent your highest-risk users — they will act on AI errors without noticing. Participants in the bottom-right (low trust, correct output) represent adoption friction — the AI is performing but users don't believe it.

Layer 3: Mental model gap analysis

Compare the baseline mental model answers (what participants expected before seeing the product) with their actual behavior during tasks. Every mismatch is a design opportunity: either change the product to match the mental model, or change the onboarding to reshape the mental model. Document mismatches by type: capability misunderstanding, output format surprise, trust level mismatch, or verification behavior mismatch.

Layer 4: Prioritize by failure type

Not all AI UX failures are equal. Rank findings by consequence: a failure that causes users to over-trust and act on a wrong output is more urgent than a failure that causes them to verify more than necessary. Map each finding to one of three tiers: safety-critical (user acts on wrong output), adoption-blocking (user abandons correct output), or friction (user completes task but with unnecessary effort).

What to include in your findings readout

The most persuasive AI user testing readout for engineering and design partners includes three things: (1) a short video clip of a participant reacting to the AI feature in a surprising way, (2) the trust calibration map showing any top-left quadrant participants, and (3) two or three verbatim quotes that capture the expectation gap. Data tables are less useful than direct observation evidence — let your teammates feel the session rather than just read the summary.