Writing Acceptance Criteria for AI Features: From Vague Ideas to Testable Requirements

A criterion like, the assistant returns a relevant answer when asked about pricing, can pass on the one example the QA engineer tries and fail on the next 47 phrasings of the same question. Traditional acceptance testing assumes a small number of inputs cover the input space. AI inputs are open ended. Single example checks give false confidence and ship features that look fine in demo and break in production.

Tradeoff: Replacing single example checks with evaluation set checks adds setup cost (you need to build and maintain a test set) but produces a defensible quality signal. Teams that skip the evaluation set inevitably ship regressions because the only quality signal is the demo.

Criteria like, the response is high quality, or, the assistant sounds professional, mean different things to different reviewers. Engineering ships what they think meets the bar, the PM sees something different, and sign off becomes a debate. This is even worse for AI because reviewers have different tolerance for variability and different intuitions about what the model can do.

Tradeoff: Quantifying qualitative criteria requires the PM to define rubrics, which takes hours. The shortcut, leaving criteria qualitative, costs more in the long run because every release requires the same argument again. Build the rubric once and reuse it.

Teams write functional acceptance criteria first and remember to add safety criteria after a near miss in staging. By that point, the design choices that determine safety properties are already locked. Acceptance criteria written this way produce features that are functional but unsafe, requiring rework before launch or worse, after launch.

Tradeoff: Writing safety criteria upfront slows the ticket creation step but eliminates the late stage scramble. Use a safety checklist (refusal scenarios, privacy boundaries, regulated content rules) on every AI feature ticket as a forcing function.

Traditional criteria assume the system either succeeds or returns an error. AI systems often produce a confident, plausible, wrong answer. Criteria that do not specify how to handle this case (escalation paths, confidence thresholds, fallback behavior, user disclosures) leave engineering to guess, and different engineers guess differently. The result is inconsistent UX across features and a product that feels uncoordinated to users.

Tradeoff: Specifying fallback behavior for every criterion lengthens tickets by 30 to 50 percent. Skipping it is faster up front but creates UX debt that takes more time to clean up than it would have taken to write.

Describe the behavior in plain language and explicitly state where variation is acceptable. Example, when a user asks for a summary of a meeting transcript, the assistant produces a summary between 80 and 200 words that captures all decisions and at least 80 percent of action items. Length, phrasing, and ordering may vary across runs as long as the content requirements are met. The explicit variability statement prevents reviewers from rejecting outputs that differ from a reference example but still meet the requirement.

Tradeoff: Allowing variability by default makes acceptance review more subjective. Counter this by attaching example outputs that are clearly inside the band and clearly outside, so reviewers calibrate their judgment. Avoid attaching one golden output, which trains reviewers to reject anything that does not match it.

Specify a measurable threshold on a test set the team owns. Example, on the Q2 meeting summary evaluation set (250 transcripts), the feature must achieve a graded quality score of at least 4.0 out of 5 with no individual case below 2.5, and a hallucination rate below 3 percent. Name the evaluation set explicitly so that engineering and the PM agree on what was measured. If the test set does not exist yet, building it is part of the ticket.

Tradeoff: Quantifying the threshold requires investment in the evaluation set. The first criterion you write this way takes a week. The tenth takes an hour because the evaluation set already exists. Teams that skip this step end up shipping features they cannot defend in a postlaunch quality review.

List the categories of input that the feature must refuse, redirect, or escalate. Example, the feature must refuse to summarize content involving HR complaints, legal matters, or content marked confidential by the source system, and must not include any personally identifying information about people not in the meeting. Tie each refusal case to a specific output (a polite decline message, a redirect, or an escalation queue). Safety criteria are not optional and must be tested as part of the evaluation set.

Tradeoff: Listing refusal cases up front takes effort and may surface gaps in your safety taxonomy that the team has not yet decided how to handle. This is the point. Surfacing these gaps in the ticket is far cheaper than discovering them in production.

Specify when the model output must be reviewed by a human or escalated to a different system. Example, if the assistant cannot identify decisions or action items with a confidence score above 0.7, the summary must be flagged for human review and presented with a banner that explains the limitation. Specify the confidence threshold, the trigger condition, and the visible UX state. Without this hook, the model becomes the only line of defense, which is unacceptable for any feature with material consequences.

Tradeoff: Human review hooks add latency (a few seconds for in product flagging, longer for queues) and require operations capacity. The alternative, no hook, transfers all risk to the user and produces incidents that are far more expensive than the operations cost.