AI Incident Response for Product Managers: The PM Playbook When AI Breaks in Production
TL;DR
AI incidents are qualitatively different from traditional software outages. The model is not down, the API is returning 200s, and the logs look clean, but the product is silently giving thousands of users wrong or harmful outputs. The PM is the person who notices this first, communicates it accurately to leadership and users, coordinates the mitigation, and owns the postmortem that prevents recurrence. This guide covers the AI-specific severity framework, what to do in the first 30 minutes, how to communicate externally, and the postmortem structure that closes the feedback loop back to your eval suite.
The AI PM Minute
One tactic to make you a sharper AI PM, twice a week. 60 seconds to read. Free.
No fluff. Unsubscribe anytime.
What Makes AI Incidents Different From Software Bugs
Traditional software incidents have clear failure signatures: the service is down, the database is returning errors, a deployment broke something. Your monitoring catches it, your on-call engineer gets paged, and the fix is a rollback or a hotfix. AI incidents rarely look like this.
Silent quality degradation
The model drifts toward worse outputs over time, or a new input distribution exposes a capability gap. No alerts fire. Users start leaving, or worse, making decisions based on subtly incorrect outputs. Detection requires ongoing eval monitoring, not infrastructure monitoring.
Stochastic failure
The model gives the right answer 95% of the time and a harmful output 5% of the time. With 100,000 daily users, that is 5,000 harmful outputs per day that no single user report can surface. Rate-based metrics, not individual feedback, are the detection mechanism.
Model provider incidents
OpenAI, Anthropic, or Google changes a model version, adjusts RLHF tuning, or rolls out a behavior change. Your product behavior changes without any code change on your side. Attribution is slow because it looks like an internal issue first.
Adversarial exploitation
A user or organized group discovers a prompt that bypasses your guardrails and produces harmful outputs at scale. The discovery often comes through social media before your own detection systems surface it.
Agentic cascade failures
An AI agent takes an action that triggers a downstream error in a connected system. The original model output may have been valid; the problem is the sequence of actions in an unfamiliar environment. Tracing the root cause requires replaying the full action sequence.
The PM-specific implication: you need to be involved in detection, not just response. AI PMs who only show up after the on-call engineer pages them are one incident late. The PM owns the eval suite and the quality monitoring that catches AI-specific failure modes before they compound.
The AI Incident Severity Framework
Standard P1/P2/P3 frameworks are built around availability. An AI incident that leaves your service fully available but silently harmful needs its own severity classification that maps to business and user impact, not uptime.
SEV-1: Immediate harm at scale
Criteria: Model is generating outputs that actively harm users: dangerous advice, privacy violations, discriminatory content, or materially incorrect outputs in a high-stakes domain (medical, financial, legal). Impact is broad (thousands of users) or severe even at low volume.
Response: Page CAIO and legal immediately. Disable the affected AI feature within minutes, not hours. Prepare external communication within 30 minutes of confirmation.
Examples: Medical AI surfacing dangerous drug interactions. Financial AI giving materially wrong investment data. Customer support AI revealing other users' PII.
SEV-2: Quality degradation affecting core user outcomes
Criteria: Model performance has measurably dropped on the primary task. Users are experiencing incorrect outputs at a rate high enough to affect trust and retention, but outputs are not actively harmful. Root cause is unconfirmed.
Response: Engineering and PM on incident within the hour. Evaluate whether to disable or rate-limit the feature. Prepare internal communication for leadership. External communication within 4 hours if user-facing.
Examples: Code generation model suddenly producing non-compiling code at 3x the normal rate. Summarization model consistently missing key points after a model version update.
SEV-3: Isolated or low-rate quality issues
Criteria: Quality issue is confirmed but affects a narrow input pattern, a small percentage of requests, or a non-critical feature. Users are not being harmed, but the issue is real and requires investigation.
Response: Standard sprint process. PM owns investigation. No external communication required unless the pattern grows.
Examples: Model fails consistently on a specific language or domain that is not your primary use case. Edge case in an optional feature produces confusing but non-harmful output.
The First 30 Minutes: PM Priorities When AI Fails at Scale
The first 30 minutes of an AI incident are where severity gets established, mitigation options get evaluated, and the communication clock starts. The PM's job is not to fix the model. It is to coordinate the people who can, communicate accurately upward, and make the call on user-facing actions.
Minutes 0-5
Confirm it is real
Validate the report. Is this a single user complaint or a signal from your monitoring? Check your quality dashboards, error rate metrics, and the raw outputs being flagged. Do not page leadership on an unconfirmed signal.
Minutes 5-10
Scope the blast radius
How many users are affected? What is the error rate (not the absolute count)? Which feature, model, and input patterns are involved? Is this growing or stable? You need numbers, not impressions, for everything that follows.
Minutes 10-15
Identify the mitigation options
Disable the feature. Rate-limit it. Fall back to a previous model version. Add a prompt-level guardrail. Route affected traffic to a safer but less capable model. Each option has a user experience cost. Get engineering to tell you what is feasible in the next hour.
Minutes 15-20
Make the disable-or-continue call
If outputs are actively harmful (SEV-1), disable immediately regardless of the business cost. If the issue is quality degradation (SEV-2), weigh the user harm against the disruption of disabling. Document the decision and the reasoning. You will be asked about it in the postmortem.
Minutes 20-25
Alert leadership
Write a crisp, factual summary: what happened, how many users are affected, what the current impact is, what mitigation is in progress, and what you need from leadership (decisions, communications resources, legal review). No speculation about root cause.
Minutes 25-30
Set up the incident channel
One Slack channel or equivalent. PM, engineering lead, legal representative, comms lead. Designate an incident commander if the organization has that role. Establish a cadence for updates (every 30 minutes for SEV-1, every hour for SEV-2).
Build the Operational Fluency That Separates Senior AI PMs
The AI PM Masterclass covers incident response, governance, evaluation design, and the full operational layer that keeps AI products healthy in production. Taught live by a Salesforce Sr. Director PM.
External Communication: What to Say and When
The biggest PM mistake in AI incidents is saying too little too late. Users who discover an AI quality problem on their own and find no acknowledgment from the company assume the company either does not know or does not care. Both damage trust more than the incident itself.
Communicate faster than users discover it
For SEV-1 issues, your first external communication should go out within 30 to 60 minutes of confirmation. For SEV-2 issues affecting core features, within 2 to 4 hours. Do not wait for root cause confirmation to acknowledge the issue.
Name what happened without over-explaining
We identified an issue with [feature name] that caused [impact in concrete terms] for some users between [time] and [time]. We have [action taken]. We are investigating root cause and will update at [time]. This is the full external message. Do not speculate about why it happened.
Quantify the impact honestly
Stating how many users were affected ends speculation. Journalists and users will assume the worst-case number if you say nothing. If 2,400 users received incorrect medical information, say 2,400. The specific number anchors the conversation to facts.
Match the channel to the severity
SEV-1 at scale: status page update, in-product banner, and proactive email to affected users. SEV-2 affecting a specific user segment: status page update and in-product message for affected flows. SEV-3: status page update only, or none if the impact is minimal.
Do not over-promise on timeline
If you say root cause will be confirmed in 2 hours and it takes 6, you have created a second trust problem. Give updates at your committed cadence even if the update is 'investigation is ongoing with no new findings yet'.
The PM-Owned Postmortem: Closing the Eval Loop
The postmortem is where AI incident response diverges most sharply from traditional software incident response. For a software outage, the engineering team owns the postmortem because the root cause is almost always technical. For an AI quality incident, the PM owns a significant share of the postmortem because the failure is usually in the eval strategy, the product design, or the monitoring setup, not just the model.
Root cause: why the eval suite missed it
The most important question in an AI postmortem is not why the model produced a bad output. It is why the eval suite did not catch this class of failure before it reached production. Map the failure to a missing test case, an out-of-distribution input pattern, or a monitoring gap. This is the PM's domain.
Contributing factors: product design and guardrails
Did the product design present AI output as authoritative when it should have included uncertainty signals? Were there input validation steps that could have caught the problematic prompt patterns? Were guardrails correctly scoped, or did the incident reveal a guardrail gap? These are product decisions, owned by the PM.
The five whys for AI incidents
Apply the five whys to the eval strategy specifically: Why did the model fail? Because it encountered an input distribution not in training. Why was that input distribution not tested? Because the eval set was built from historical production traffic. Why was novel input not in production traffic? Because the feature was new. Why did we ship without testing novel inputs? Because we did not have a process for adversarial eval before launch. That is the actionable root cause.
Specific, owned action items with dates
Add N test cases covering the failure pattern to the eval suite by [date]. Set up a monitoring alert for [specific quality signal] at [threshold] by [date]. Run adversarial eval on the next feature in this category before launch, with PM sign-off on the eval results. Generic action items ('improve monitoring') do not prevent recurrence.
The postmortem delivery standard
A good AI incident postmortem is shared within 5 business days, not 5 weeks. The delay is rarely justified and always signals that the learnings are not being captured while the details are fresh. Publish it to the whole product and engineering organization, not just the incident team. AI failure patterns generalize across products. Sharing the learning is one of the few ways to prevent every team from rediscovering the same failure independently.
Ship AI Products That Fail Safely and Recover Fast
The AI PM Masterclass covers production operations, eval design, and incident response for AI products. Learn to build AI that you can ship confidently and fix quickly when something goes wrong.
Related Articles
Before you go: get the AI PM Minute
One tactic to make you a sharper AI PM, twice a week. 60 seconds to read. Free.
No fluff. Unsubscribe anytime.