AI Incident Management: How to Respond When Your AI Product Fails

The AI Incident Taxonomy

Quality regression

AI output quality deteriorates after a model update, prompt change, or infrastructure change. Often silent — no error messages, no alerts — just gradually declining quality scores and rising negative feedback. Detection requires continuous quality monitoring, not just uptime monitoring. Severity depends on the magnitude of degradation and the criticality of the affected use case.

Safety incident

AI produces harmful, offensive, illegal, or policy-violating content. May be triggered by adversarial prompting, a guardrail failure, or unexpected model behavior on edge cases. Severity ranges from minor policy violations to critical safety failures. High-severity safety incidents can escalate to media coverage and legal exposure within hours.

Hallucination / factual error incident

AI produces confident but false information that causes user harm — wrong medical information, incorrect legal advice, false factual claims. Particularly damaging in high-stakes domains. Users may not detect the error immediately, meaning the harm can compound before the incident is discovered.

Availability and latency incident

Traditional infrastructure incidents — API downtime, elevated latency, rate limiting — affect AI products just as they do traditional software, but user impact is often higher because AI interactions are synchronous and latency-sensitive. A 10-second timeout in a chat interface is more damaging than a 10-second timeout in a background batch job.

Detection: How to Know Before Users Tell You

The goal of AI incident detection is to identify quality or safety failures before users experience significant harm — ideally before they're even aware of the problem. This requires monitoring infrastructure beyond standard error rates and uptime.

Continuous quality scoring

Run a sample of production outputs through automated quality evaluation (LLM-as-judge, CLIP scores, or custom evaluators) on a continuous basis. Alert when quality scores drop below threshold. This is your primary detection mechanism for quality regressions.

Thumbs-down and negative feedback rate

A spike in thumbs-down ratings, low stars, or 'not helpful' responses is a leading indicator of quality problems. Monitor this metric in near-real-time with anomaly detection. A 50% increase in negative feedback rate should trigger investigation, not wait for end-of-week reporting.

Safety filter trigger rate

If your safety filters are suddenly triggering at 3x their normal rate, something has changed — either a new attack pattern is spreading, a model update changed behavior, or an upstream data change is affecting inputs. Monitor filter trigger rates as an anomaly signal, not just for compliance.

Support ticket volume and content

Classify support tickets by topic and monitor for spikes in AI-quality-related tickets. A sudden increase in 'wrong answer' or 'inappropriate response' tickets often precedes a formal incident report. Build AI-topic classification into your support intake.

Response Playbook by Incident Type

Quality regression response

1) Confirm regression with a representative sample of outputs. 2) Identify the change that caused it (model update, prompt change, data change) using your deployment log. 3) Roll back if the cause is identifiable and reversible. 4) If rollback isn't possible, activate mitigations (narrow scope, increase human review, add disclaimers) while working on a fix. 5) Communicate to affected users proactively once you have a clear picture of scope.

Safety incident response

1) Page incident commander and safety lead immediately. 2) Assess scope: how many users were exposed, what was the nature of the harmful content, is it still being generated. 3) Activate kill switch or feature flag to stop ongoing harm. 4) Document the specific input/output chain that triggered the incident. 5) Notify legal and leadership within 1 hour of a critical safety incident. 6) Prepare user communication and status page update.

Hallucination incident response

1) Identify the scope of affected queries (is this one query or a pattern?). 2) If pattern: identify the failure mode in your evaluation framework and add a test case. 3) Contact directly impacted users if the hallucination could have caused real-world harm. 4) Assess whether disclaimers or human review steps need to be added to the affected use case. 5) Post-incident: update your evaluation suite to catch this class of error going forward.

Build Production-Ready AI Skills in the Masterclass

Incident management, quality frameworks, and AI product operations are part of the AI PM Masterclass. Taught by a Salesforce Sr. Director PM.

AI Incident Communication

Communicate before you're asked

Proactive communication about an AI quality issue — before users flood your support channel — is the single best trust-preserving move available. 'We identified a quality issue affecting X type of responses — we've fixed it, here's what happened, and here's what we're doing to prevent recurrence' builds trust. Silence followed by a social media pile-on destroys it.

Be specific about scope

Vague incident communications ('some users may have been affected') are more anxiety-inducing than specific ones. Tell users: what specifically went wrong, which use cases were affected, what time period, what they should do with outputs generated during that period. Specificity is reassuring even when the news is bad.

Don't over-explain the AI

Users don't need a technical explanation of why the model failed. They need to know: what happened, what it means for them, and what you're doing about it. Technical explanations in incident communications often come across as blame-shifting to the AI rather than taking accountability for the product.

Post-incident follow-through

The communication after an incident that builds the most trust: a follow-up that says what changes you made to prevent recurrence. This closes the loop and demonstrates that the incident wasn't just managed but actually fixed. Teams that close this loop consistently build a reputation for reliability that survives individual incidents.

AI Incident Management Readiness Checklist

Detection infrastructure

Continuous quality scoring with alerting thresholds configured. Negative feedback rate monitored with anomaly detection. Safety filter trigger rate tracked. Support ticket AI-quality classification in place. On-call rotation covers AI quality incidents, not just infrastructure.

Response playbook

Written playbook for each incident type (quality regression, safety, hallucination, availability). Incident severity levels defined with clear escalation criteria. Roles and responsibilities assigned (incident commander, communications lead, engineering lead). Kill switches and feature flags available for rapid incident containment.

Post-incident process

Blameless post-mortem template for every P0/P1 AI incident. Root cause analysis process that examines model changes, data changes, and prompt changes. Evaluation suite update required for every incident — each incident adds a test case. Follow-up communication to affected users within 48 hours of resolution.