AI Feature Health Scorecard Template
TL;DR
Traditional feature health checks ask: is it deployed, are users clicking it, is it crashing? Those questions miss the most dangerous AI failure modes: model drift, rising hallucination rates, trust degradation, and cost creep that compounds silently for weeks. This template gives you a repeatable monthly review across four dimensions of AI feature health: model, product, business, and operations. Each dimension has specific signals, amber and red thresholds, and decision prompts for the PM owning the feature.
The AI PM Minute
One tactic to make you a sharper AI PM, twice a week. 60 seconds to read. Free.
No fluff. Unsubscribe anytime.
Why Traditional Health Checks Fail AI Features
A traditional software feature is mostly stable after launch. Bugs are deterministic: a broken button breaks for everyone, every time. The health check question is essentially: is it up, is it fast, are there crashes?
AI features break in different ways. The model can degrade silently as the world it was trained on diverges from the world it is now being applied to. A sentiment classifier trained before a major company scandal may start misclassifying emotional tone. A recommendation model trained in one season may perform poorly in another. A summary generator may start producing longer, more verbose outputs after an upstream model update, creating a downstream UX problem that the crash rate will never catch.
Model drift without product signal
The model accuracy degrades but users do not complain because they do not know what a good output looks like. By the time NPS drops, the damage is months in.
Hallucination rate increase after upstream model change
Your API provider quietly updates the model. The outputs are different enough to shift hallucination rates in your use case, but nothing in your standard monitoring catches it.
Cost spike from inference changes
A new model version generates 40 percent more tokens per call. Your per-request cost jumps. Usage stays flat, so revenue and engagement metrics look fine. The unit economics are quietly breaking.
Trust degradation from one high-visibility failure
A single visible error by an AI feature can cause users to stop trusting it entirely even if overall accuracy is unchanged. Quantitative health metrics miss this qualitative shift.
The health scorecard catches these failure modes before they surface as customer escalations. Monthly cadence is appropriate for most AI features. Features with high stakes (medical, financial, legal) or high traffic warrant weekly reviews.
The Four Dimensions of AI Feature Health
The scorecard covers four dimensions, each targeting a distinct failure mode. They are designed to be answered by different team members and combined into a single health status for executive reporting.
1. Model Health
Owner: ML Engineer
Is the model performing as expected given the current input distribution? Measures accuracy on a held-out eval set, hallucination rate, output length drift, and confidence calibration.
Catches silent degradation before it affects users at scale.
2. Product Health
Owner: Product Manager
Are users finding and using the AI feature? Are they succeeding when they do? Measures activation rate, feature retention, error recovery rate, and qualitative satisfaction signals.
Catches adoption and UX problems that model metrics will never surface.
3. Business Health
Owner: Product Manager + Finance
Is the feature delivering value at acceptable cost? Measures cost per AI call, cost per successful task, revenue influence, and the trend in each metric over time.
Catches unit economics deterioration before it becomes a P&L problem.
4. Operations Health
Owner: ML Engineer + SRE
Is the feature reliable enough to not become a support burden? Measures p95 latency, uptime, incident count, on-call escalations, and time to detect and resolve issues.
Catches operational debt that accumulates when teams optimize for features over infrastructure.
The Full Scorecard Template
Copy this structure into your team's monthly review doc. Fill in the current value and status for each metric. Combine individual statuses into a dimension-level status. Escalate any red dimension before the meeting ends.
| Metric | Amber threshold | Red threshold | Data source |
|---|---|---|---|
| Eval set accuracy (vs. baseline) | More than 3 percentage points below launch baseline | More than 7 percentage points below launch baseline or below minimum acceptable threshold | Offline eval pipeline run against held-out labeled set |
| Hallucination rate (factual errors per 100 outputs) | 20 to 50 percent increase from launch baseline | More than 50 percent increase or any increase in a high-stakes use case (medical, legal, financial) | AI-assisted annotation of sampled outputs against ground truth |
| Output length drift (average tokens per response) | More than 25 percent change from baseline in either direction | More than 50 percent change or UX-breaking output lengths observed in production | Production inference logs aggregated over 7-day rolling average |
| Confidence calibration (predicted confidence vs. actual accuracy) | Model is 15 to 30 percent overconfident on out-of-distribution inputs | Model is more than 30 percent overconfident or showing systematic confidence collapse | Calibration curve analysis from eval pipeline |
| Metric | Amber threshold | Red threshold | Data source |
|---|---|---|---|
| Feature activation rate (users who trigger the AI feature at least once) | 10 to 25 percent decline month over month among eligible users | More than 25 percent decline or below minimum viable adoption threshold for the feature | Product analytics (Mixpanel, Amplitude, or equivalent) |
| Feature retention (users who return to the AI feature in subsequent sessions) | Single-session use above 40 percent (most users try once and do not return) | Single-session use above 60 percent or declining week-over-week in the most recent month | Product analytics: feature session frequency per user cohort |
| AI task success rate (user achieves their goal without abandoning or overriding) | Task success below 70 percent | Task success below 50 percent or declining by more than 15 percentage points from the previous month | Task completion events or user-defined success signals in your analytics |
| User satisfaction on AI outputs (thumbs up/down, CSAT, or error report rate) | Negative feedback rate above 15 percent or 2x baseline | Negative feedback rate above 25 percent or trending up 3 consecutive months | In-product feedback mechanism (thumbs up/down inline on AI outputs) |
| Metric | Amber threshold | Red threshold | Data source |
|---|---|---|---|
| Cost per AI call (inference cost per model invocation) | More than 20 percent increase from launch baseline without corresponding value increase | More than 40 percent increase or cost structure incompatible with current pricing | Cloud inference billing (AWS Bedrock, Azure AI, GCP Vertex, or API provider invoices) |
| Cost per successful task (inference cost divided by successful completions) | Rising due to accuracy decline rather than usage growth | More than 2x the business case cost-per-task assumption or growing faster than revenue | Combine inference logs with task success events |
| Revenue or retention influence (estimated lift from AI feature presence) | No measurable lift detected in cohort analysis | Negative correlation detected: users with AI feature active have lower retention or conversion than control | A/B holdout analysis or cohort comparison between feature users and non-users |
| Metric | Amber threshold | Red threshold | Data source |
|---|---|---|---|
| p95 inference latency | More than 30 percent above target SLA or noticeably affecting user experience | More than 50 percent above target SLA or causing user abandonment (confirmed by analytics) | APM tool (Datadog, Grafana, or equivalent) with AI request tagging |
| Feature uptime | 99.0 to 99.5 percent (degraded but acceptable for lower-stakes features) | Below 99.0 percent or any downtime affecting a committed SLA to a customer | Uptime monitoring with AI feature-specific checks, not just infrastructure checks |
| On-call escalations related to AI feature | 3 to 5 escalations in the past month with no root cause identified | More than 5 escalations or any escalation without a documented root cause and remediation timeline | Incident management system (PagerDuty, OpsGenie, or equivalent) |
Learn the Full AI PM Toolkit
The AI PM Masterclass covers metrics frameworks, evaluation design, and production operations for AI products, taught live by a Salesforce Sr. Director PM.
Running the Monthly Health Review
The scorecard is only as useful as the process around it. Here is a lightweight process that keeps the review from becoming either a missed calendar item or a two-hour slog.
Week before: data prep (owner: PM)
- •Pull all scorecard metrics from their source systems and fill in current values.
- •Flag any metrics where data is unavailable. Those gaps are themselves health signals.
- •Assign preliminary green, amber, or red status to each metric based on thresholds.
- •Identify any significant changes from last month that need context before the meeting.
Review meeting (30 to 45 minutes, 4 to 6 attendees)
- •PM walks through amber and red items only. Green items are acknowledged and not discussed.
- •ML engineer provides technical context for any model health items in amber or red.
- •Team agrees on root cause for each non-green item. Do not leave the meeting without a hypothesis.
- •Team agrees on the response: investigate further, remediate now, or accept and monitor.
After: action log (owner: PM)
- •Capture each non-green item, its root cause, the agreed response, and the owner.
- •Set a 2-week check-in for any items where the response is 'investigate further.'
- •Add remediation items to the sprint backlog if they require engineering work.
- •Share the scorecard summary with stakeholders as a one-paragraph status update.
Common Red Flag Patterns and What They Signal
When multiple metrics change together, the pattern is more diagnostic than any single metric in isolation.
Model drift up + product satisfaction stable
Likely diagnosis: Your users do not know what a good output looks like, so they are not noticing the degradation. This is common in summarization, drafting, and classification features where quality is subjective. Do not interpret stable satisfaction as a sign the model is fine.
Recommended action: Investigate the root cause of drift (upstream data change, model update, input distribution shift). Improve your eval set to catch quality changes before they reach users.
Activation rate drops + model health stable
Likely diagnosis: This is a discovery or UX problem, not a model problem. Users are not finding or trying the feature. Common causes: UI placement changed, competing feature launched, onboarding flow skipped the feature, or the trigger moment is not where users are.
Recommended action: Run usability sessions to find the discovery gap. Check whether a recent product change moved or buried the feature entry point.
Cost per call up + usage flat
Likely diagnosis: An upstream model update changed token efficiency or pricing. Your per-request cost is higher with the same number of requests. The P&L impact is real even though your traffic looks fine.
Recommended action: Identify the specific driver (more output tokens, higher input token count, price increase from provider). Evaluate whether caching, prompt compression, or a cheaper model variant can restore unit economics.
On-call escalations up + uptime stable
Likely diagnosis: The feature is technically available but producing outputs that require human intervention to correct. The incidents are not causing outages, but they are creating support burden and trust risk.
Recommended action: Categorize the escalations by type. If they cluster around specific input patterns, add input validation or graceful degradation for those cases. If they are random, investigate whether the model has regressed on the affected output types.
All four dimensions amber at the same time
Likely diagnosis: A significant change happened to the system and its effects are cascading. Common causes: major model provider update, significant traffic spike from a new channel, or a product change that altered the input distribution the model was trained on.
Recommended action: Prioritize root cause identification over remediation. Many teams make this worse by shipping fixes to individual dimensions before understanding the common cause.
Build the Operational Discipline That Separates Good AI PMs
The AI PM Masterclass covers evaluation, monitoring, and production operations for AI features, so you ship products that stay healthy after launch.
Related Articles
Before you go: get the AI PM Minute
One tactic to make you a sharper AI PM, twice a week. 60 seconds to read. Free.
No fluff. Unsubscribe anytime.