AI Agent KPI Dashboard Template: The Metrics That Matter When Agents Run in Production

Why Standard Product Metrics Break for Agents

Most AI teams launch their first agentic product, hook it up to their existing analytics dashboard, and immediately find that the data does not tell them what they need to know. There are three reasons standard metrics fail for agents:

Agents operate across multi-step chains, not single turns

A chatbot session metric captures one exchange. An agent task completion captures a chain of tool calls, sub-tasks, and handoffs that may span minutes or hours. A session that ends before agent task completion is not a 'completed session' the way a chatbot session is. You need task-level instrumentation, not session-level.

Failure modes are distributed across the execution graph

A chatbot fails by giving a wrong answer. An agent can fail at any node: wrong tool selection, API error not handled, context window exceeded mid-task, hallucinated input to a downstream tool, incorrect final output. Each of these appears differently in logs. Aggregated error rate tells you something went wrong, not where.

Human escalation is a first-class outcome, not a fallback

For agents running in production, the rate at which the agent correctly recognizes it cannot complete a task and escalates to a human is a positive quality metric, not a failure rate. An agent that never escalates is more dangerous than one with a 15% escalation rate that is always appropriate.

Tier 1: Operational Metrics (Real-Time Dashboard)

Operational metrics tell you whether your agent is functioning. These belong on a live dashboard reviewed daily. Anomalies in these metrics require immediate investigation.

Task Completion Rate (TCR)

TCR = (Tasks completed autonomously / Tasks started) x 100

Target: Target band: 70% to 90% for general purpose agents. Below 70% means the agent is failing or over-escalating. Above 90% may indicate under-escalation — verify quality alongside TCR.

Cadence: Daily review. Alert threshold: TCR drops more than 5 points in 24 hours.

Tool Error Rate (TER)

TER = (Tool call failures / Total tool calls) x 100

Target: Target band: under 3%. Breakdown by tool is required. A 2% average TER masking a 15% error rate on one tool (often an external API) is an active incident, not a healthy metric.

Cadence: Daily review with tool-level drill-down. Alert threshold: any single tool exceeds 5% in a rolling hour.

Median Task Duration (MTD)

MTD = Median time from task start to task close (completion, escalation, or timeout)

Target: Calibrate to your SLA, then track deviation. A 20% increase in MTD with no corresponding increase in task complexity indicates a degrading tool or model performance issue.

Cadence: Daily review. P95 latency matters more than median for user-facing agents.

Cost Per Task (CPT)

CPT = (Inference tokens + Tool API calls + Storage) / Tasks completed

Target: Target varies by product. For an agent that replaces a $15 human support ticket, a $0.40 CPT is healthy. Trending CPT upward without corresponding value increase is a margin problem to address before it becomes a business model problem.

Cadence: Daily review, weekly trend analysis. Alert threshold: CPT increases more than 15% week over week.

Learn to Measure AI Products That Actually Work

The AI PM Masterclass covers AI metrics, agent design, and production evaluation in depth. Taught live by a Salesforce Sr. Director PM.

Tier 2: Quality Metrics (Weekly Review)

Quality metrics tell you whether the agent is completing tasks correctly, not just completing them. These require sampling and human evaluation. Automating quality measurement entirely with LLM-as-judge is fine for directional signals, but at least 5% of your sample should involve human review for calibration.

Output Accuracy Rate (OAR)

OAR = (Tasks with correct output / Tasks sampled and reviewed) x 100

Target: Target band: 90%+ for high-stakes agents (financial, medical, legal). 80%+ for general productivity agents. Use stratified sampling: review a higher proportion of escalated tasks than completed ones, since escalations are higher-signal for failure mode analysis.

Cadence: Weekly sample review. Sample size: minimum 100 tasks or 10% of weekly volume, whichever is larger.

Appropriate Escalation Rate (AER)

AER = (Escalations rated as appropriate by reviewers / Total escalations sampled) x 100

Target: Target: 80%+ of escalations rated appropriate. An AER below 80% means the agent is escalating cases it should handle, either due to poor confidence calibration or overly conservative guardrails. Track the inverse: inappropriate completion rate, where the agent completed tasks a reviewer would have escalated.

Cadence: Weekly. Segment by task type: customer support, scheduling, data retrieval, and content generation often have very different AER baselines.

Context Retention Score (CRS)

CRS = (Tasks where agent correctly referenced prior context when required / Tasks requiring prior context) x 100

Target: Target: 85%+. Context retention failures are among the hardest to catch because they produce outputs that look correct but are based on stale or missing state. This metric requires specific test cases: tasks where prior context changes the correct answer.

Cadence: Weekly, with specific regression testing after every model update or memory layer change.

Hallucination Rate (HR)

HR = (Tasks containing at least one factual fabrication / Tasks reviewed) x 100

Target: Target: under 2% for domain-specific agents with grounded retrieval. Under 5% for general agents without RAG. Segment by task type: open-ended generative tasks have structurally higher HR than retrieval tasks. Reporting a single HR across all task types misleads the review.

Cadence: Weekly, automated with LLM-as-judge plus monthly human calibration of the judge itself.

Tier 3: Business Metrics (Monthly Review)

Business metrics connect agent performance to the outcomes the company actually cares about. These belong in monthly reviews and board-level reporting. They lag operational and quality metrics, so by the time a business metric degrades, you should already have caught the underlying issue in Tier 1 or Tier 2.

Human Effort Displacement Rate (HEDR)

HEDR = (Human hours saved by agent / Human hours previously required for same workload) x 100

Target: Target varies by use case. A customer support agent replacing tier-1 tickets should target 60%+ displacement. An agent augmenting human decision-making (rather than replacing it) may target 30%. Never report HEDR without also reporting output quality: 60% displacement at 70% accuracy is not a win.

Cadence: Monthly. Compare against baseline established before agent deployment.

Cost Per Outcome vs Human Baseline (CPO ratio)

CPO ratio = Agent CPO / Human CPO. Where CPO = Total cost / Outcomes achieved

Target: Target: CPO ratio below 0.4 (agent costs 40% of human for equivalent outcomes). A ratio above 0.7 means the agent is not yet economically justified unless it provides non-cost benefits like speed or 24/7 availability.

Cadence: Monthly. Include all costs in agent CPO: inference, monitoring, human oversight, error remediation.

User Trust Score (UTS)

UTS = Weighted average of: (1) % of agent outputs users act on without modification, (2) % of users who report confidence in agent outputs in periodic surveys, (3) return usage rate among users who had one agent-assisted task

Target: Target: overall UTS above 70. A UTS below 50 means users are overriding or ignoring the agent more than using it. This often precedes abandonment even when completion metrics look healthy.

Cadence: Monthly. Segment by user cohort: new users have structurally lower UTS than experienced users.

Governance Incident Rate (GIR)

GIR = (Agent actions flagged as policy violations or requiring rollback / Total agent actions) x 100

Target: Target: under 0.1%. Even a 0.5% GIR at scale means hundreds of incidents per day. Track GIR separately from operational error rate: a tool API failure is an operational error. An agent that sends an unauthorized email or modifies a record it should not have touched is a governance incident.

Cadence: Monthly trend, real-time alerting for individual incidents above a severity threshold.

The Weekly Agent Health Review: A Meeting Template

Most teams reviewing agent dashboards spend 40 minutes on data that does not lead to decisions. This 30-minute weekly format prioritizes action over reporting:

0 to 5 min: Traffic light status

Each of the 4 Tier 1 metrics gets a red, yellow, or green status. Red means an alert fired this week. Yellow means a metric is within range but trending in the wrong direction. Green means no action needed. No discussion here: just status.

5 to 15 min: Dig into red and yellow metrics only

Pull the root cause for any non-green Tier 1 metric. Decision: is this a known transient issue (external API outage, model update), a new failure mode that needs a fix sprint, or a systemic issue that needs architecture review? Assign owner and resolution date.

15 to 25 min: Quality sample review

Two team members independently reviewed 25 task samples each this week. Report: how many disagreed on accuracy rating, how many escalations were rated appropriate, top 2 failure mode categories. One action item: the most common failure mode is added to the evaluation test suite if not already there.

25 to 30 min: One business metric update

Rotate through the 4 Tier 3 metrics on a monthly basis: one metric gets a brief update each week. Prevents monthly business reviews from being the first time leadership hears about a trend that has been visible in data for 4 weeks.

How to Set Up the Dashboard: Instrumentation Checklist

The template is only useful if your agent is instrumented to produce the raw data. Before you build the dashboard, verify these logging requirements are in place:

Task-level lifecycle events

Log: task_started, tool_called (with tool name, input hash, success/fail, latency), task_completed (with outcome type: autonomous, escalated, timed out), and task_reviewed (with reviewer rating if sampled). Each event needs a task_id, timestamp, and session_id.

Token and cost tracking per task

Your inference provider charges per request. But you need cost per task, not per request. If one task makes 8 tool calls and 3 model calls, those 11 events need a shared task_id to aggregate cost. Implement this at launch, not as a retrofit.

Context snapshot on escalation

When the agent escalates, log the full context state: messages in memory, tool calls made so far, reason for escalation if the agent produces one. This is the single most valuable debugging artifact for quality review.

Model version tag on every completion

When your provider updates their model, your metrics may shift. Without a model_version tag on every completion log, you cannot separate model update effects from product changes. Tag every event with the model ID returned by the API.

The one metric most teams miss at launch

Downstream outcome tracking. Most agent dashboards measure what the agent did. Few measure what happened after. For a customer support agent: did the ticket reopen? For a scheduling agent: did the meeting actually happen? For a coding agent: did the PR pass review? These downstream outcomes are the real accuracy signal, but they require integrating your agent logs with your outcome systems. Build that integration in your first sprint, not when stakeholders start asking "but is it actually working?"