The AI Agent Evaluation Framework: How to Test Autonomous Systems Before You Ship
TL;DR
Standard AI evaluation checks single outputs against expected answers. That method does not work for agents, because agents fail across trajectories, not on individual responses. A correct step 1 can still lead to a catastrophic step 4. This framework covers the five dimensions every agentic product needs to evaluate before launch: task completion, trajectory quality, tool use accuracy, error cascade behavior, and scope adherence. Each dimension comes with a practical measurement approach and a pass/fail threshold to put in your launch checklist.
The AI PM Minute
One tactic to make you a sharper AI PM, twice a week. 60 seconds to read. Free.
No fluff. Unsubscribe anytime.
Why Single-Output Evals Fail for Agents
Traditional AI product evaluation works like this: you provide an input, the model produces an output, you compare the output to a known-good reference answer or apply a rubric. This works for chatbots, classifiers, and summarization features because each response is self-contained.
Agents break this model in a specific way: they take sequences of actions, and each action changes the state of the world the agent then acts on. A wrong decision at step 3 does not just produce a bad step-3 output. It produces a corrupted context that makes steps 4, 5, and 6 operate on false premises. The failure is not visible at step 3. It becomes visible at step 6, when the agent has sent an email, modified a record, or called an API based on that corrupted state.
The cascading failure pattern
An agent tasked with processing a refund request correctly identifies the customer at step 1. At step 2, it misreads the order status field and marks the order as eligible when it is not. At steps 3 through 5, it correctly executes the refund workflow, because each step only checks whether the previous action succeeded, not whether the original classification was correct. The output at each individual step looks correct. The end result is a fraudulent refund that the model processed correctly according to its own corrupted internal state. A single-output eval at any one step would not have caught this.
This is why agent evaluation requires trajectory-level testing. You need to evaluate the full sequence of actions the agent took to reach an outcome, not just whether the final output matches an expected answer.
The Five Eval Dimensions for Agentic Systems
An agent evaluation framework needs to cover five distinct dimensions. Missing any one of them creates a blind spot that is likely to surface as a production incident.
1. Task completion rate
What it measures
Does the agent successfully complete the assigned task to a standard that a human reviewer would accept as correct?
How to measure it
Build a task evaluation set of 200 to 400 representative tasks drawn from your target distribution. Include happy path tasks (70%), edge cases (20%), and tasks the agent should decline or escalate (10%). Run the agent on each task in a sandboxed environment. Have a human reviewer rate each completion as: fully correct, partially correct, or failed. Calculate the fully correct rate.
Launch threshold
Set a minimum fully correct rate before launch. For high-stakes tasks (financial, medical, legal), this bar should be 90%+. For lower-stakes tasks (scheduling, search, summarization), 80%+ is a reasonable starting point. The number is context-dependent, but it must be explicit and agreed before you run the eval.
2. Trajectory quality
What it measures
Did the agent reach the correct outcome via a reasonable and efficient path, or did it arrive at the right answer for the wrong reasons?
How to measure it
Log every step the agent takes: tool calls, intermediate reasoning, state reads, and writes. After each eval run, compare the agent's trajectory to a reference trajectory you defined for that task. Score on: number of steps taken relative to reference (efficiency), absence of unnecessary actions (precision), and absence of wrong-but-corrected actions (reliability).
Launch threshold
An agent that produces correct outputs via systematically wrong trajectories is a liability, not an asset. If more than 15% of correct completions use a trajectory that deviates significantly from the reference path, investigate the pattern before launch. These agents are fragile and will fail on slight variations of the trained distribution.
3. Tool use accuracy
What it measures
When the agent calls a tool or API, is it calling the right tool with the right parameters?
How to measure it
Stub out all external tools with mock implementations that record call signatures. For each task in your eval set, capture: which tools were called, in what order, with what parameters, and whether the parameters were valid. Compare against expected tool calls. Measure: correct tool selection rate (did it call the right tool?), correct parameter rate (were the parameters valid and appropriate?), and hallucinated tool call rate (did it call tools that do not exist or that are not available for this task?).
Launch threshold
A hallucinated tool call rate above 2% is a launch blocker. A correct parameter rate below 90% is a launch blocker for most use cases. These numbers will depend on your specific tool set and task complexity, but any hallucinated tool calls in production create audit risk you cannot accept.
4. Error cascade behavior
What it measures
When the agent makes a mistake at an early step, how does it behave in subsequent steps? Does it compound the error, self-correct, or escalate?
How to measure it
Deliberately introduce errors into the agent's environment during eval runs. Provide incorrect context, return malformed data from mocked tools, or give the agent a task that has no valid completion path. Observe: does the agent recognize the error state? Does it escalate to a human? Does it attempt a workaround that makes the situation worse? Does it proceed as if nothing is wrong?
Launch threshold
The agent must not take consequential actions after encountering an error state that suggests it is operating on corrupted context. 'Consequential action' means any write, any external call, any user-facing output. If the agent cannot identify that it is in an error state, it must have a maximum action budget that stops it from running indefinitely on bad state.
5. Scope adherence
What it measures
Does the agent stay within the boundaries of the task it was assigned, or does it take actions outside the intended scope?
How to measure it
Define a scope boundary for each agent: the set of tools it is allowed to use, the data it is allowed to read and write, the external services it is allowed to contact, and the action types it is allowed to take. During eval, test whether the agent ever exceeds these boundaries, especially when given ambiguous tasks that could reasonably be interpreted as requiring out-of-scope actions.
Launch threshold
Any out-of-scope action in the eval set is a launch blocker. This is non-negotiable. Agents that independently expand their own scope in production create liability, audit failures, and trust damage that is very difficult to recover from. Scope adherence must be enforced at the tool permission layer, not just through prompt instruction.
Building Your Eval Set: What to Include
The eval set is the core artifact of your agent evaluation program. Its quality determines the reliability of your signal. Here is how to build one that catches real failures rather than giving you false confidence.
Happy path tasks (70%)
Representative tasks drawn from the most common user requests your agent will handle. These should be solvable and within scope. If the agent cannot pass 90%+ of happy path tasks, it is not ready for any production traffic.
Edge case tasks (15%)
Tasks that are unusual but still within scope. Ambiguous phrasing, missing context, multiple valid completion paths, tasks that span multiple tool calls in non-obvious ways. These are the cases most likely to produce trajectory deviations.
Out-of-scope tasks (5%)
Tasks the agent should decline or escalate rather than attempt. Define what declining looks like: a clear explanation, a human handoff, a scoped alternative. The agent should never attempt out-of-scope tasks by expanding its own permissions.
Error-state tasks (5%)
Tasks where you deliberately inject a problem: a missing required field, a tool that returns an unexpected error, a context that contradicts itself. These test error cascade behavior specifically.
Adversarial tasks (5%)
Tasks designed to probe safety and scope constraints. Requests that ask the agent to exceed its permissions, bypass its instructions, or take actions on behalf of someone other than the assigned user. These test scope adherence under intentional pressure.
Eval set sizing
A minimum viable agent eval set is 200 tasks. Below that, your confidence intervals are too wide to draw reliable conclusions from pass/fail thresholds. For high-stakes agents (financial, medical, legal) start at 400 tasks and plan to expand to 1000+ as you build a feedback loop from production. Add tasks from real production failures as they occur, so your eval set grows more robust over time.
Learn to Ship Agents Safely
The AI PM Masterclass covers agent evaluation design, production monitoring, and safe launch frameworks. Taught live by a Salesforce Sr. Director PM.
Sandboxing: How to Run Evals Without Side Effects
Agents take actions. Running an agent eval against real external systems will send real emails, create real records, make real API calls. You need a sandboxed evaluation environment that produces realistic agent behavior without real-world side effects.
Mock all write operations
Replace any tool that writes to an external system (email, CRM, database, API) with a mock that records what the agent attempted to write, returns a success response, but does not actually execute the write. Your eval infrastructure captures the intended actions for review, not the consequences.
Use realistic read data
The quality of your eval signal depends on the realism of the data the agent reads during eval tasks. Synthetic data that is too clean will not surface the parsing failures and edge case misinterpretations that production data contains. Use anonymized samples from production data wherever possible and compliant with your data governance policy.
Inject controlled errors into the environment
Build your mock infrastructure to support error injection. Mocked tools should be able to return: network errors, malformed responses, rate limit errors, and unexpected data schemas on command. This lets you run your error cascade eval dimension without waiting for these conditions to occur naturally.
Reset state between eval tasks
If your agent reads state that it or a prior agent modified, eval tasks can interfere with each other. Your sandbox must reset to a known state between tasks. In practice this means versioned test fixtures, not a shared database that accumulates changes.
Track everything the agent does
Log every action the agent takes during an eval run, including internal reasoning steps if your model provides them. This trace is your trajectory-quality evaluation artifact. Without it, you cannot diagnose why a task succeeded or failed, only whether it did.
The Agent Eval Launch Checklist
Use this checklist as the minimum bar before any agentic feature goes to production traffic. Add criteria specific to your domain and use case, but do not remove any of these.
Eval infrastructure
- Sandboxed environment with mock write operations is in place
- Trajectory logging captures all agent actions and intermediate state
- State reset between eval tasks is implemented and tested
- Error injection is available for all external tool mocks
Eval set
- Minimum 200 tasks covering all five distribution types
- Task distribution reviewed by domain expert (not just the PM who wrote the spec)
- Out-of-scope and adversarial tasks are included
- At least 20 tasks drawn from real user requests or analogous real scenarios
Eval results
- Task completion rate meets or exceeds your defined threshold
- Trajectory quality score reviewed for the bottom-quartile tasks
- Tool use hallucination rate is below 2%
- All out-of-scope tasks resulted in a decline or escalation, not an attempt
- No error cascade failures resulted in a consequential action without human review
Production readiness
- Human escalation path is defined and tested
- Action audit log is implemented for all write operations
- Agent scope is enforced at the tool permission layer, not just by prompt
- Rollback plan exists for the first 30 days of production operation
Build Agents You Can Ship With Confidence
Evaluation design, sandboxing, and production monitoring are the skills that separate experimental AI projects from production-grade agents. Learn them in the Masterclass.
Related Articles
Before you go: get the AI PM Minute
One tactic to make you a sharper AI PM, twice a week. 60 seconds to read. Free.
No fluff. Unsubscribe anytime.