AI Agent Reliability Testing: Why One Success Is Not Enough

The Gap Between "Can Do" and "Will Do Reliably"

The standard playbook for evaluating an AI agent goes something like this: write a test case, run it once, check the output. If the output looks right, ship it. If it doesn't, iterate. This approach works reasonably well for deterministic software. For agentic AI, it is fundamentally misleading.

Agents are probabilistic systems. The same prompt, the same tools, the same starting state can produce different actions and different outcomes on different runs. An agent that handles a customer refund request correctly 7 out of 10 times is not a 70% correct agent. It is an unreliable agent, because the 3 incorrect runs are essentially random and invisible until a real customer experiences one.

Microsoft's ThinkingBox, released in August 2026, is the most rigorous public attempt to measure this gap. The framework runs each task 20 separate times and checks not what the agent said it did, but what actually changed in the back-end database. The gap between "looked successful" and "actually was" turned out to be substantial.

The silent failure problem

Many ThinkingBox failures involved clean agent terminations with no visible errors. The agent completed its workflow, reported success, and exited normally. The database showed something different. From the transcript alone, there was no way to distinguish these runs from genuine successes. Transcript-based evaluation would have labeled them correct.

How ThinkingBox Tests Differently

The core insight in ThinkingBox is deceptively simple: don't ask the agent what it did. Check what actually changed. The benchmark is built around stateful business workflows, tasks where the agent must modify a real database, and the measure of success is the actual state of the database afterward, not the agent's own account of its actions.

This matters because most existing evaluation frameworks rely on one of two approaches: transcript analysis (did the agent say the right things?) or LLM-as-judge (did a second model assess the first model's output as correct?). Both approaches are susceptible to the same failure mode: an agent that sounds plausible but executes incorrectly.

Transcript-based eval

Measures what the agent claimed to do. Misses cases where the agent executed a plausible but wrong action, then described it accurately.

LLM-as-judge

A second model evaluates the first. If the first model's output sounds coherent, the judge tends to approve it even if the underlying action was wrong.

ThinkingBox (state-based eval)

Checks the actual database records. An action that was described correctly but executed incorrectly fails. An action that was described confusingly but executed correctly passes.

ThinkingBox was tested across five business domains: order management, HR operations, financial transactions, customer support ticketing, and inventory management. Each domain included tasks with interdependent state: updating one record required correctly reading another, and partial execution left the database in an inconsistent state that was no better than failure.

507 unique tasks. 12 models tested, including both proprietary frontier models and leading open-weight alternatives. Each task run 20 times. The result is a 6,084-run dataset that gives a statistically meaningful picture of agent reliability under production-like conditions.

The 65% Benchmark: What Current Models Actually Achieve

The headline finding: the best-performing model in ThinkingBox achieved a 65.36% pass@1 rate. Pass@1 measures whether the agent completes the task correctly on its first attempt. In a production environment where each task is a real customer interaction, 65% means roughly 1 in 3 tasks fails on the first try.

For context: the industry standard for a production-ready agentic system targeting routine tasks is typically 85% to 90% autonomous completion. The ThinkingBox benchmark suggests current frontier models hit that bar in simple, single-step tasks, but fall significantly short when tasks require multi-step stateful reasoning across business domains.

65.36%

Best model pass@1

On stateful business workflow tasks across 20 trials

507

Unique tasks tested

Across order management, HR, finance, support, and inventory

20x

Trials per task

Enough trials to measure reliability, not just peak performance

The variance across tasks matters as much as the average. Models approached 100% on tasks that could be completed in under a minute with a single tool call. Performance dropped sharply on tasks requiring four or more sequential steps, or tasks where a later step depended on the output of an earlier one that could itself fail silently.

Task duration is the strongest predictor of failure across all models tested. Agents succeed on nearly all tasks that a human could complete in under four minutes. Success rates fall below 20% on tasks that take more than two hours for a human. The implication: autonomy level should be calibrated to task complexity and duration, not just task type.

Build AI Products That Pass Production Scrutiny

The AI PM Masterclass covers agent evaluation frameworks, reliability engineering, and how to spec agentic products that actually work in production.

Why Stateful Workflows Are Harder Than They Look

A stateful workflow is one where intermediate steps change the world, and later steps depend on the state created by earlier ones. Most business workflows are stateful. Processing a refund requires reading an order, checking return eligibility, issuing the credit, and updating the order status. Each step changes state that the next step depends on.

Three failure modes compound in stateful workflows in ways they don't in single-step tasks:

State dependency failure

The agent reads state correctly but acts on a stale or incorrect version. If step 2 produces a result the agent doesn't verify before running step 3, the error compounds silently. Each subsequent step looks locally correct but is built on a wrong assumption.

Partial completion failure

The agent completes 4 of 5 required steps. The workflow appears to have run. The database is now in an inconsistent state that may be harder to fix than a clean failure: some records were updated, others weren't, and downstream systems have already processed the partial result.

Recovery path failure

When agents encounter unexpected states, they improvise. The improvisation is often plausible-sounding but wrong. Without a defined recovery path, agents in error states tend to produce creative, confident, and incorrect actions rather than stopping and escalating.

ThinkingBox specifically probes these failure modes. The benchmark includes tasks where the correct action requires reading state set by a prior step, tasks where partial completion leaves inconsistent state, and tasks where the right answer is to escalate rather than act. Models performed worst on the last category: they rarely chose to escalate when they should have.

Reliability Engineering for AI PMs: What to Do About It

ThinkingBox is a diagnostic, not a death sentence for agentic products. The 65% figure is a baseline for the full task distribution across all five domains. Reliability on the specific workflow you're deploying may be significantly higher or lower. The point is to measure it, not to assume it.

Four reliability engineering practices that directly address the failure modes ThinkingBox exposes:

State verification checkpoints

After each step that changes state, the agent must read back the result and verify it matches expectations before proceeding. This catches silent failures at the step level before they propagate. Add explicit verification prompts to the agent's system instructions, not as an afterthought in the eval suite.

Task scope limitation

Break long workflows into agent-sized chunks with human checkpoints between them. A 6-step workflow has a compounding failure rate: if each step succeeds 90% of the time, the workflow succeeds only 53% of the time overall. Shorter agent scopes and structured handoffs dramatically improve end-to-end reliability.

Escalation as a first-class action

Design the agent to escalate when it encounters unexpected state, not to improvise. Explicitly list escalation triggers in the agent's instructions and test that it uses them. In ThinkingBox, models rarely escalated when they should have. You have to train and test for this behavior explicitly.

Multi-trial reliability benchmarking

Run your eval suite at least 10 to 20 times per task before shipping, not once. Pass rate on a single run is not a reliability measure. Measure pass@1 (first trial), pass@5 (at least one success in 5 trials), and variance. High variance is a signal of latent failure modes your eval cases may not be surfacing.

At the product specification level, reliability engineering starts before a line of code is written. The system prompt is a product requirement. Escalation triggers are acceptance criteria. The multi-trial eval pass rate is a shipping threshold, not a nice-to-have metric.

The ThinkingBox finding that agents rarely escalate voluntarily suggests a specific spec requirement: escalation conditions must be enumerated explicitly, not left to the agent's judgment. "Escalate when uncertain" is not a sufficient instruction. "Escalate when the order status is not in ['PENDING', 'PROCESSING', 'SHIPPED'] or when the refund amount exceeds $500" is.

The PM takeaway from ThinkingBox

Showing an investor or a stakeholder one successful agent demo is easy. Every model tested in ThinkingBox could produce a convincing successful run. The question is what happens on run 2, 5, and 17. The reliability gap between "can do" and "consistently will do" is where most agentic product launches fail in production. Measure it before you ship it.