Representative test set
Use real, anonymized or synthetic-equivalent cases spanning common work, edge cases, conflicts, missing data, policy boundaries, and known failures. Freeze versions so changes are comparable.
AI agent evaluation checks whether an agent completes the task, follows its permissions and leaves the right result in your systems. Use the same cases to compare releases, then monitor accepted results, corrections and failures in production.
By Naman Kumar · Updated November 3, 2026
The implementation test record separates expected behavior from observed passes: a stale edit must preserve newer state; retrying an already-applied request must not create another write; unauthorized access must fail. The supplied excerpts show the assertions. Read the associated case for the business problem and bounded retrieval result.
| Layer | Questions | Example measures |
|---|---|---|
| Task quality | Was the result correct, complete, grounded, and useful? | Acceptance rate, factual error, citation validity, rubric score. |
| Decision safety | Did the agent stay within policy and escalate when required? | Unauthorized-action attempts blocked, escalation recall, approval compliance. |
| Execution reliability | Did every step complete once and leave a usable result? | Completion, timeout, retry, duplicate, recovery, and fallback rates. |
| Human burden | How much review and correction does an accepted result require? | Review minutes, edit distance, override rate, exception queue age. |
| Economics | What did it cost to reach an accepted outcome? | Model and tool cost, attempts, cost per accepted result. |
| Business outcome | Did the workflow improve the intended operating constraint? | Cycle time, capacity, conversion, leakage, service level, risk events. |
Start with one task and its acceptance rules. For every test, save the input, expected result, grader, trace and final system state. Repeat cases when behavior can vary, compare releases on the same cases and add production failures to the next version of the test set. The checks below define what your framework needs to record and who decides whether to release.
Use real, anonymized or synthetic-equivalent cases spanning common work, edge cases, conflicts, missing data, policy boundaries, and known failures. Freeze versions so changes are comparable.
Choose deterministic checks where possible, expert rubrics where judgment is needed, and model-based graders only after calibrating them against human decisions.
Capture retrieved sources, model and version, instructions, tool calls, intermediate state, latency, policy decisions, approvals, action receipts, and final outcome.
Review a risk-weighted sample of completed work plus every serious exception. Feed corrections and new failure modes back into the test set.
Watch changes in inputs, sources, model versions, tools, acceptance, cost, latency, and escalation. Set alerts for changes that require investigation or action.
Name who reviews the scorecard, how often, and which threshold pauses, rolls back, limits, or expands the system.
A polished demo is not a test set. A model score is not a business result. An average can hide a dangerous subgroup. User satisfaction cannot prove factual correctness. Logs are not useful observability unless someone can reconstruct the work and make a decision from them. And a test set decays when the workflow, sources, policies, or customer mix changes.
OpenAI describes built-in evaluation and optimization loops as a way to see what works and improve agents over time; its investment guidance recommends real-task evals, edge cases, an agreed “good enough” threshold, and full cost of acceptance. Google Cloud’s agent observability guidance places agent traces and telemetry within the broader production monitoring system. See OpenAI Frontier, OpenAI’s investment guidance, and Google Cloud agent observability.
Suppose a support agent answers routine requests well, but a release gives it access to the wrong customer’s record. Pause the affected access immediately, preserve the trace and test the fix before restoring it. A strong average acceptance rate does not cancel that failure. If accuracy is stable but review time rises, investigate the changed cases before expanding volume. Those are different decisions and need different alerts.
A dashboard is not a monitoring system until thresholds lead to action. Define which failure pauses the workflow immediately, which trend triggers investigation, which regression rolls back a version, and which sustained improvement permits broader volume or authority. Keep the decision, evidence, owner, and follow-up in the same operating record.
Compare every release against the current production version on the same frozen set, then add newly discovered failures without rewriting historical results. Segment results by risk, customer type, language, source quality, and workflow branch when averages might conceal harm. Review human corrections as evidence: repeated edits can reveal a missing rule, weak context, a bad evaluator, or work that should remain with a person.
Finally, verify the business measure on its own cadence. A technically better agent can still worsen the operation if it creates new review queues, shifts work downstream, or optimizes the wrong unit.
| Required feature | Evidence produced | Required outcome |
|---|---|---|
| Intended-use validation plan | Approved purpose, population, environment, risk analysis, acceptance criteria, prohibited uses, and accountable sign-off. | The system is validated for the job it performs, not for a generic benchmark. |
| Representative and adverse testing | Normal cases, edge cases, affected groups, missing and conflicting data, prompt attacks, unsafe requests, tool failures, and misuse scenarios. | Known limitations, differential failures, security weaknesses, and escalation needs are measured before scale. |
| Independent review where warranted | Qualified reviewer, separation from the builder, reproducible test package, findings, disposition, and residual-risk acceptance. | High-consequence releases are not approved solely by the team that built them. |
| Production traceability | Versioned model, prompt, policy, source references, tools, approvals, output, action receipt, reviewer, and outcome. | A material result can be reconstructed and attributed after the fact. |
| Drift, subgroup, and control monitoring | Quality, safety, overrides, complaints, latency, cost, unauthorized attempts, source changes, and performance by relevant segment. | Degradation and uneven harm are detected before averages conceal them. |
| Incident and change response | Alert thresholds, stop criteria, rollback, notification, root cause, corrective action, revalidation, and closure evidence. | Material failures lead to controlled response and verified prevention of recurrence. |
A regulated validation package should be proportionate to consequence. It may need privacy, security, accessibility, fairness, safety, records, clinical, financial, or other domain review. The page describes an operating baseline, not a certification or substitute for sector expertise.
The NIST AI RMF applies trustworthiness throughout design, deployment, use, testing, and evaluation. Its Generative AI Profile adds governance, pre-deployment testing, provenance, and incident considerations; ISO/IEC 42001 adds continual management review and improvement.
Choose tools for the evidence you need to collect. For an agent built on an LLM, response quality is one part of evaluation. You also need to check retrieved information, tool calls, changes to business records and the final result.
| Your question | What the tool must support |
|---|---|
| Did the agent complete the task correctly? | A versioned set of cases, expected outcomes, repeat runs and deterministic checks or calibrated human rubrics. |
| Why did a run fail? | Traces that connect source records, tool calls, approvals, errors and the resulting state. |
| Can the workflow recover safely? | Controlled interruption, replay and readback tests against the system that stores the result. |
| Did the change improve the operation? | Accepted outcomes, review time, exceptions and operating cost compared over the same period. |
Before choosing a platform, run one known failure through it and check whether another person can reproduce the result. Check export, retention and access controls for traces containing company data. Use the recovery test examples for interrupted or repeated actions, then assign shared tooling and workflow-specific acceptance through the infrastructure guide.
Anthropic’s evaluation guidance explains tasks, trials, graders and outcomes. Apply those distinctions to your own workflow rather than selecting a tool by its dashboard alone.
The agent-coordination case includes a three-size retrieval check, the Python runner and its dated results. It measures returned context and checks that the relevant completed work and evidence remain available.