Representative test set
Use real, anonymized or synthetic-equivalent cases spanning common work, edge cases, conflicts, missing data, policy boundaries, and known failures. Freeze versions so changes are comparable.
AI quality and observability
How do you evaluate and monitor an AI agent in production?
Define “good” on representative work before launch, inspect how the agent reached each result, and monitor quality, exceptions, reliability, cost, and business outcomes after launch. Evals tell you whether behavior is acceptable; observability helps explain why it happened.
Short answer: build a versioned test set from real cases and edge cases; define pass, fail, and escalation criteria; record full workflow traces; sample production outcomes; capture human corrections; alert on material drift; and connect technical metrics to the operational number the workflow is meant to improve.
Production scorecard
Measure six layers, not one accuracy number.
| Layer | Questions | Example measures |
|---|---|---|
| Task quality | Was the result correct, complete, grounded, and useful? | Acceptance rate, factual error, citation validity, rubric score. |
| Decision safety | Did the agent stay within policy and escalate when required? | Unauthorized-action attempts blocked, escalation recall, approval compliance. |
| Execution reliability | Did every step complete once and leave a usable result? | Completion, timeout, retry, duplicate, recovery, and fallback rates. |
| Human burden | How much review and correction does an accepted result require? | Review minutes, edit distance, override rate, exception queue age. |
| Economics | What did it cost to reach an accepted outcome? | Model and tool cost, attempts, cost per accepted result. |
| Business outcome | Did the workflow improve the intended operating constraint? | Cycle time, capacity, conversion, leakage, service level, risk events. |
Evaluation system
Offline evals and production monitoring answer different questions.
Explicit evaluator
Choose deterministic checks where possible, expert rubrics where judgment is needed, and model-based graders only after calibrating them against human decisions.
Workflow trace
Capture retrieved sources, model and version, instructions, tool calls, intermediate state, latency, policy decisions, approvals, action receipts, and final outcome.
Production sampling
Review a risk-weighted sample of completed work plus every serious exception. Feed corrections and new failure modes back into the test set.
Drift and alerts
Watch changes in inputs, sources, model versions, tools, acceptance, cost, latency, and escalation. Alert on decision-relevant movement, not every fluctuation.
Decision cadence
Name who reviews the scorecard, how often, and which threshold pauses, rolls back, limits, or expands the system.
Avoid false confidence
Common evaluation failures
A polished demo is not a test set. A model score is not a business result. An average can hide a dangerous subgroup. User satisfaction cannot prove factual correctness. Logs are not useful observability unless someone can reconstruct the work and make a decision from them. And a test set decays when the workflow, sources, policies, or customer mix changes.
OpenAI describes built-in evaluation and optimization loops as a way to see what works and improve agents over time; its investment guidance recommends real-task evals, edge cases, an agreed “good enough” threshold, and full cost of acceptance. Google Cloud’s agent observability guidance places agent traces and telemetry within the broader production monitoring system. See OpenAI Frontier, OpenAI’s investment guidance, and Google Cloud agent observability.
Review cadence
Turn monitoring into operating decisions
A dashboard is not a monitoring system until thresholds lead to action. Define which failure pauses the workflow immediately, which trend triggers investigation, which regression rolls back a version, and which sustained improvement permits broader volume or authority. Keep the decision, evidence, owner, and follow-up in the same operating record.
Compare every release against the current production version on the same frozen set, then add newly discovered failures without rewriting historical results. Segment results by risk, customer type, language, source quality, and workflow branch when averages might conceal harm. Review human corrections as evidence: repeated edits can reveal a missing rule, weak context, a bad evaluator, or work that should remain with a person.
Finally, verify the business measure on its own cadence. A technically better agent can still worsen the operation if it creates new review queues, shifts work downstream, or optimizes the wrong unit.
Enterprise and regulated environments
Validation must prove intended use—and reveal unsafe limits.
| Required feature | Evidence produced | Required outcome |
|---|---|---|
| Intended-use validation plan | Approved purpose, population, environment, risk analysis, acceptance criteria, prohibited uses, and accountable sign-off. | The system is validated for the job it performs, not for a generic benchmark. |
| Representative and adverse testing | Normal cases, edge cases, affected groups, missing and conflicting data, prompt attacks, unsafe requests, tool failures, and misuse scenarios. | Known limitations, differential failures, security weaknesses, and escalation needs are measured before scale. |
| Independent review where warranted | Qualified reviewer, separation from the builder, reproducible test package, findings, disposition, and residual-risk acceptance. | High-consequence releases are not approved solely by the team that built them. |
| Production traceability | Versioned model, prompt, policy, source references, tools, approvals, output, action receipt, reviewer, and outcome. | A material result can be reconstructed and attributed after the fact. |
| Drift, subgroup, and control monitoring | Quality, safety, overrides, complaints, latency, cost, unauthorized attempts, source changes, and performance by relevant segment. | Degradation and uneven harm are detected before averages conceal them. |
| Incident and change response | Alert thresholds, stop criteria, rollback, notification, root cause, corrective action, revalidation, and closure evidence. | Material failures lead to controlled response and verified prevention of recurrence. |
A regulated validation package should be proportionate to consequence. It may need privacy, security, accessibility, fairness, safety, records, clinical, financial, or other domain review. The page describes an operating baseline, not a certification or substitute for sector expertise.
The NIST AI RMF applies trustworthiness throughout design, deployment, use, testing, and evaluation. Its Generative AI Profile adds governance, pre-deployment testing, provenance, and incident considerations; ISO/IEC 42001 adds continual management review and improvement.