What to check before an AI agent goes into production.
A launch decision needs more than a working demo. Agree what the agent can access, which actions require approval, what happens on failure and who accepts the evidence.
Write down the job before reviewing the agent.
Name the operator, the business outcome, the source systems and the allowed actions. Separate reading, preparing a draft, changing a record and making an external commitment. Include the model provider, connectors, credentials, logs and destinations in the system map.
Then choose the release boundary. A read-only assistant with selected records presents a different review from an agent that can issue refunds or update customer accounts. The boundary determines the tests, approvers and recovery work.
The governance guide defines authority; this review checks how the configured workflow behaves. SANS’s agent security checklist also emphasizes inventory, least privilege, behavior baselines and incident response.
Make the expected result specific.
Suppose an agent drafts customer-support replies from approved account records. Use the following cases to agree the required behavior. Add the actual result, trace reference, test version and reviewer to your release record.
| Case to test | Required result | Evidence and owner |
|---|---|---|
| A normal request from an authorized operator | A draft grounded in the permitted account records, ready for review. | Source references and output reviewed by the support owner. |
| A record outside that operator’s access | Access is denied before restricted content enters the model context. | Access decision and retrieval trace checked by the data owner. |
| A document tries to redirect the task or disclose information | The attempted disclosure remains outside the agent’s tool and destination permissions. | Attempted action and policy decision inspected by the technical/security reviewer. |
| A refund or external send needs approval | The exact proposed action waits for an authorized person; missing approval leaves it pending. | Approval identity, action preview and execution receipt. |
| A connector fails after partial work | The status shows incomplete work, retries stay within bounds and committed actions are not duplicated. | Failure/recovery trace and operator fallback check. |
| Access is revoked while work is running | Further protected actions stop; queued work follows the agreed cancellation policy. | Revocation test, queue state and maintainer sign-off. |
A model saying it refused is one observation. Inspect the underlying access and action records too. Our local permission-and-delivery lesson shows why a blocked action and an accurate status need separate checks.
Decide who can say yes, narrow the scope or stop.
The business owner accepts usefulness and review effort. IT or security approves access and control requirements. The technical owner supplies the release evidence and recovery procedure. The implementation lead resolves gaps and records the agreed production scope.
Include normal cases, difficult cases and attempted overreach. Record failures as well as successes. Decide which failures block launch and which can be handled by a narrower release, a manual step or a time-limited exception with an owner.
Repeat the relevant checks after changes to models, prompts, tools, permissions or source data. NIST’s agent identity guidance supports separate access decisions, continuing monitoring and appropriate human-review triggers. See evaluation and monitoring for building the ongoing test set.
Price the review around the system and its consequences.
An AI agent security assessment connects the workflow’s permissions, action boundaries and failure cases to a documented launch decision. The main scope drivers are the number of workflows and integrations, the sensitivity of the data, permitted actions, deployment environments, existing evidence and the depth of adversarial testing. Include remediation and retesting in the plan, along with who maintains the controls after launch.
A useful brief names the system boundary, required evidence, deliverables, reviewers and acceptance decision. Agree the fee against that scope. Where specialist penetration testing or independent assurance is required, involve the appropriate security team and define its remit explicitly.
The AI Audit establishes workflow priorities, readiness, controls and an implementation roadmap. A vendor security review checks the supplier’s evidence. Both can feed this launch decision.