What evidence should you ask an AI vendor for?
Start with the workflow you are buying. Then ask the vendor to show how data, models, permissions and actions behave in the configuration your company will use.
Review the system you will actually deploy.
Name the business task, users, data, connected systems and permitted actions. Identify the service tier and deployment environment. A vendor may offer several models, retention settings and integration patterns; the review needs the one you will use.
Use an AI vendor security questionnaire to organize answers, evidence and open decisions in one place. Keep the normal security and privacy review, then add the AI-specific questions. The ACLU’s published generative-AI procurement questionnaire, for example, asks about training use, prompt leakage, logging, vendor access and adversarial testing. Those are concrete requests for evidence, not just a policy statement.
Pair every important answer with something inspectable.
| Ask the vendor | Ask to inspect | Decision it supports |
|---|---|---|
| Where does a request go, including tools and subprocessors? | A current data-flow diagram and provider list for the intended configuration. | Which data can be processed in which environment. |
| What is retained, who can access it and can it be used for training? | Applicable contract terms, retention settings, administrative controls and deletion procedure. | Whether the service matches the organization’s data requirements. |
| How are user and source permissions enforced? | An access model and a demonstration using both permitted and restricted records. | Whether deployment can preserve your existing access boundaries. |
| Which actions need approval, and who approves them? | Action scopes, approval identity, exact action previews and execution receipts. | What authority the integration may receive. |
| What has been tested against manipulation and misuse? | Relevant test scope, versions, findings, remediation and retest evidence. | Which risks are addressed and which gaps need work before launch. |
| Can we reconstruct and stop an unwanted action? | Logs with source/action context, export options, access revocation and a recovery demonstration. | Whether your team can investigate, contain and recover. |
| What happens when the model, tool or provider changes? | Change-notification terms, version records, release checks and named incident contacts. | Who owns continuing assurance and when review must repeat. |
Record the date and configuration alongside the evidence. An audit report covering one product boundary and a demonstration of your proposed integration answer different questions. Map each artifact to the claim it supports.
Give a useful answer a defined boundary.
Suppose you are buying an assistant that drafts support replies using customer account records. The first release can prepare drafts from records the operator may access. Sending a reply and issuing a refund stay with authorized people.
Your decision record can name that boundary, the configured data-retention setting, the source-permission test, the reviewer and the owner of any open issue. If the service cannot demonstrate separation between customer records, keep that gap visible and resolve it before granting access to those records.
If a later release adds external sending, revisit the action permissions, approval evidence and failure cases. Use the production launch checklist to test the configured workflow, not just the vendor’s demonstration environment.
Prepare one coherent evidence packet.
Keep an accurate architecture and data-flow diagram, model/provider list, relevant data terms, permission design, test and remediation records, logging example and incident contact ready for review. Show which configuration each artifact describes and when it was last checked.
Assign one owner to maintain the packet as the product changes. Ask the customer which evidence is required for its actual use case, and route contractual or specialist security questions to the appropriate owner. Reuse consistent answers while preserving differences in deployment and customer requirements.
This can make the handoff between product, sales, engineering, security and procurement clearer. Track unresolved review questions and their owners; a completed questionnaire is one stage of the buying process, not the final deployment decision.
Agree who closes the gaps.
A vendor review needs a system boundary, evidence requests, decision owners and a way to resolve findings. The scope expands with sensitive data, write access, multiple integrations, deployment environments and specialist testing requirements. Agree time and fees against those deliverables, including any remediation and retest.
Measured Intuition can connect the review to workflow selection, architecture, implementation and ongoing ownership with your internal team. See how an AI Audit scopes the operating decision, hosting and data boundaries, and implementation leadership.