How to evaluate an AI vendor before you buy: a twelve-question checklist

Procurement did not have to think much about AI features until recently; now they are in every product pitch. The problem is that “AI-powered” describes both a genuinely useful capability and a thin wrapper over a public model with a markup. These twelve questions are the ones we ask on behalf of clients, grouped into four areas. Ask them in writing and keep the answers; vague replies are answers too.

Capability: what does it actually do?

1. Which tasks does the AI perform, specifically? “Summarises support tickets and drafts replies” is an answer. “Uses AI to transform your workflow” is not.

2. Which model, and can it change? You are entitled to know whether the product runs its own model or calls a third party, and whether the vendor can swap models without telling you. Both are acceptable; not knowing is not.

3. What is the evidence? Ask for a trial on your data with a defined task and a measurable result. Case studies with named clients and specific numbers count; anonymous percentages do not.

Data: where does ours go?

4. Is our data used to train models? Get the answer in the contract, not the FAQ. The correct answer for business data is no, or opt-in only.

5. Where is data processed and stored? Region, residency options, and the list of sub-processors. Canadian public-sector, health and financial clients usually need this in-country; see our security and compliance page.

6. What is retained, for how long, and can we delete it? Prompts, outputs, uploaded files, logs. A vendor that cannot describe its retention cannot honour a deletion request.

Failure: what happens when it is wrong?

7. How are errors detected and reported? Every model is wrong some of the time. A serious vendor can describe how wrong outputs are caught, flagged and fed back.

8. Where does a human review? For anything that reaches a customer, a ledger or a regulator, there should be an approval step you control. If the product has no such step, you will build one or accept the risk.

9. What is logged? Inputs, outputs, who approved what, and when. You will need this for a complaint, an audit or an insurer, and you cannot add it retroactively.

Commercials: what does it cost, and how do we leave?

10. What does real usage cost? Per seat, per call, per outcome, plus overages. Ask for a price at three times your expected volume; AI features are the part of a bill that grows.

11. What are the rate limits and service levels? AI features often have separate limits from the rest of the product. Find out before a busy day does.

12. How do we export and exit? Data, configurations, prompts, fine-tuned models if any. A product you cannot leave is a product you cannot negotiate with. Our thinking on this is in build versus buy.

A vendor that answers all twelve questions clearly may still be the wrong choice. One that cannot is never the right one.

Use the checklist as a written questionnaire and score the answers. It takes a vendor an hour to answer honestly and it saves you a year of regret. If you want the review run for you, including a technical look under the hood, that is part of our IT and technology consulting work.