AI vendor evaluation scorecard
Score vendors on what happens with your data and in your workflow, not on their demo. Ask for production references at your scale, run a trial on your own cases with your own pass criteria, check data handling and residency, test integration, model the cost at your expected volume, and know how you'd leave.
Demos aren't evidence
A demo shows the best case on the vendor's data. Public benchmarks may not match your task, and some benchmark data may have been seen in training, so results on your own cases are the evidence that counts.
Scorecard
| Criterion | Question | Score 1 to 5 |
|---|---|---|
| Production proof | Who uses it in production, at what volume, for how long? | |
| Quality on your data | Pass rate on 50 of your real cases, with your pass criteria | |
| Data handling | Where is data processed and stored? Used for training? Retention? | |
| Security and compliance | Certifications, access control, audit logs | |
| Integration | Works with your systems and identity provider? | |
| Cost at your scale | Price at expected volume, plus growth | |
| Agent claims | Real autonomous capability, or a rebranded chatbot? | |
| Model changes | How are model upgrades announced and tested? | |
| Exit | Can you export data and switch? |
Running the trial
1 - Pick 50 real cases, including hard ones, with a domain expert. 2 - Write pass criteria before seeing outputs. 3 - Run each vendor on the same cases. 4 - Have the expert grade blind where possible. 5 - Compare pass rate, failure types and cost per passed case.
Agent washing
Gartner has warned about "agent washing", where existing products such as assistants and chatbots are rebranded as agents without real agentic capability, and estimated that only about 130 of the thousands of vendors claiming agentic AI are genuine. Ask the vendor to show the agent taking a multi-step action in your environment, and what happens when a step fails.
Model changes
A vendor's model upgrade can change your results without any change on your side. Ask how upgrades are announced, whether you can pin a version, and send change notices to a named technical owner who reruns your eval set.
From the floor
When I evaluate a tool or vendor with leaders, the questions that matter are who is using it in production, at what data volume and for how long, because prototypes and proofs of concept are fine, and the step that counts is implementation and scaling to production.
In the AI field report session, I pointed out that public benchmark datasets may have been used to train the models being tested, so high scores on them don't prove much, which is why the trial on this page uses your own cases.