An AI agent with write access to your systems of record is not a chatbot procurement decision. It is closer to hiring a contractor and giving them a badge. These twelve questions are the ones that separate a system designed for failure from one that has only been demonstrated succeeding.
The questions are below in full. Answer two fields and the page also shows you what a real answer sounds like, what a rehearsed one sounds like, and the follow-up that tells them apart.
This is the single most common production failure for anything holding credentials, and the answer tells you whether recovery was designed or improvised.
Webhook redelivery is normal, not exceptional. Duplicate sends and duplicate payments are how automation loses trust in week one.
Two employees with the same name, two vendors with the same DBA. Guessing here is how an agent does the right action to the wrong person.
Every vendor says there is a human in the loop. The real question is which gates you are not allowed to turn off.
Any agent that reads inbound content has an attack surface made of that content. This is not theoretical for anything touching a shared mailbox or a public queue.
If it can, then "your session expired, sign in here" is a working attack. If it cannot, that entire class of attack has nothing to act on.
This determines the length of your security review more than any other answer.
Scope enforced by convention drifts. Scope enforced at the network does not.
Everyone has tests. The question is whether the tests break things on purpose.
This separates a system that improves from a system that is described as improving.
A right-to-be-forgotten request is a real operational event, and the storage design decides whether you can honor it.
Ambiguous scope is what turns a working deployment into a slow argument.
For each question: the answer that means they built it, the answer that means they have a slide about it, and the follow-up that separates the two.
This list is the one we built our own product against, which is why we are comfortable handing it to you before you have talked to us. Bring it to every vendor you evaluate, including this one.
Request a demo