A vendor demo is optimized to show you the tool at its best. The questions that actually predict whether it'll work for you are the ones that probe what happens outside the demo: when content is stale, when the AI doesn't know, when your usage spikes, when you want to leave.
Ask these directly, in the sales call, and note which ones get a specific answer versus a reassuring one.
Direct answer
Before buying an AI support tool, ask: does every answer show a visible citation to its source, and is the AI restricted to answering only from approved content (grounded) or can it generate more freely; how does the content it's grounded in stay current, and who's responsible for updating it; what exactly triggers an escalation to a human, and does the human get full conversation context; what's the actual pricing model and what does it cost at your real monthly volume, not the advertised starting price; and what access does it need to your systems (help desk, codebase, customer data), scoped how, and revocable how. A vendor that gives specific, verifiable answers to all five is a stronger candidate than one that answers confidently but vaguely.
The five categories of question, and why each matters
| Category | Ask this | Why it matters |
|---|---|---|
| Grounding | Does every answer cite its source, and is generation restricted to approved content? | Determines whether wrong answers are traceable or just opaque model output |
| Content freshness | How does the source content stay current as our product changes? | A grounded AI is only as accurate as content someone keeps updating |
| Escalation | What exactly triggers a handoff to a human, and what context transfers? | A cold handoff makes customers repeat themselves, undermining the automation |
| Pricing | What's the real monthly cost at our expected volume, not the starting price? | Per-resolution and seat-based models can flip which is cheaper depending on usage |
| Access and data | What systems does it need access to, how scoped, and how do we revoke it? | Determines your actual exposure and how easily you can leave later |
Follow-up questions worth pushing on
- If they claim a low hallucination rate: ask whether that's measured against invented answers, or against factually correct answers. The two are different, and a low invention rate doesn't mean high accuracy if the content is stale.
- If pricing is usage-based: ask exactly what counts as a billable event. Some vendors count any conversation with no follow-up question as a resolution, which can inflate your bill faster than expected.
- If they request broad system access: ask for the specific permission scope, not a description of what the product does. "We connect to GitHub" is not the same as a specific, revocable permission list.
- If they say content updates automatically: ask who reviews those updates before they go live, and what happens if the update is wrong.
Frequently asked questions
What's the single most revealing question to ask an AI support vendor?
How does the content the AI answers from stay current? Most vendors have a good answer for grounding and citations; far fewer have a real answer for freshness, and that's usually where accuracy actually breaks down in practice.
Should I ask about hallucination rate specifically?
You can, but push further: ask whether that rate measures invented answers or overall answer accuracy. A vendor can have a low rate of the former and still give customers wrong answers often if the grounding content is out of date.
Is it reasonable to ask a vendor for their exact GitHub or system permissions?
Yes, and a vendor with a well-scoped integration should be able to give you a specific permission list, not just a general description. Vague answers to a specific access question are a signal worth weighing.
How do I evaluate pricing when the model is usage-based?
Ask for the exact definition of a billable event, then estimate your real monthly volume against it before comparing to a flat-fee competitor. Headline per-unit prices are misleading without that context.
Conclusion
- Ask about grounding and citations, content freshness, escalation design, real-volume pricing, and access scope, in that order.
- Push past reassuring answers to specific, verifiable ones, especially on content freshness and access permissions.
- A low hallucination rate doesn't guarantee accurate answers if the underlying content is stale.
- DocsKoala answers each of these directly: cited answers, human-approved content updates, clean escalation, and scoped, revocable GitHub access.