Grounded AI Answers: The 30-Question Trust Test Before Your Bot Touches a Customer

When the bot is wrong, the business pays

In early 2024, an Air Canada passenger asked the airline's chatbot a simple question about bereavement fares. The bot answered confidently: book at full price now, claim the discount retroactively within 90 days. That policy did not exist. When the passenger asked for the promised refund and was refused, the dispute went to British Columbia's Civil Resolution Tribunal — case 2024 BCCRT 149, decided February 14, 2024. The tribunal ruled the airline liable for negligent misrepresentation and ordered it to pay. Air Canada argued the chatbot was effectively a separate legal entity for which it could not be held responsible; the tribunal rejected that argument outright.

The legal analysis from McCarthy Tétrault notes this may be the first ruling to affirm that principle specifically for information provided by a chatbot: what your bot says is what you said.

That was a chatbot on a website in 2024. The cost of a wrong answer has since gone up in a second, more literal way. Since October 1, 2026, Meta charges businesses on the WhatsApp Business Platform per message for service replies sent inside the 24-hour customer window — replies that had been free since November 2024. The change was announced in a Platform update in July 2026: from October 1, service messages — replies inside the 24-hour customer window — are billed per delivered message, at each market's utility rate, with a monthly free allowance per business number. Customer messages to the business stay free, and ordinary users and the free WhatsApp Business app are unaffected — the bill lands only on businesses running at scale (The Telegraph Nigeria, DEV, Nagmelater).

Wrong answers now cost you twice: once in trust, once in money.

Put those two facts together and the arithmetic of a wrong answer changes. A bot that invents a policy doesn't just risk a tribunal ruling or an angry customer — every confused follow-up earns another bot reply, and bot replies inside the customer window now draw on a per-message meter — past the monthly free allowance, each one is billed. (Customer messages to the business stay free; it's the business's replies that carry the charge.) This post is the discipline that stops both — and the one-afternoon test that proves your bot deserves to talk to customers.

Why smart AI gives wrong answers

The instinct is to blame the AI model, as if a smarter model would fix it. The retrieval-engineering research says the failure happens earlier: at the moment the system finds the information. Anthropic's Contextual Retrieval work showed that when a document is chopped into pieces for search, each piece loses the context that made it findable. Their own example: a chunk reading "The company's revenue grew by 3%" is meaningless without knowing which company and which quarter.

• Prepending a short, automatically written passage that situates each chunk in its document cut retrieval failures by about 35%.
• Combining that with hybrid search (embeddings plus old-fashioned keyword matching) cut failures by about 49%.
• Adding a reranker — a second, more careful scoring pass over the shortlist — cut them by about 67%.

The production pattern the industry converged on is the same shape everywhere, and it has three stages:

• Retrieval — optimizes for recall: cast a wide net with embeddings and pull back dozens of candidate passages — better to fetch too much than to miss the right page.
• Rerank — optimizes for precision: a cross-encoder scores each query–document pair jointly, which is far slower per document than the initial embedding pass, so it runs only on the shortlist. One practitioner summary reports a 50–75 candidate sweet spot, with gains plateauing beyond 100, and a Databricks benchmark of up to 48% retrieval-quality improvement on complex queries (Medium). Anthropic's own tests independently confirm the reranker's leverage: adding it took their failure-rate reduction from 49% to 67% (Anthropic).
• Generation — stays grounded: every claim in the answer is attributable to the retrieved passages, with citations (CustomGPT describes this same three-stage stack).

If this sounds familiar, it should: our post RAG Without the Hype: A Practical Guide to AI Over Your Company Docs covered the mechanics — how retrieval works, including its own evaluation habit with a 100-question engineering test set. This post is deliberately different. That was the mechanics guide for understanding the pipeline. This is the trust and reliability playbook: the grounding discipline your answers follow, the refusal rules, the live-sheet treatment of fast-changing facts — and a pre-launch test designed for a business owner to run in an afternoon, not an engineer to maintain in a pipeline. Read the other post to understand the machinery; read this one to make the answers trustworthy.

The prove-it discipline: four rules every answer must follow

Here is the discipline, in plain words. It fuses the retrieval lessons above with what actually works in small-business deployments — and it fits on an index card you can tape to the monitor of whoever owns the bot.

• Answer only from your documents. The bot's knowledge is the fee schedule, the course catalogue, the refund policy, the FAQ — the documents you uploaded. Not its training data, not its general knowledge. Real deployments that hold up follow this boundary: one WhatsApp agent pattern answers "only from uploaded documents — price lists, FAQs, policies, catalogs — and when a question falls outside them it says it doesn't know and routes to the team" (DEV).
• Every factual claim names its source page. "The weekend batch starts June 6" is not an answer; "The weekend batch starts June 6 (Source: Schedule page, updated 2026-09-15)" is. Citations are what turn a confident sentence into a checkable one. The production pattern ends at generation with citations for exactly this reason — every claim attributable (CustomGPT).
• Volatile facts bypass the index. Fees, batch timings, seat availability — anything that changes week to week — should never be baked into the searchable documents. They get read live, at answer time, from a sheet you control. ChatMaxima, writing from real SMB onboarding conversations, reports this as one of the fixes that actually works: read prices, stock, and timings live from a Google Sheet instead of re-indexing them.
• When in doubt, say so — and offer the human exit. The bot must be constrained to say it is not sure rather than guess, and specifically must never promise refunds, discounts, or delivery dates it cannot verify (ChatMaxima). And every answer carries a visible one-tap exit: "Talk to our team." A refusal that routes to a human is a good answer. A confident guess is a liability.

These four rules — plus a tie-breaker for contradicting documents — go into the bot as one system prompt — the "prove-it" prompt. Copy it, fill in the brackets, paste it into your bot tool's system-instructions field. This is the actual artifact; don't paraphrase it:

You are the customer-support assistant for [BUSINESS NAME], answering on [WhatsApp / the website].

You may answer ONLY from the retrieved documents in <context> and the live sheet in <live_sheet>.

1. Every factual claim must cite its source page by name and date, e.g. "(Source: Fees page, updated 2026-09-15)".
2. Prices, schedules, seat availability, and timings come ONLY from <live_sheet>. Never quote them from a document.
3. If <context> and <live_sheet> together do not contain the answer, say exactly: "I don't have that in our documents — let me connect you to our team." Do not guess. Do not rephrase around the gap.
4. Never promise refunds, discounts, or delivery dates the documents do not state.
5. If two documents contradict each other, follow the most recently dated one and say which one you followed.

One honest limitation: this prompt is only as good as the documents behind it. If your fee page is six months out of date, the bot will cite a wrong fee beautifully. The prompt proves the answer is grounded; the next section proves the grounding is true.

The 30-question trust test

Here is the pilot. It takes one afternoon, needs no engineer, and it must pass before the bot touches a real customer. We'll walk it through with a worked example: imagine a coaching institute — call it Prime Classes (fictional, but the questions are the real kind). Parents message on WhatsApp asking about fees, batch timings, and admission dates. The owner has heard the Air Canada story and does not want to be its sequel.

Step 1 — Collect the 30 questions

• Action: Open a spreadsheet with two columns: Question and Verdict. Ask your front-desk staff (or whoever answers the phone) for the questions customers ask most often. Add questions from your actual chat history until you have 30 rows.
• Expected result: 30 real questions in the sheet. ChatMaxima reports that SMB owners describe their recurring questions as "the same 10–20 questions" — start with those, then add the trickier ones from chat history (the "is there a discount for two siblings?", the "what if we join mid-batch?") to reach 30. For Prime Classes, the sheet fills with fee-per-course, batch timings, installment options, refund policy, and admission deadlines.

Step 2 — Load the documents, connect the live sheet

• Action: Upload the documents the answers must come from: fee schedule, course catalogue, batch timetable, refund policy, admission FAQ. Then create a separate sheet — the live sheet — with the facts that change: current fees, current batch start dates, seats remaining. Connect it as a live lookup in your bot tool, not as an uploaded document.
• Expected result: The bot's knowledge base holds stable documents; volatile facts live in the live sheet and are read at answer time. (This is the fix ChatMaxima reports working in real SMB deployments.)

Step 3 — Install the prove-it prompt

• Action: Paste the prove-it system prompt from the previous section into the bot's system-instructions field, with your business name, channel, and the live sheet connected as <live_sheet>.
• Expected result: Ask the bot "Where do your answers come from?" in a test chat. It should say it answers only from your documents and the live sheet. If it claims general knowledge, the prompt isn't installed correctly.

Step 4 — Run all 30 questions

• Action: In a private test chat (not your customer-facing number), ask each of the 30 questions exactly as a customer would phrase them — typos, shorthand, and all. Paste each answer into the sheet next to its question.
• Expected result: 30 answer rows, each with its cited sources or its clean refusal visible.

Step 5 — Grade every answer with the 3-check rubric

• Action: For each answer, a human — you or a staff member who knows the business — applies three checks. One "no" on any check is a fail.
• Expected result: Every row marked Pass or Fail, with the failing check noted.

The 3-check rubric:

• Grounded? Every factual claim in the answer appears in the retrieved documents or the live sheet. (A claim with no document behind it fails — even if it happens to be true.)
• Answers the question? It actually addresses what was asked — not a related topic, not a brochure paragraph.
• Cites or refuses cleanly? Either every factual claim carries its source page and date, or the bot said it doesn't have the answer and offered the team. Anything in between — a confident uncited claim — fails.

This is the owner-sized version of the evaluation discipline production teams use: systematic answer-quality testing before deployment, using the standard evaluation triad — groundedness (every answer supported by retrieved spans), answer relevance, and context relevance — as the checks. Production RAG practice gates deployment on an evaluation step before anything reaches customers (Medium), and formal deployments build judge-based evaluation into the workflow so quality is tested as the system evolves (Vuono Group). Your version adds the refuse-cleanly rule to the rubric and runs 30 questions on a spreadsheet — same idea, afternoon-sized.

Step 6 — Run the stale-doc audit

• Action: For every failed answer, ask one question first: was the document wrong? Walk this checklist over the knowledge base:
• Every policy and schedule page carries an effective date at the top. "Updated 2026-09-15" beats "latest fees" every time — it lets the bot (and you) follow the newest version when two pages disagree.
• Superseded pages are archived, not left lying around. Last year's fee PDF still in the knowledge base is a wrong answer waiting to happen.
• Each document has one owner and a review date. If nobody owns the refund policy, nobody updates it.
• Fees and timings duplicated in prose are replaced with a pointer. If the brochure PDF and the fee page both state the course fee, delete the number from one and point at the live sheet instead. One source of truth per volatile fact.
• Expected result: A short fix-list of document edits, not model tweaks.

Imagine Prime Classes runs the test and hits the pattern the audit is built for: some failures trace back to the uploaded timetable being last quarter's — the bot cited it correctly, and the citation is exactly what makes the failure catchable. The fix is documents, not the model. When a failure traces to a stale document, the remedy is the cheapest kind there is: edit the page, re-run the failed questions.

Step 7 — Fix, re-test, then connect

• Action: Make the document edits from the audit, re-run only the failed questions, and confirm they pass. Repeat until all 30 pass or refuse cleanly. Only then connect the bot to your customer-facing WhatsApp number.
• Expected result: A 30/30 sheet — every row passing the 3-check rubric — dated and saved. That sheet is your evidence that the bot earned its customers. Re-run the test whenever documents change materially (new fee structure, new batch calendar): a quick re-run with the new questions beats discovering a stale answer from a customer.

After the test: what "passing" buys you

A bot that passes the 30-question test is a bot you can put on WhatsApp without the Air Canada dread. Its answers are cited, so a customer who doubts one can check it. Its volatile facts come from the live sheet, so a fee change propagates without re-indexing anything.

And when it doesn't know, it says so and routes to your team — which, under the new per-message billing, is also the cheaper outcome: a clean refusal plus one human handoff costs fewer billed messages than a wrong answer and the round of bot replies it drags along. Past the monthly free allowance, every bot reply inside the window is a billed message; customer messages stay free.

The trust discipline and the money discipline turned out to be the same discipline.

What to do Monday morning: pick the business's ten most repeated questions (you already know them), put them in a sheet, and upload the five documents that answer them. That is Steps 1 and 2 — the smallest honest start. If you want help setting up the knowledge base, the live-sheet wiring, and the prove-it prompt for your own documents, that's exactly what our LLM integration and RAG work covers — but the test itself needs nothing except an afternoon and an honest grader.