Your AI Quoted Last Quarter's Price. Blame the Documents, Not the Model

The answer was confident, cited — and last quarter's

A customer asks your docs bot a service price. It answers instantly, names its document, and gets it wrong. The price was real — last quarter. The new price list sits in the same index, next to the old one. Nothing in its answer betrayed the choice.

When the answer matches an old document, your bot is not hallucinating. It is reading your outdated documents. Once you believe that, the fix stops being "get a smarter model" and becomes "clean up the documents the model is allowed to read."

Swapping models does not remove the old price list from the index. Deleting the old price list — or, where your tool supports it, marking it superseded and excluding superseded documents from search — does, once the deletion reaches the index, which Step 5 verifies: Step 5's retired-copy check asks the bot a question the retired copy used to answer, and confirms it no longer quotes that copy.

Why your docs bot quotes the old version

A docs bot built on retrieval-augmented generation, or RAG, does not learn your documents. As SmartChunks explains, it looks things up every time someone asks: it searches your documents, adds the passages it finds to the request, and the model writes from those passages. SitePoint's guide to keeping agent knowledge current describes the same loop: if the knowledge base is re-indexed regularly, the agent's answers are only as old as the last refresh.

Three consequences follow:

• The model has never seen your own private documents in training. SmartChunks' examples are a refund policy, an onboarding wiki, and a signed PDF contract; a price list is this post's example of the same class. Without retrieval, it must admit it does not know, or guess fluently.
• The model's evidence for an answer is what the search hands it — with one qualification: the model is instructed to write from those passages, and that instruction works most of the time, not always. SmartChunks notes it may still blend in outside knowledge, especially when passages are vague.
• Old and new versions of the same fact look almost identical to meaning-based search. SmartChunks explains that embeddings place similar texts near each other — and without date labels or cleanup, the search has no reason to prefer the current one. TechThrilled's RAG failure guide corroborates the outcome: an old policy left searchable alongside the new one may be retrieved instead of the new one.

So a stale answer is usually a search failure, not a model failure. SmartChunks opens with exactly this failure: a chatbot answers a parental-leave question instantly, cites the handbook, and quotes last year's policy, which still sits next to the new one. "The AI model did nothing wrong. The search step failed."

The five symptoms hiding in your own documents

Look for the five audit categories from the freshness-audit write-up (October 2026), which describes its output as a prioritised fix list. The table below states each category, what it looks like in a small business, and what to do about it. All five categories are the audit write-up's taxonomy; TechThrilled's stale-or-conflicting-data problem corroborates the outdated, conflicting, and duplicated elements of the first three categories — the undated half of the outdated-or-undated category is dev.to-only.

Two warnings from the same write-up:

• The audit never decides which fact is correct. If two pages state different refund windows, it flags the contradiction and asks. Only the business knows its current rule.
• The audit keeps personal data out of its report: it refers to files and lines instead of copying customer names, phone numbers, or account details. Your own audit should do the same.

The Freshness Audit: one sitting, three documents

This follow-along build needs no new software: your price list, refund policy, delivery terms, and the docs bot you use. Work through it in one sitting, on three documents.

Step 1: Gather the three documents

Open the price list your team sends customers, the refund policy your bot quotes, and your delivery terms. Note where each one lives: folder, file name, and date last updated (plus publication date if it differs — Step 3's priority rule turns on date last updated). If a document has no date, write "undated" next to it.

Expected result: three documents, each with a location and date last updated (or "undated"), plus a publication date where it differs.

Step 2: Hunt for the five symptoms

If you use a chat assistant for the hunt, start with this starter prompt — paste only document text you are comfortable sharing with that assistant's provider, and strip personal data first. The prompt asks for flags, not final decisions.

Audit three business documents for freshness. Do not change any document or invent facts.

Documents (paste each under its heading):
1. Price list: [paste]
2. Refund policy: [paste]
3. Delivery terms: [paste]

Flag only what you can see in the text:
- Any fact stated differently elsewhere (quote both versions, name the documents)
- Any document with no date, or a date that looks old
- Any document with no named owner
- Any duplicate copy of the same document
- Any draft mark, TBD, or empty section
- Any question left unanswered that a customer would reasonably ask

Do not decide which version is correct. List the flags.

Copy the checklist below into a note and work through it once per gathered document — and once for each duplicate copy found, before Step 3.

Freshness checklist — one copy per document
Document: [file name and folder]
Date last updated: [date, or UNDATED; plus publication date if it differs]
Owner (recommended extra): [name, or NO OWNER]
- Contradiction? Another document states this fact differently? [yes / no — if yes, name it]
- Outdated? Has this fact changed since this document was written? [yes / no]
- Duplicate? Does another copy of this document exist? [yes / no — if yes, name the folder]
- Unfinished? Does it contain a draft mark, a TBD, or an empty section? [yes / no]
- Gap? A customer question this document does not answer? [yes / no — if yes, write it]
Provisional decision: [keep candidate / retire candidate / rewrite — provisional only]

Expected result: three completed checklists for the gathered documents, each ending in a provisional decision, plus one per duplicate copy found, before Step 3.

Step 3: Pick one current copy per fact and retire the rest

First, set aside every document whose Step 2 decision was "rewrite" — they do not enter this step yet. Rewrite them, re-run the checklist, and only then bring them here; the termination paragraph below states how that loop ends.

Next, before you retire or edit anything, write one priority rule down as the opening part of this step: when two documents disagree, which source wins. A usable default, to adapt to your business, is: "When two documents disagree, the newest copy by date last updated wins; an undated copy cannot win until it is reviewed and dated." This rule uses date last updated (not publication date), not the Step 4 Current marking — Step 4 marks the winner Current afterwards, so there is no circularity.

The default has three explicit fallbacks, all recorded in the table's rule column: if the owner knows the newest copy is wrong, the owner's confirmed fact overrides the default; if both copies bear the same date, the owner picks the winner; and if both copies are undated, follow two branches, never both at once. Branch 1, contradictory facts: at most one copy can be confirmed correct — if the owner says both are correct, that is itself a contradiction to resolve first, so date neither until the owner picks the true one; then confirm the true one and retire the other without dating it.

Branch 2, non-contradictory copies: for each copy, the owner confirms whether its fact is still correct, and only a confirmed copy gets the review date as its date last updated — an unconfirmed copy is retired, not dated; the dating step is recorded as an owner decision. Once both copies are dated under Branch 2 (or the unconfirmed one retired), judge by the newest-dated rule; a tie there falls under the same-date fallback above.

No undated copy therefore enters Step 4 as a Current winner — an uncontested UNKNOWN document enters Step 4 as On hold, not as Current, per Step 4's exception, which is a different entry from winning — and no contested fact with a confirmed copy is left without a Current copy for want of a date — the both-copies-retired-unconfirmed case is the separate exception stated in the expected result below, where no confirmed copy exists at all, and a later check failure returning a document to On hold is a pipeline gap, not a dating gap.

Phase order matters: decide every fact's winner and record it first; only then copy unique facts, edit losing passages, and update receiving documents' dates. Do not interleave deciding and copying, or a copied fact's confirmation date will distort later decisions. Confirmation and dating belong to the deciding phase: confirm Case A unique facts and date Branch 2 copies while deciding, before any copying or editing; the dates written then are the inputs the copying phase uses.

Only then work fact by fact — the working set is the three gathered documents plus any duplicate copies the Step 2 checklist found, each with its own Step 2 checklist, even though Step 1 gathered only three documents. Step 2's decision was per document and provisional, and a document can contain several facts, so Step 3's decisions are finer-grained. "Pick one copy per fact" is a per-fact instruction, not a per-document one, and it splits into two distinct cases — do not confuse them.

Case A, a document that wins no fact and holds duplicated facts plus unique facts (a no-win document with no unique facts is simply retired whole): before copying, confirm each unique fact with its owner and write the confirmation date as that fact's date last updated — record that date alongside the fact's row in the retire-or-keep table during deciding, and recompute the receiving document's date as the latest among its facts only in the copying phase, after every winner is recorded — a unique fact has no competitor for the winner rules to vet, so an unconfirmed unique fact must not be laundered into a Current document. While it is an open item, the retiring file is still retired and the fact has no Current copy until confirmed. Do not wait for open items before retiring: retire the old file once its confirmed facts are copied; and if the owner confirms a unique fact is wrong, discard it rather than copying it. Then copy its confirmed unique facts into the kept current copy (or a new current document if no kept copy holds them), recording each fact's receiving document in the retire-or-keep table, so it is identifiable later as the interim copy, and only then retire the old file as a whole.

Case B, a document that wins at least one fact but loses another: it is kept for its winning fact, so it is not retired whole — instead, delete its losing passages before Step 4, and after every kept document's edits in this step are done, do the repair pass last: read the whole kept document and search the other kept documents for cross-references to the deleted passage, and repair any heading, cross-reference, or empty section left pointing at it — repair means removing the now-empty heading or dangling cross-reference, not writing filler, which would recreate the unfinished symptom Step 2 flags. Do not replace a deleted losing passage with the winning fact, since the winning fact already lives in the winning copy and replacing it would create a second current copy of the same fact.

Retirement is always whole-copy, and unique facts survive because they were copied first. Copying unique facts out first is this post's own rule, not TechThrilled's: TechThrilled prescribes removing superseded content and defining a priority rule. As this post's rule for termination: a document gets at most two rewrites in total; if it still fails after its second rewrite, copy any unique facts out first, subject to the same confirmation rule as any retirement, then retire it to the archive folder the bot does not index (or, where your tool supports it, mark it superseded and exclude it from search) until its owner can finish it — when finished, it re-enters at Step 2 while still unsearchable, where the checklist is the gate that decides whether it proceeds to Step 3 and Step 4; it stays unsearchable through Steps 2 and 3, and at Step 4, moving it back from the archive folder (or removing its Superseded marking, for the marking mode) and labelling it Current together return it to searchable status — there is no separate earlier un-marking step, and it is not searchable before Step 4.

At that Step 3, reconcile the interim copy made at retirement — look up that fact's row in the retire-or-keep table, whose Current-copy column names the interim copy for that fact (look up per fact, not per document): if the finished original wins, delete that fact from the interim copy (Case B edit); if the interim copy wins, delete that fact's passage from the re-entered document — never let both end up Current for the same fact. The Case B repair pass applies to these re-entry deletions too, and if deleting the fact empties the interim copy entirely, retire that copy whole, since it now holds nothing unique. Symmetrically, if deleting the fact's passage empties the re-entered document entirely, retire that document whole instead of keeping an empty document. And the cap still terminates: if the re-entered document fails again, it is retired indefinitely until its owner replaces it with a new document — not another rewrite of the old one; and before that indefinite retirement, copy any unique facts out first, subject to the same confirmation rule. It does not enter Step 4 before then, and its retired copy still meets the expected result below.

Record each decision in a retire-or-keep table, one row per fact. The last column is the audit trail, not a remaining conflict. Update the table whenever an interim copy is later edited or retired, to keep its Current-copy column accurate for re-entry.

Expected result: one current copy per fact, a written priority rule with any owner override or tie-break recorded, and every retired copy somewhere the bot cannot reach. The one-copy rule has three stated exceptions, in which a fact has no document labelled Current yet. Exception 1: a fact whose only copy is UNKNOWN-dated (On hold until dated). Exception 2: a fact whose undated copies were both retired unconfirmed (no winner until a confirmed copy exists). Exception 3: a Case A unique fact still open and unconfirmed while its retiring file is already retired.

How Exceptions 2 and 3 exit: retrieve the archived file (or, in Superseded-marking mode, work from an extract while the original stays marked superseded and unsearchable — do not un-mark it at retrieval, and do not un-mark it at Step 4 either: the original stays superseded and only the new document is labelled Current at Step 4 (un-marking at Step 4 belongs only to rewrite re-entry, where the original itself returns)), and bring only the fact to be confirmed through Steps 2, 3, then 4 as a new current copy — by creating a new document for that fact (the Case A new-document option), not by attaching it to an existing kept copy, so the Step 2 checklist fields and the Step 4 label fields describe that new document — the whole retired file does not return, so duplicated facts already copied to Current copies are not reintroduced. The retrieved copy stays unsearchable through Steps 2 and 3. In archive mode, the retrieved file stays in — or returns to — the archive at Step 4; only the new document is labelled Current there, and that labelling returns the fact to searchable status (re-index, or wait for the next refresh, noting which). For Exception 2, pick one of the two retired copies to work from; that picked copy stays retired too once its fact is copied into the new document, and the other stays retired. Otherwise, confirmed unique facts are copied first, and losing passages deleted.

Step 4: Label every kept document

Give each kept document a label carrying the sourced metadata fields (TechThrilled: title, location, product/department, dates, version, permissions; SmartChunks: title, section, date, department/access level), plus owner as this post's own recommended extra. Where your tool supports it, mark kept documents as current in the Status field below.

One exception, stated plainly: a kept document whose last-updated date is still UNKNOWN is not marked Current — it is marked On hold until dated, so the current-only filter does not surface it as current. That holds for uncontested single-copy facts too: a fact with no dated copy has no Current copy until its document is dated, and Step 4 is not complete for that document until it is dated or explicitly left On hold with that gap noted.

To exit On hold, in this order: have the owner confirm the fact; write the confirmation date as its date last updated; label it Current; re-index (or wait for your tool's next refresh, noting which one you did). This exit covers uncontested documents directly, and contested documents once Step 3's both-undated procedure has dated or retired their copies. It also covers a dated winner returned from Step 5 for a pipeline fix: re-running the exit keeps that winner's original confirmation date — do not re-date it — and verifies it by labelling it Current, re-indexing, and asking the bot.

Current here means the Status label; Step 5's check then verifies it like any other Current document, and a failed check is a pipeline gap to fix. If the check fails, fix the named pipeline gap before retrying — do not simply re-run the exit — and until fixed, the document returns to On hold, now meaning needs a pipeline fix as well as (or instead of) needs dating. To verify the exit itself, ask the bot the question that fact answers: pass means the bot shows the confirmed fact. That exit check only asks about the already-confirmed fact; Step 5's change check is different, since it changes a fact and verifies the change.

Document label
Title: [document title]
Source location: [folder or URL]
Section: [section, if the document has one]
Product / category / department: [product, category, or department]
Publication date: [date first published, if known]
Date last updated: [date, or UNKNOWN if not known — an UNKNOWN document cannot win under the newest-dated rule until it is dated]
Version: [version number, e.g. v2 — not the word Current; Current lives in Status]
Status: [Current / On hold — needs dating, or needs a pipeline fix — Superseded is a Step 3 retirement marking, not a Step 4 label (the archive folder is a location, not a status); mark kept documents Current, except UNKNOWN-dated ones, which are On hold until dated]
Who may see it: [who is allowed to see this document]
Owner: [name — this post's recommended extra field]

Expected result: every kept document carries a title, source location, section (if it has one), product / category / department, publication date (if known), last-updated date (or UNKNOWN), version number, a Status of Current — or On hold for UNKNOWN-dated documents until dated, or for documents returned after a failed check until their pipeline gap is fixed — who may see it, and an owner. Terminology note: 'current copy' in Step 3 means the Step 3 winner; 'Current' in Step 4 means the Status label.

Step 5: Make one real change and check it reaches the bot

Change one real fact in one kept document marked Current — not an On-hold document, whose change would not surface under a current-only filter and would falsely read as a pipeline gap; if every kept document is On hold, Step 5 cannot run yet, so date at least one document first via the Step 4 On-hold exit (a contested document in that set is a dated winner returned for a pipeline fix — fix the gap first, then use the exit's verification, not the both-undated procedure, whose undated pair no longer exists), then run Step 5's change check on it — a delivery window, a refund detail — then re-index (or wait for your tool's next refresh, and note which one you did), then ask the bot the question that fact answers. If you retired copies in Step 3, also ask the bot a question the retired copy used to answer (its answer now lives in the Current copy, or nowhere). Pass criterion: the bot does not quote the retired copy — it either cites the Current copy or says the documents do not cover the question. If the answer still shows the old fact, you have found a pipeline gap: the change did not reach the index. TechThrilled prescribes exactly the change-reaches-index test.

Expected result: the bot's answer shows the new fact, or you have a named pipeline gap to fix before you trust the bot again. The same holds for a failed retired-copy check: if the bot still quotes a retired copy, the retirement has not reached the index — fix that gap before trusting the bot again.

Step 6: Build the test set and re-run it after every change

Write down the questions your team asks, with the correct answer and the page each one lives on. Include:

• A question with an exact code, product, or service name
• A question whose answer needs an exception clause, not just the headline rule
• One or two questions whose honest answer is "the documents don't say"

That test-set shape is SmartChunks' recommendation, framed there as a pre-commitment evaluation. The re-run rule is TechThrilled's: maintain regression tests and re-run known failure cases whenever you change the model, index, prompts, or retrieval configuration. This post applies that rule to document changes, since a document change becomes an index change once the update pipeline propagates it — a reasoned application, not a verbatim trigger TechThrilled lists. Run the set after every document change. That differs from this site's pre-launch trust test, Grounded AI Answers: the test is a habit. And keep a dated log of failures over time — TechThrilled's advice is to track failures and add important production failures to the regression test set.

Expected result: a written test set, a first run recorded with a date, and a dated failure log started.

The routine that keeps it fresh after the audit

The audit is one sitting. The routine stops it returning. Its five parts are TechThrilled's prescription; SmartChunks corroborates the date-label and superseded-removal parts, and SitePoint the re-indexing element in the update-pipeline row.

No routine here claims a specific tool does any of this automatically. Before you buy or build, ask — do not assume.

Five questions to ask before you buy or build

These five questions adapt the docs-bot explainer's pre-purchase framing — not claims that any product offers these features:

• What happens to old versions? Look for automatic replacement, date labels, or superseded marking.
• Can I see exactly which passages each answer used? Without this, you cannot run the next section's diagnosis.
• How does it handle scanned PDFs, tables, and slide decks? Ask it to ingest your three ugliest files during a trial, not its samples.
• Does retrieval respect permissions? Per SmartChunks, access control must apply at the search step — not only in the final answer.
• Which model writes the answers, and can I change it? Models improve; lock-in is a cost you will feel later.

When the answer is still wrong, look at what it was shown

Every bad RAG answer comes from one of two places, and the fixes are completely different — so diagnose first, then verify after the fix, following SmartChunks and TechThrilled.

1. Look at the passages behind the answer. If the correct passage is not among them, you have a search problem: fix the documents, chunking, labels, or search. Swapping models will not help: no model can quote a page it never received.
2. If the correct passage is there and the answer is still wrong, you have a model or prompt problem. Tighter instructions pay off here: answer from the supplied evidence, and say when it does not contain enough information.
3. Run the original failed question and a broader regression set after the fix. Add the failure to the test set.

A docs bot that shows its sources beats a confident one that will not. For consequential answers, the citation is the product.

The bot that answers from today's price list is not smarter — its documents were dated, de-duplicated, and kept current. The site's LLM Integration & RAG page describes the service, and the general introduction is RAG Without the Hype.