Your AI Worker Changed Overnight — and Nobody Touched Your Code

The change came from upstream, not from you

Picture a shop owner whose enquiry bot has answered price, stock, and delivery timing questions well for months. One Monday, the answers get thinner. A regular customer is told an item is out of stock when it is on the shelf. Another gets a delivery window the shop never offered. Nobody on the shop's side changed anything — no new prompt, no new documents, no new settings. The bot simply started behaving differently, and the shop heard about it from a customer, not from any alert.

This is a real failure class, and it has a cause most owners never suspect: the company supplying the AI changed something on its side. Your AI worker is not a machine you bought and installed. It is a service running on someone else's computers, and the part that answers your customers can move without your permission. This post explains how that happens, why your usual dashboards will not warn you, and the two small habits that catch it early: pinning the version, and running a weekly canary set — twenty questions to start, fewer if that is all your shop has — that you already know the answers to.

The clearest public example comes from Anthropic's April 2026 postmortem, as reported. Over about six weeks, developers using the company's coding assistant noticed its answers getting worse, while the underlying model weights never changed. The cause, per the postmortem coverage, was three product-layer changes made on the vendor's side during March–April 2026, resolved in April 2026. The reports differ on exact dates, mechanisms, and the final fix version, which is itself the lesson: even the people watching closely could not agree afterwards on exactly what moved, because the moving parts were on the vendor's side of the wall. (Quality Clouds, Claude AI News, Septim Labs)

The same risk applies to the enquiry bot in your shop, in three ordinary forms. (Dev.to, Dev.to)

None of this requires a villain. Providers improve their products constantly; a change that helps most customers can quietly hurt your specific setup. The point is structural: when the intelligence lives on someone else's servers, "we changed nothing" is not a defence. Something changed — just not on your side.

• Your prompt stayed the same, but the model reading it moved.
• Your documents stayed the same, but the retrieval behind them moved.
• Your settings stayed the same, but the harness around the model moved.

Why your dashboards stay green while your AI worker drifts

The natural reaction is: surely something would alert us. Usually, nothing does. A swapped or changed model still returns fast, successful responses within token budget, so latency, HTTP status, and error rates can all stay green while answer quality drops. (Dev.to) The failure is in the shape of the output, not in the infrastructure, and infrastructure dashboards measure infrastructure. Coverage of the Claude episode makes the same point about status pages: the status page stayed green while output quality degraded. (Septim Labs)

The same blindness applies to document search. If the embedding model behind your bot's document search updates, new question vectors no longer match the vectors stored in your index, and answer quality degrades while latency, throughput, and error-rate dashboards stay green. The bot still answers confidently. It just starts pulling the wrong paragraph. (Bigyan, Tianpan)

Green dashboards prove the system is running. They do not prove the answers are still right. Only checking the answers themselves proves that.

Two habits that catch AI drift early

Practitioners name two mitigations, and both are small routines, not big projects. The first is version pinning: use an explicit model version instead of an alias wherever your vendor offers one, and if your vendor does not offer it, ask — pinning is a question to put to a vendor, not a feature to assume. The second is a scheduled eval run: run your test set against your production prompts on a weekly schedule, precisely because providers change behaviour without you changing any code, and treat a question that fails two weeks running against a still-correct key as your signal that something moved. (CloudRPS, Dev.to, Future Proof Technology)

The loop these habits come from is the one evaluation guides describe: a fixed set of real test cases, re-scored on a schedule and whenever something changes, and every production failure converted into a permanent regression case so the same failure cannot return unnoticed. Bigger teams also score a sample of live traffic; your canary set is the small-business version of the fixed-set half of that loop — the same discipline, sized to one weekly run. (Future Proof Technology, ContextQA)

Start small, as the guides advise: 20 to 50 real cases drawn from real requests, not invented ones. Guides also advise checking the steps the bot took — which tools it called, in what order, how many steps — with simple checks before reaching for anything fancier. Those checks need the bot to show its work; where yours doesn't, your canary deliberately checks something simpler and still useful: whether the final answer is right. A small set you have read carefully is worth more than a large generated set nobody has checked. (Programming Line, Future Proof Technology)

Follow the build: your 20-question canary set this week

Here is the build. A canary set starts as twenty questions your customers actually ask — fewer if that is all your shop has — with the correct answers written down by you. Once a week, you run the questions in your set through your bot and compare. If the answers match, your worker is behaving on these questions. If the same question fails two weeks running against a still-correct key, you have a pattern worth working through the failure steps below — the re-run there confirms it before you escalate. Your own weekly checks are watching for it, rather than leaving discovery to a customer complaint. Set aside one sitting to build it.

1. Open your last month of enquiry messages — WhatsApp, email, or your website chat, wherever customers actually write to you. Pick the twenty questions that come up most often. Price, stock, and timings will cover most of them. If your shop has fewer than twenty distinct recurring questions, start with what you have — never invent questions to fill the list — and add real ones as they arrive until the set reaches twenty. If your enquiry volume never produces twenty distinct questions, a smaller set run faithfully every week beats an invented twenty — the floor is a target, not a reason to fabricate.
2. For each question, write the correct answer in your own words, exactly as you would want a customer to hear it. This is your answer key. Keep it in one document or spreadsheet where you can find it again.
3. Ask your vendor, or check your bot's settings: "Which exact model version are we running, and can it be pinned so it does not change without us knowing?" Ask the vendor to pin the version and confirm the pinned version in writing. Write the pinned version down next to your answer key. Also ask how the vendor notifies you of model changes, and write that down too. If pinning is not available, write that down too — knowing is the point, and it decides whether the roll-back step later is open to you.
4. Once a week, at a fixed time, send the questions in your set to your bot one by one and write each answer next to the correct one. Run the canary in your bot's test or preview mode if it has one — first confirm with your vendor that the test mode runs the same version as production and takes the same routing. If the vendor never confirms, default to the live chat. If there is no test mode, or it doesn't match production, send the questions through the live chat from your own phone or number, as a customer would, and note 'test messages' in the Test channel line of your log. Mark each question pass or fail against your answer key — the key is the baseline, not last week's answers. Keep each question's first run as its starting record (week one means that question's first run, not the set's first run): if a question fails in week one, it was never passing, so check your answer key and documents first — that is a setup problem, not drift. If you correct the key or the document, re-run that question once: if it passes, record the corrected key as the baseline, reset that question's fail count, and move on; if it still fails, raise it with your vendor as a setup problem, using the setup message below. If the key and document were already correct, re-run once: a pass means the fail was a one-off — reset that question's fail count and move on; a second fail goes to your vendor as a setup problem, using the setup message below. While a setup problem awaits the vendor's fix, keep running the question weekly but do not count its fails toward the drift pattern — mark it 'awaiting setup fix' in the Status column, and resume normal counting once the vendor replies. If the vendor has not replied two weekly runs after the setup message was sent, treat the hold as expired: remove the hold mark and resume normal counting. If the question starts passing during the hold, reset its fail count and remove the hold mark. The "one fail is a curiosity" rule below applies to a question that previously passed.
5. Save each week's run in the same document, dated. Your history is what turns a single odd answer into a visible drift: one fail on a previously passing question is a curiosity, the same fail two weeks running is a pattern — and that is when you work the failure steps below. When a real customer question goes wrong and your set doesn't cover it, write it down with its correct answer and add it to the set in time to ask it at the next weekly run, letting the set grow past twenty — twenty is the starting size and the floor for a shop with twenty questions, not a ceiling. Ordinary new recurring questions can join the set the same way, once you have written down the correct answer. A newly added question starts with a fail count of zero. If it was added because it failed in the real world, check the key and documents first, as with any week-one fail; if they are correct, that first canary fail counts as the first fail in a row toward the two-week pattern — skip the week-one re-run and setup escalation for such a question, since its real-world failure already confirmed the problem, and enter the failure steps when it fails its second consecutive weekly canary run. If the newly added question passes its first canary run, its fail count stays zero — the real-world failure stays in the question's history, not in the count — and it enters the normal weekly pattern from there. If the key or document was wrong, correct it and re-run once as in step 4: a pass records the corrected key as the baseline and resets the count; a second fail goes to the vendor with the setup message. Questions added as ordinary new recurring questions follow the normal week-one rules in step 4. If your vendor notifies you of a model change, run the canary again out of cycle when the notification arrives, even if that week's fixed run already happened — treat that run as diagnostic only: it does not add to or reset the fails-in-a-row count, which only the fixed weekly runs move, and it never triggers escalation on its own; note a diagnostic fail and wait for the next fixed weekly run to confirm. If the set grows past fifty, prune back to fifty, starting with the questions that have never failed, then those that failed least recently — never prune a question marked known-open, awaiting setup fix, or in a no-reply wait.

Your weekly log template

Copy this template into your document to log each run. Copy the correct answers from your answer key into the log each week; never rewrite them from memory, or the key itself drifts. Diagnostic (out-of-cycle) runs are logged separately, under a 'Diagnostic runs' heading with the date, the question, and the bot's answer, and never touch the Fails in a row column. Confirmation re-runs inside the failure steps are not diagnostic runs — they move the fail count as their step directs. Mark known-open cases in the Status column by writing 'known-open' there, and note a question's real-world failure history there as well (e.g. 'failed live before joining'); mark a question awaiting a vendor reply on an escalation or re-escalation as 'awaiting vendor reply' there, so the prune exemption for a no-reply wait is checkable from the log; leave it blank otherwise. Add a row whenever the set grows past twenty.

CANARY RUN — week of: __________
Bot / model version on record: __________
Pinning status (pinned version / unavailable): __________
Previous pinned version (if changed): __________
Test channel (test mode / live chat; note 'test messages' if live): __________
Vendor notification method on record: __________
Question | Correct answer | Bot's answer | Pass / Fail | Fails in a row | Status
1. [your most-asked question] | [correct answer] | __________ | ___ | ___ | ___
2. [your most-asked question] | [correct answer] | __________ | ___ | ___ | ___
3. [your most-asked question] | [correct answer] | __________ | ___ | ___ | ___
4. [your most-asked question] | [correct answer] | __________ | ___ | ___ | ___
5. [your most-asked question] | [correct answer] | __________ | ___ | ___ | ___
6. [your most-asked question] | [correct answer] | __________ | ___ | ___ | ___
7. [your most-asked question] | [correct answer] | __________ | ___ | ___ | ___
8. [your most-asked question] | [correct answer] | __________ | ___ | ___ | ___
9. [your most-asked question] | [correct answer] | __________ | ___ | ___ | ___
10. [your most-asked question] | [correct answer] | __________ | ___ | ___ | ___
11. [your most-asked question] | [correct answer] | __________ | ___ | ___ | ___
12. [your most-asked question] | [correct answer] | __________ | ___ | ___ | ___
13. [your most-asked question] | [correct answer] | __________ | ___ | ___ | ___
14. [your most-asked question] | [correct answer] | __________ | ___ | ___ | ___
15. [your most-asked question] | [correct answer] | __________ | ___ | ___ | ___
16. [your most-asked question] | [correct answer] | __________ | ___ | ___ | ___
17. [your most-asked question] | [correct answer] | __________ | ___ | ___ | ___
18. [your most-asked question] | [correct answer] | __________ | ___ | ___ | ___
19. [your most-asked question] | [correct answer] | __________ | ___ | ___ | ___
20. [your most-asked question] | [correct answer] | __________ | ___ | ___ | ___
Fails this week: ___ / ___ (questions in set)
Action (if any question meets a failure-step trigger; known-open rows only on a re-escalation trigger): __________

The three vendor messages

The vendor message used in step 4 of the failure steps below is this — fill in the bracketed parts from your canary log. If your record says 'pinning unavailable', delete the final line before sending.

Hi, our enquiry bot's answers have changed on a question that used to be
correct. The question is: "[paste the question]".
The correct answer is: "[paste your answer-key answer]".
The bot now answers: "[paste the bot's answer]".
This answer failed two weeks running, and failed again on re-run.
Version on record: "[paste the pinned version from your answer key, or
'pinning unavailable' if that is what your record says]".
Which model version are we running now?
If a pinned version exists, please restore the version on record.

For a week-one setup problem (not a drift case), use this shorter message instead:

Hi, our enquiry bot has not answered this question correctly in our checks.
The question is: "[paste the question]".
The correct answer is: "[paste your answer-key answer]".
The bot answers: "[paste the bot's answer]".
Please advise what needs to change on our side or yours.

For a non-version finding (the version did not move), use this message instead:

Hi, our enquiry bot is answering this question wrong although the model
version has not changed. The question is: "[paste the question]".
The correct answer is: "[paste your answer-key answer]".
The bot answers: "[paste the bot's answer]".
The document behind the answer is: "[paste the document name]".
What I found: "[paste the specific finding — which document is wrong, or what the bot pulled instead]".
Please advise what needs to change in the prompt, document, or retrieval.

That is the whole system. A canary set to start, one sitting to build, one short run a week. It is the small-business version of the production-hardening discipline described on our MLOps, Evals & Production Hardening page — measured, not guessed.

When a canary fails: what to do first

This section applies when a question has failed two weeks running, except a question marked as a known-open case that has not met a re-escalation trigger. A single fail on a previously passing question is a curiosity, not a case — wait for the repeat before working these steps.

1. Check your side first. Re-read your answer key and the document behind the answer. If a price, timing, or rule changed on your side, update the answer key and the bot's underlying document, then re-run that question once to confirm the fix: if it passes, record the new key as the baseline for that question, reset its fail count, and resume the weekly run — the fail is explained, not drift. (This confirmation re-run is separate from step 3's re-run below; each serves its own gate.) If it still fails against the now-correct key, continue with the steps below — the key is no longer the explanation. Only a fail against a still-correct key continues below.
2. Note the version on record from your answer key. You will compare it with the vendor's reply to the message in step 4: a reply showing the version moved identifies your suspect. If pinning is unavailable and your record says so, there is no baseline version to compare against — note that the version cannot be verified and proceed anyway; the repeat-fail pattern itself is the evidence.
3. Re-run the failed question once, here in this sequence — a diagnostic out-of-cycle pass does not count. If it passes this time, the two-week pattern is broken: log it as a one-off, reset the fail count for that question, and keep watching — escalate only if it fails two weeks running again. If it fails again, proceed to escalate.
4. Escalate to the vendor with the message above, including the question, the correct answer, the bot's answer, the two-week repeat history, the re-run result, and the version on record, and asking which version you are running now. Specifics get fixes; "the bot seems worse" gets sympathy.
5. Roll back: if the vendor's reply shows the version moved and a pinned version exists, the message above already asks them to restore the version on record — after a restore, re-run the question once to confirm the fix: if it passes, reset the fail count for that question and keep the canary running weekly. If it fails that confirmation re-run, reset the fail count to zero and treat the next two weekly fails as a new pattern — escalate that pattern, once it fails again on the failure-step re-run, with the drift message above, rewritten to state that a restore was already tried and did not fix the answer, with the restore request replaced by a request for what else could explain the failure. If the vendor's reply says the version did not move and the question still fails against a correct key after re-run, the cause is not a version change: treat it as a prompt, document, or retrieval problem — re-check the document behind the answer (the price list, policy, or timing note the answer comes from) and raise that specific finding with the vendor, using the non-version message above. Mark the question 'awaiting vendor reply' in the Status column and reset its fail count; while it waits, do not count its fails toward the drift pattern and do not re-escalate it. Once the vendor replies, drop the wait mark and resume normal counting: if the vendor replies with a fix, apply it and re-run once to confirm — a pass records the fix and the question resumes the weekly run, and a failed confirmation counts as the first fail in a row toward a fresh two-week pattern; if the vendor replies without a fix, answer any follow-up they ask for, keep the weekly run going, and re-enter the failure steps only if the question fails two weeks running again. If the vendor has not replied two weekly runs after the non-version message was sent, drop the wait mark, reset the fail count again, and resume normal counting. If pinning is unavailable, there is nothing to roll back to: reset the question's fail count, so it re-enters the failure steps only if it fails two weeks running again, keep the weekly run going and press the vendor on notification of changes, so the next change doesn't arrive unannounced. The won't-restore decision that follows does not apply here — there is no pinned version to restore. If pinning was available and the vendor confirms the version moved but will not restore it, or has not replied two weekly runs after escalation, you have a decision, not a fix. If the vendor named a new version, make it the version on record going forward, keeping the old pinned version noted alongside it — never erase the pinned baseline. If the vendor has not replied, there is no new version to record: keep the old version on record. If the vendor confirms the move but names no new version, keep the old version on record and note the move as unversioned. Judge each changed answer on its own: if a new answer is acceptable, update that key entry deliberately and reset that question's fail count — a deliberate key change, never a copy of bot output made in passing. If a new answer is not acceptable, keep its key unchanged and mark the question as a known-open case: reset its fail count and do not re-escalate it while it stays a known-open case — re-escalate only under the same threshold as any other failure, with the changed answer failing two weeks running and failing again on re-run, sending the drift message again with the new answer's own two-week history, with the Version on record line showing the current version on record and the restore line asking for the old pinned version noted alongside the record, or if the vendor's position changes (they now offer a restore, or name a new version), sending the drift message again once the changed answer has failed two weeks running and failed again on re-run, with that history and noting that change. Detect a position change through the vendor's notification channel, or by asking again the next time you contact them — if the vendor cannot notify you, ask at least once a quarter whether a restore is now offered. If the vendor again declines or does not reply two weekly runs after the re-escalation, mark the question known-open again and reset its fail count. If a restore offered on re-escalation then fails, follow the post-restore branch above. Remove the known-open mark when the case resolves — the answer becomes acceptable, or a restore fixes it. The canary now watches the version on record. Every no-reply wait in this step applies only while the question keeps failing: if it passes during a wait, reset its fail count and drop the pending decision.

• A canary set catches drift in the questions you chose. It does not cover questions you never thought to include, and it does not fix your documents or your prompts — those need their own checks. What it gives you is an early warning for a failure class most routine checks don't watch: the vendor changing something upstream.