Your AI Quoted the Wrong Price. The Problem Is the Chunk, Not the Model

The wrong price came from the right document

A customer messages your shop on WhatsApp: "What is the warranty on the ceiling fan?" Your AI answers, confidently and instantly: one year. The customer buys. Two months later the fan fails, and your staff discover the truth: the ceiling fan carries a one-year warranty, but the answer the AI gave came from the table fan's entry — same words, different product. The document was right. The bot read the right document. And the answer was still wrong.

That is the failure this post is about. It is not a model failure. A smarter model would have made the same mistake, because the mistake happened before the model ever saw the question — at the moment the document was cut into pieces for search. Anthropic gives the canonical example: a chunk reading "The company's revenue grew by 3%" is meaningless on its own, because it does not say which company or which quarter. Cut a price list the same way and you get the same shape of failure: a chunk reading "Warranty: 1 year" that does not say which product. Three products in the sample list in this post share the identical line "Warranty: 1 year." A chunk that says "1 year" and nothing else cannot be answered correctly, no matter how good the model is.

If your AI has ever quoted the wrong price, the wrong warranty, or the wrong delivery time — and the document it quoted from was correct — this is almost certainly what happened. The rest of this post is the research on why it happens, the decision that comes before any build, and a follow-along you can run on your own list in an afternoon.

Why chunks lose their context

Retrieval-augmented generation, the technique behind an AI that answers from your own documents, works in four steps. Anthropic lays them out plainly: break the knowledge base into smaller chunks, usually no more than a few hundred tokens; convert each chunk into a vector embedding that encodes meaning; store the embeddings in a database that supports similarity search; then, at query time, find the closest chunks and add them to the prompt. That is the whole machine. Everything else is refinement.

Two kinds of search do the finding, and they fail in different ways. Embeddings search by meaning: they are good at "the customer asked about fans, find the fan passages." BM25, the older keyword technique, searches by exact words: it is what finds "Error code TS-999" when an embedding model would return generic passages about error codes (Anthropic). Serious systems combine both — embeddings plus keyword search, merged by rank fusion — because each catches what the other misses. Anthropic's own tests found embeddings plus BM25 better than embeddings alone.

But both kinds of search share one blind spot, and Anthropic names it directly: "these traditional RAG systems have a significant limitation: they often destroy context." The chunk is the unit the model sees. If the chunk does not carry its own identity — which product, which section, which document — then the search can return the right words attached to the wrong product, and the model will answer from them in good faith. Compare the two ways of cutting the same product list:

The difference is not the model, the search, or the size of the database. It is where the knife falls. And in October 2026, that question — where to cut, and whether to write extra context onto each chunk — became a live research debate with measurements a small business can actually use.

October 2026: the retrieval debate, in plain language

Anthropic's answer to the context problem is called contextual retrieval, published in September 2024 and updated in October 2026. The method is simple to describe: a small model reads each chunk together with its whole document and writes one or two sentences situating the chunk — which document, which section, what it refers to — and that context is prepended to the chunk before embedding and before building the keyword index (Anthropic). Their measured ladder, on top-20 retrieval failures across codebases, fiction, and papers:

That ladder stood for two years. Then, on 7 October 2026, a placebo-controlled study asked whether the LLM-written context was doing the work, or whether something cheaper was. Kuehlkamp et al. tested four ways of using document structure — structure-aligned chunking, LLM-generated chunk contexts, heading-path metadata, and hierarchical retrieval — on two distant corpora (200 Wikipedia featured articles and 1,585 QASPER papers), with shuffled heading paths as a placebo arm so that any gain from merely adding tokens would show up as a placebo effect. Their finding: structure-aligned chunks with real heading paths beat contextualised fixed windows (+0.022 and +0.012 coverage-aware nDCG@10), and the gain came from boundary placement — from cutting at the right place — not from the added words.

A second study, published the same month by AQUA LLC, tested both claims on a production system: the 150-article Japanese blog behind a deployed assistant, with 295 paraphrased questions. Its results split the difference in a useful way. On dense (embedding) retrieval, heading-aligned chunks beat LLM-written contexts clearly (nDCG@10 +0.070, Holm p < 0.001), replicating Kuehlkamp. On BM25 keyword retrieval, the LLM-written contexts helped (+0.030) — the side Kuehlkamp did not test. After hybrid fusion of both retrievers, no indexing variant was significantly better than any other. And the reranker dominated everything: reranking the top 50 candidates halved top-20 failures from 18.0% to 9.8%, and erased the differences between indexing variants entirely.

Read together, the three sources give a small business a clear order of operations, and it is not the order most vendors pitch:

1. Cut at the headings first. It is free — your documents already have headings — and on embedding search it beat the method that spends a model call on every chunk.
2. Add written context for keyword search, not as a default. The extra words help BM25 find exact terms. If your customers ask with the document's own wording (product names, model numbers), that side matters.
3. Rerank the shortlist. In AQUA's experiment the reranker had the largest effect of any single change — larger than either indexing trick. Anthropic's ladder agrees on the direction: their largest single step was contextual embeddings (−35%), and reranking took the total reduction from 49% to 67%.

One honest caveat from AQUA, worth keeping: after hybrid fusion plus reranking, the indexing variants no longer differed. That does not make chunking irrelevant — the reranker can only rank what retrieval found, and on their production weights a badly fused candidate pool stayed bad (22.7% failures even after reranking). It means chunking is the foundation and reranking is the multiplier, and neither rescues a pipeline that never retrieves the right passage in the first place.

The decision before you build anything

Before the follow-along, there is a decision that saves most small businesses from building anything at all. Anthropic states it as a rule of thumb: if your knowledge base is smaller than 200,000 tokens — about 500 pages of material — you can just include the entire knowledge base in the prompt, with no need for RAG or similar methods, and prompt caching makes that approach more than twice as fast and up to 90% cheaper (Anthropic). JY Labs, writing for business owners in October 2026, restates the same line as roughly 150,000 words: under that, skip the index and use a long-context prompt with caching; over that, you need retrieval.

That gives you a three-way decision. Measure first, then pick:

A note on cost, since this post quotes none: vendor prices move, and this post will be read months after it is written. If you are comparing a per-seat assistant against a custom build, check the current prices on each vendor's own pricing page before you budget. The decision above does not depend on any price — it depends on the size of your library and on whether the work is answering or acting.

The follow-along: a product-list answerer that finds the right product

This is the build. It takes an afternoon, needs nothing but Python on any computer, and its only purpose is to show you the chunking effect on your own list before any customer-facing bot touches it. Two design choices, stated plainly so you can judge them. First, the demo uses warranty, delivery, and installation terms instead of prices — the failure mode is identical (a fact detached from its product), and terms do not go stale the way a price list does. Swap the columns for prices in your own list and the lesson is the same. Second, the demo searches with keywords only, not embeddings: the chunking lesson does not depend on which search you use, and keywords run anywhere with no account and no API key. In production you add embeddings and a real reranker; the chunking stays exactly as shown here.

The sample data is fictional — a made-up electrical shop, made-up terms — and is labelled as such in the script. Replace it with your own list and your own twenty questions, and keep the script's verdict as your baseline.

Step 1: Write down your twenty questions

Ask your team for the twenty questions customers actually ask most. JY Labs puts it as a rule: write down the twenty your team asks most, because those become your evaluation set — the test you run before launch and after every document update. Write the correct answer for each one before you run anything. If you cannot write the answer, the document is the problem, not the AI — fix the document first.

Step 2: Put your list in one document, one product per heading

Write the list as a plain document with one heading per product, one fact per line. The sample in the script is eight products with warranty, delivery, and installation lines. The shape is what matters, not the products: every fact lives under the heading of the product it belongs to, and no fact is shared between two headings. If your real list is a spreadsheet, export it to this shape for the test — one row per product, one column per fact — and check that no two products share a line of text. (In the sample, three products share "Warranty: 1 year." — that is deliberate. It is exactly the ambiguity the test exists to catch.)

Step 3: Run the comparison script

Save the block below as chunk_demo.py and run it with python3 chunk_demo.py. It needs nothing but the Python standard library. It cuts your list two ways — fixed 300-character windows that ignore structure, and one chunk per product heading — searches each version with the same twenty questions, and prints how many answers came back clean (the right fact, and only that product's facts), mixed (the right fact, buried among other products' facts), or missed (the right fact not in the top chunk at all). The rerank step then re-scores the shortlist, preferring chunks whose heading matches the question.

"""Demo: does how you cut a product list change whether the AI finds the
right product? Pure standard library. Run: python3 chunk_demo.py

The product list is SAMPLE data (fictional shop, fictional terms) — replace
it with your own list and your own 20 questions, then compare the two scores.
"""

import re

PRODUCTS = """# Product list — Sharma Electricals (sample data)

## Ceiling Fan — HighSpeed 1200mm
Warranty: 1 year.
Delivery: 2 days within the city.
Installation: included.

## Mixer Grinder — TurboMix 750W
Warranty: 2 years on the motor, 1 year on the jar.
Delivery: 3 days.

## LED Bulb — BrightLite 9W (pack of 4)
Warranty: 6 months.
Delivery: next day within the city.

## Water Heater — AquaHeat 15 litre
Warranty: 5 years on the tank, 2 years on the heating element.
Delivery: 4 days.
Installation: charged separately.

## Table Fan — BreezeMate high speed
Warranty: 1 year.
Delivery: 2 days within the city.

## Electric Kettle — QuickBoil 1.5 litre
Warranty: 1 year.
Delivery: next day within the city.

## Room Heater — WarmGlow 2000W
Warranty: 2 years.
Delivery: 3 days.
Note: not for bathroom use.

## Extension Board — SafePlug 5 socket with surge guard
Warranty: 6 months.
Delivery: next day within the city.
"""

QUESTIONS = [
    ("What is the warranty on the ceiling fan?", "Ceiling Fan", "Warranty"),
    ("How long is the mixer grinder motor covered for?", "Mixer Grinder", "Warranty"),
    ("What warranty does the LED bulb pack carry?", "LED Bulb", "Warranty"),
    ("How many years is the water heater tank covered?", "Water Heater", "Warranty"),
    ("What is the delivery time for the table fan?", "Table Fan", "Delivery"),
    ("Is installation included with the ceiling fan?", "Ceiling Fan", "Installation"),
    ("Does the water heater installation cost extra?", "Water Heater", "Installation"),
    ("What is the warranty on the electric kettle?", "Electric Kettle", "Warranty"),
    ("Can the room heater be used in a bathroom?", "Room Heater", "Note"),
    ("How fast can the extension board be delivered?", "Extension Board", "Delivery"),
    ("What is the warranty on the mixer grinder jar?", "Mixer Grinder", "Warranty"),
    ("How long does delivery take for the water heater?", "Water Heater", "Delivery"),
    ("What is the warranty on the table fan?", "Table Fan", "Warranty"),
    ("What is the delivery time for the LED bulb pack?", "LED Bulb", "Delivery"),
    ("Is the room heater covered for two years?", "Room Heater", "Warranty"),
    ("What is the warranty on the extension board?", "Extension Board", "Warranty"),
    ("Does the mixer grinder come with a two-year motor warranty?", "Mixer Grinder", "Warranty"),
    ("How long is the ceiling fan warranty?", "Ceiling Fan", "Warranty"),
    ("What is the delivery time for the electric kettle?", "Electric Kettle", "Delivery"),
    ("Is the water heater tank covered for five years?", "Water Heater", "Warranty"),
]

STOPWORDS = set(
    "what is the a an on for how long does do is it in of to with per".split()
)


def words(text):
    return [w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOPWORDS]


def naive_chunks(text, size=300):
    """Fixed-size windows: cut every `size` characters, ignoring structure."""
    return [text[i : i + size] for i in range(0, len(text), size)]


def heading_chunks(text):
    """One chunk per product: cut at headings, keep the heading on the chunk."""
    chunks, current_head, buf = [], "", []
    for line in text.splitlines():
        if line.startswith("## "):
            if buf:
                chunks.append((current_head, "\n".join(buf)))
            current_head, buf = line[3:], []
        elif line.strip():
            buf.append(line)
    if buf:
        chunks.append((current_head, "\n".join(buf)))
    return [f"{head}\n{body}" for head, body in chunks]


def score(query, chunk):
    qw = set(words(query))
    return sum(1 for w in qw if w in chunk.lower())


def retrieve(query, chunks, top_n=3):
    ranked = sorted(chunks, key=lambda c: score(query, c), reverse=True)
    return ranked[:top_n]


def rerank(query, candidates):
    """Second pass over the shortlist: exact product-name match wins."""
    qw = set(words(query))

    def rerank_score(chunk):
        head = chunk.split("\n")[0].lower()
        head_words = set(re.findall(r"[a-z]+", head))
        return (3 if qw & head_words else 0) + score(query, chunk)

    return sorted(candidates, key=rerank_score, reverse=True)


def product_lines(product):
    """The Warranty:/Delivery: lines belonging to one product."""
    start = PRODUCTS.index("## " + product)
    end = PRODUCTS.find("\n## ", start + 1)
    block = PRODUCTS[start : end if end != -1 else len(PRODUCTS)]
    return [ln for ln in block.splitlines() if ln.startswith(("Warranty:", "Delivery:", "Installation:", "Note:"))]


ALL_LINES = sorted(
    {ln for p in set(q[1] for q in QUESTIONS) for ln in product_lines(p)}
)


def run(name, chunks, use_rerank):
    clean, mixed, missed = 0, 0, 0
    notes = []
    for question, product, attr in QUESTIONS:
        top = retrieve(question, chunks, top_n=10 if use_rerank else 3)
        if use_rerank:
            top = rerank(question, top)[:3]
        best = top[0] if top else ""
        expected = [ln for ln in product_lines(product) if ln.startswith(attr)][0]
        own = set(product_lines(product))
        present = {ln for ln in ALL_LINES if ln in best}
        if expected in best and present == own:
            clean += 1
        elif expected in best:
            mixed += 1
            notes.append(
                (question, f"answer present but mixed with {len(present - own)} other fact(s)")
            )
        else:
            missed += 1
            notes.append((question, "right answer not in the top chunk"))
    print(f"{name}: {clean} clean / {mixed} mixed / {missed} missed (of {len(QUESTIONS)})")
    for q, why in notes:
        print(f"  {why}: {q}")
    return clean


if __name__ == "__main__":
    naive = naive_chunks(PRODUCTS)
    headed = heading_chunks(PRODUCTS)
    run("Naive fixed windows        ", naive, use_rerank=False)
    run("Heading-aligned chunks     ", headed, use_rerank=False)
    run("Heading-aligned + rerank   ", headed, use_rerank=True)

Step 4: Read the verdict on your list

On the sample list, the verdict is stark: the naive fixed windows score 0 clean answers out of 20 — 17 mixed, 3 missed — while the heading-aligned chunks score 20 clean. Read that carefully, because it is the whole post in one line. The naive version usually found the right fact; it just returned it mixed with two to five other products' facts, so a model answering from that chunk would have to guess which warranty belonged to which product. That guess is your wrong price. Your numbers will differ — your list, your questions — and what matters is the gap between the two methods on your list, not the sample's exact score. If the gap is small, your documents are already well-structured and you can stop here. If it is large, the fix is in the documents, and it costs nothing: give every fact a heading.

Step 5: Add the rerank pass and the refuse rule

The script's rerank function is the production shape in miniature: take the top candidates from the first search, then score them again with a stricter test — here, whether the chunk's heading matches the question — and keep only the best. On the sample list the rerank changes nothing, because the heading chunks are already clean; that is honest, and it is the point. A reranker earns its keep when the shortlist is crowded, which is exactly what AQUA LLC measured: reranking halved their top-20 failures. In a production build the reranker is a model (Anthropic tested with a Cohere reranker; AQUA used a small language model), and the principle is the same: the first search optimises for recall, the second pass optimises for precision, and only the winners reach the answer.

Add one rule alongside it, before any customer sees an answer: if no chunk scores above your threshold, the bot refuses and hands the question to a human. A refusal is a recoverable event. A confident wrong price is a refund, a complaint, and a customer who stops trusting the bot. The 30-question trust test in our earlier post on grounded answers is the companion check for that rule — run it after this one.

Step 6: Re-run the twenty questions after every document change

Your price list changes. A warranty changes, a delivery time changes, a product is discontinued. Every change is a chance for the retrieval to go stale — the failure our earlier post on stale documents covers in full. Make the twenty questions a habit, not a launch event: run the script after every update, and the day the score drops, fix the document before a customer meets the wrong answer. Ten minutes of checking beats a week of corrections.

What production deployments actually measured

Two deployments, measured before and after, show what the chunking and reranking work is worth when it leaves the lab:

AQUA's production assistant

Switching from document-level to chunk-level hybrid retrieval with reranking raised the share of questions whose evidence reached the generator from 59% to 81%, and answer accuracy from 67% to 87% — while cutting the context handed to the model from about 30,000 characters to about 4,000, at the cost of about 1.2 seconds of added latency (AQUA LLC). That is one deployment's measured result, not a guarantee — but note the direction of the trade: smaller context, better answers, slightly slower. For a customer enquiry, that trade is worth taking.

A law firm's document retrieval

A regional law firm put 50,000+ documents behind a hybrid keyword-and-vector pipeline with clickable citations to the exact paragraph, tested it against 200+ real attorney questions until accuracy exceeded 95%, and cut attorney research time by 80% — from 3.2 hours to 38 minutes a day (JY Labs). That is a vendor-reported case study, not an independent audit, and the numbers belong to a law firm, not a shop. What transfers is the shape of the win: the evaluation set of real questions, the citations, and the hybrid retrieval — the same three ingredients as the AQUA deployment.

The shared lesson is the one this post has been making throughout: both wins came from retrieval quality — how documents were cut, searched, and re-ranked — not from a bigger model. If you are choosing where to spend your next rupee of AI effort, spend it on the retrieval, and measure it with your twenty questions.

The rules that keep it honest

Four rules, and they do not change when the system gets bigger:

• Every answer carries its source: the product, the heading, and the line it came from, so a human can verify it in one click. A citation is what turns a confident answer into a checkable one.
• The bot refuses when it cannot find the answer — a handoff to a human beats a guessed price every time.
• The twenty questions run after every document change, not just at launch; a price list is a living document and retrieval goes stale with it.
• Headings are a habit, not a one-time fix: every new document enters the library with product headings already on it, so the chunking never has to be repaired after the fact.

None of this needs a bigger model. It needs documents that say what they are about, a search that respects that structure, a second pass over the shortlist, and a human at the edge where a wrong answer costs money. Build that, measure it with your twenty questions, and the next time a customer asks about the ceiling fan, the answer will come from the ceiling fan.