Your Customer Asks in Hindi, Your Price List Is in English: The Cross-Language AI Answer Gap

Your price list is in English. Your customer asks in Hindi.

Picture a shop in Bengaluru that sells RO water purifiers and annual service plans. The price list is in English — model names, SKUs like RO-AMC-STD, coverage tables. The customers on WhatsApp write things like "yeh kitne ka hai?" and "तीन साल की AMC में क्या-क्या शामिल है?" The shop owner has heard the promise: an AI agent can "answer from your price list and catalog in your customer's language, on WhatsApp, Instagram, and your website" — that is the exact pitch Querygen makes to Indian businesses, with an explicit FAQ saying an AI agent can reply in Hindi or Hinglish while an old button bot can only reply in the language its menus were written in.

This post is about the quiet gap inside that promise. Getting a cross-language answer right has two gates, not one — and they fail differently.

The first gate is getting better fast, and this month Google shipped an open model aimed squarely at it. The second gate is where systems quietly fail — and new research from Amazon's AGI team shows it failing even in well-resourced languages.

There is also a hard-honesty note to make early, because it shapes everything below. Nobody has measured the second gate in Hindi or Hinglish — not the researchers, not the model vendors. What follows fuses three things: a benchmark that names the failure mode (in other languages), a practitioner's retrieval experiment in romanized Hinglish (anecdotal, retrieval-only), and the production pattern practitioners converged on. No single source connects them; the connection is this post's work. Where a claim is inference rather than measurement, I will say so.

Two gates between a Hindi question and a Hindi answer

Cross-language retrieval-augmented generation is the formal name for this shape: the customer writes in one language, the documents sit in another. Our earlier posts RAG Without the Hype: A Practical Guide to AI Over Your Company Docs and Grounded AI Answers: The 30-Question Trust Test Before Your Bot Touches a Customer covered the machinery and the answer-quality discipline — both monolingual. This post is the cross-language variant.

Think of the pipeline as two gates in series, as above. The Hindi question enters gate one — retrieval — which must find the right English row in the price list. That row enters gate two — generation — which must answer in the customer's language, correctly. A failure at either gate produces a wrong answer, but they fail differently: gate one finds the wrong row (the answer is confident but about the wrong product); gate two finds the right row but answers in English the customer won't read — or, worse, answers confidently in Hindi that doesn't match the row.

The uncomfortable part is that passing gate one can make gate two's failure invisible. A demo that shows "it found the right document" is a demo of gate one. The customer's experience is gate two. If you evaluate only retrieval — most dashboards do — you will believe a broken system is working.

Gate one: finding the English price row

The good news first. On October 6, 2026, Google DeepMind launched EmbeddingGemma 2 (launch post by Sahil Dua and Henrique Schechter Vera): an open-weight model under the Apache 2.0 license, 740 million parameters, that maps text, code, images, video, and audio into one shared embedding space. It "matches the strong multilingual text performance" of the first EmbeddingGemma — the model card puts that at 61.36 against 61.15 for v1 on MTEB multilingual v2 (via fourweekmba) — with the model card claiming understanding of 100+ languages (unite.ai). It is sized for on-device inference: about 191MB of active RAM for text-only weights on a Pixel 11 Pro, per Google's developers blog, and its vectors can be truncated from 768 dimensions down to 512, 256, or 128. The retrieval gate, in other words, is increasingly something you can download as an open model rather than build from scratch.

The predecessor passed 20 million downloads, with builders using it for on-device search tools and "privacy-first retrieval augmented generation (RAG) pipelines" — Google's own words, about the first version, not the new one.

But "multilingual" is not "Hinglish-ready." The launch post reports gains on code and audio — and, in one practitioner's exact words, "gives no Indic or romanized numbers." Nobody at the launch measured romanized Hinglish, the way your customers actually type.

So a practitioner did it himself: an October 2026 experiment (Medium) tested eight embedding models on 664 romanized-Hinglish paraphrases against an English question cache. EmbeddingGemma 2 won Recall@1 — 76.5 on romanized Hinglish against 87.7 on English — yet at the untuned 0.80 threshold an answer cache would actually use, it served every query and 156 of the 664 were wrong: 23.5%.

The signature failure is the near-miss: a customer's "mera order abhi tak kyun nahi aaya? overdue ho chuka hai." matched "How do I qualify for free shipping?" at 0.8887 while the right question scored 0.8882.

A gap of 0.0005 — no threshold can separate scores that close. And one prompt flag — a single sentence telling the model the task — moved the hit rate by 21 points without meaningfully changing Recall@1.

Two honest caveats ride along with those numbers, from the author's own "Limits" section:

• 0.80 was an untuned threshold and "the wrong operating point for EmbeddingGemma 2 in particular," so the 23.5% is a statement about that threshold, not the model's ceiling.
• This is one practitioner's experiment — anecdotal, synthetic intents, one domain. It is also the only public Hinglish-specific retrieval test anyone has published. So treat it as a warning about thresholded serving, not a ranking of models.

The takeaway for gate one: retrieval across the language wall is genuinely improving, but romanized Hinglish is the hard case and it fails silently — by serving the wrong cached answer confidently, at scores that look like success. A 23%-wrong cache doesn't announce itself. Only a test catches it.

Gate two: answering in the customer's language

Now the harder gate — and the one nobody has measured in Hindi or Hinglish. In May 2025, Amazon's AGI team published XRAG (Liu et al.), a benchmark built specifically for cross-lingual RAG: the user's language does not match the retrieval results. It covers two settings, and the failure modes differ:

The paper's own motivating example is your shape exactly: "a corporate helpdesk with an internal database available only in English." And XRAG states plainly that response-language correctness "has received little attention from the research community."

The guardrail, stated once and clearly: XRAG tested Arabic, Chinese, German, and Spanish — not Hindi, not Hinglish. Its findings cannot be presented as Hindi findings. What we have for Hindi and Hinglish is informed inference: the only Hinglish evidence is the retrieval-stage experiment above, and no measurement of the answer-language gate exists for Hindi/Hinglish anywhere. That gap is itself the story. If you are buying or building a system that claims to answer your customers in their language, understand that nobody — not the researchers, not the vendors — has published the measurement that would prove it. The 10-question test below is the owner's substitute for that missing measurement.

Why gate two is sneaky, in owner terms:

• It fails where demos don't look. A vendor demo shows retrieval: "see, it found the right price row." Nobody in the demo reads the Hindi answer back against the row, word by word. Gate two is where the model must hold numbers, SKUs, and conditions constant while switching languages — and that is exactly what XRAG found models bad at.
• Language-switching is real customer behavior. Customers don't stay in one language. MyOperator's WhatsApp buying guide describes a multilingual chat agent that "catches the context switch in real time" when a customer switches languages mid-chat — Hindi, English, Hinglish, and the major Indian languages — and hands off to a human with the full conversation attached. Your test questions should include a mid-chat switch.
• The market already treats this as a buying criterion. A 2026 comparison of AI chat agents for Indian SMBs (MyOperator) ranks vendors by Indian-language coverage and Hinglish handling; its verdict on one vendor reads, verbatim: "Suits Indian SMBs that need multilingual chat automation without a developer team." And an India SMB WhatsApp guide (aiinsider — note it carries an affiliate disclosure, so read its advice as affiliate-influenced, not neutral) lists "sirf English mein responses likhna" — "only writing responses in English," because "Indian customers expect Hinglish" — as a common new-buyer mistake.

The 10-question cross-language trust test

If you read our earlier Grounded AI Answers post, you ran the 30-question trust test: same afternoon, same spreadsheet, but it checked answer correctness — groundedness, relevance, clean refusals. This is explicitly its cross-language sibling: 10 questions, 5 in Devanagari Hindi and 5 in romanized Hinglish, all against your English price list. What it checks is different: not just "is the answer right," but "is it right in the customer's language, and did the prices, model names, and SKUs survive the language crossing intact?" A pass needs all three marks on all ten questions. Run it in a private test chat on an afternoon, before the bot talks to a real customer.

Step 1 — Set up your test sheet

Action: Open a blank spreadsheet with these seven columns, copied exactly:

Question # | Question (as the customer typed it) | Expected language | Right information? (Y/N) | Right language? (Y/N) | Prices & SKUs intact? (Y/N) | Verdict (pass = YYY)

Expected result: Seven columns, ten empty rows. This sheet is your evidence; keep it. If a vendor disputes a failure later, you will point at rows, not adjectives.

Step 2 — Write the ten questions

Action: Copy these ten questions into the sheet, exactly as written — typos, script-mixing, and all. They are built for an RO purifier dealer with an English price list, but the pattern adapts to any business: five information types (price, coverage, delivery, compatibility, policy) in two scripts.

1. यह वॉटर प्यूरीफ़ायर कितने का है?
2. तीन साल की AMC में क्या-क्या शामिल है?
3. मेरा ऑर्डर अभी तक क्यों नहीं आया? देर हो चुकी है।
4. क्या यह प्यूरीफ़ायर हार्ड वॉटर के लिए ठीक है?
5. रिफंड के लिए कितने दिन इंतज़ार करना पड़ेगा?
6. yeh purifier hard water ke liye theek hai?
7. 3 saal ki AMC me installation free hai kya?
8. mera order kab tak aayega, abhi tak nahi aaya
9. RO-AMC-STD aur RO-AMC-PLUS me kya difference hai?
10. warranty me motor cover hota hai ya nahi?

Expected result: Questions 1–5 expect Hindi answers (Devanagari). Questions 6–10 expect Hinglish answers (Roman script). Question 9 is the entity trap: two near-identical SKUs that a sloppy answer will merge. Question 8 is the near-miss trap, modeled on the Hinglish experiment's own failure case — a delivery question that embeddings confuse with shipping-policy questions.

Step 3 — Ask them in a private test chat

Action: In your bot tool's private test mode — never your customer-facing number — ask each question exactly as written, one per message. For questions 7 and 10, first ask an English question ("What models do you have?"), then ask the Hinglish one: this simulates the mid-chat language switch MyOperator says real customers do. Paste each answer into the sheet next to its question.

Expected result: Ten answers in your sheet. Do not fix anything yet — a bad answer is data.

Step 4 — Grade three marks per answer

Action: For each answer, mark Y or N on the three checks. All three must be Y for a pass:

• Right information? Does the answer match what your English price list actually says? Check against the document, not your memory.
• Right language? Is the answer in the customer's language — Devanagari Hindi for questions 1–5, Roman-script Hinglish for 6–10? An English answer to a Hindi question is a fail, even if the facts are right. This is the XRAG failure mode, made gradable.
• Prices and SKUs intact? Are every price, model name, SKU, date, and measurement copied exactly — not translated, rounded, or rephrased? RO-AMC-STD must not become "RO AMC standard plan" and a number must not change. This is where cross-language answers silently corrupt facts.

Expected result: A sheet with thirty marks. If any question fails any mark, the system fails the test — note which gate broke (wrong row found = gate one; right row, wrong language or corrupted entities = gate two) and send the sheet back to whoever built it. Repeat the test after each fix; a retest of all ten takes under an hour.

This test is a substitute, not a benchmark. It tells you whether your system holds up on your documents. It does not tell you which model is best — and remember, the only published Hinglish retrieval numbers come from one practitioner's experiment. Run the test; don't shop for leaderboard scores.

Stop translating your documents

The build prescription, if your test fails — or if you're briefing a developer or vendor — is one rule: stop translating your documents. The practitioner guides converge on the same shape while differing on how much translation to do.

Either way, the knowledge base stays in one language — your English price list, as it is.

The non-negotiable parts of that shape:

• One index, one knowledge base. Never maintain a Hindi copy of your price list. Two copies drift; the drift becomes wrong answers. Multilingual retrieval (models like the one above, tuned for your domain) does the cross-language finding.
• Answer in the customer's language, with protected entities. Both guides agree: SKUs, order IDs, brand names, model numbers, and measurement values get placeholders — detected, replaced before any language step, restored after — so no translation step can corrupt them. This is the mechanical answer to gate two.
• Evaluate per language. HoverBot's pattern measures recall separately per language and checks agreement between passages retrieved for equivalent questions across languages — the multilingual version of your 10-question test, kept as a standing habit.

The single most leveraged artifact is the system prompt. This is the language-aware version — copy it, fill in the brackets, paste it into your bot tool's system-instructions field:

You are the enquiry assistant for [BUSINESS NAME].

Answer ONLY from the business documents below. If the answer is not in the
documents, say you do not know and offer to connect the customer to the team.

ANSWERING LANGUAGE: Reply in the customer's language. If the customer writes
in Hindi, answer in Hindi. If they write in Hinglish (Hindi in Roman script),
answer in Hinglish. If they write in English, answer in English. Never answer
in a different language than the customer's message. If the customer switches
languages mid-conversation, switch with them.

ENTITY PROTECTION: Prices, model numbers, SKUs, dates, and measurements are
facts. Copy them EXACTLY from the documents. Never translate, round, or
rephrase them. Keep product codes exactly as written (for example,
RO-AMC-STD stays RO-AMC-STD). A number that changes is a wrong answer.

CITATIONS: End every answer with the source document name, for example:
[Source: Price List, updated 2026-10-18]

This prompt is where gate two gets enforced — and it is also what your 10-question test grades. If an answer fails the language mark or the entity mark, the fix usually starts here, not in the retrieval settings. For the broader build around it — how the retrieval layer connects to your documents — our LLM integration work is the standing reference.