RAG Without the Hype: A Practical Guide to AI Over Your Company Docs

RAG in one paragraph

Retrieval-augmented generation — RAG — connects a language model to your documents. When someone asks a question, the system first retrieves the relevant passages from your knowledge base, and then the model writes its answer using only those passages, with citations pointing back to the source pages.

The result is an assistant that answers from your SOPs, manuals, policy documents, and past support tickets instead of guessing. A new hire gets the right answer in seconds. A senior engineer stops answering the same question for the tenth time. The knowledge that today lives inside three people's heads becomes available to the whole company, any time of day.

The technology itself is not the hard part. The hard part is everything around it: making sure the right passages get retrieved, keeping documents current, and making sure people only see what they're allowed to see. Get those right and RAG is one of the most useful things AI can do for a company. Get them wrong and you have an expensive chatbot that people stop trusting within a month.

Where companies go wrong

The most common failure is treating RAG as a weekend project. Somebody dumps a pile of PDFs into a vector database, wires up a chatbot, and then everyone is surprised when the answers can't be trusted. The demo looked great because the demo used five clean documents. Production has five hundred messy ones, and the difference matters.

Retrieval quality is the whole game. If the system retrieves the wrong passages, the model will write a confident, fluent answer based on the wrong information — which is worse than no assistant at all. Nobody checks an answer that sounds right. Bad chunking, missing metadata, and stale documents are the usual culprits, and all three are boring pipeline problems, not AI problems. A serious RAG build spends most of its effort on the pipeline, not the chat widget.

The second common failure is skipping evaluation. Companies launch without ever writing down what "good" looks like, so nobody notices when accuracy quietly degrades. Documents change, people ask new kinds of questions, and six months later the assistant is answering from last year's price list. Without a test set, you find out from an angry customer.

The third failure is ignoring access control. A chatbot that can read every document in the company will eventually repeat something it shouldn't have — salary details from an HR letter, a margin figure from a finance sheet, a customer complaint that was meant to stay internal. This is the failure that turns a tool project into an incident.

What good retrieval actually looks like

Retrieval is the step where the system finds the passages that answer the question. Everything else — the fancy model, the chat interface — is window dressing on top of it. If retrieval is bad, nothing downstream can fix it. Here's what good retrieval is actually made of, in plain terms.

First, chunking. Your documents have to be cut into small passages before the system can search them, because the model can't read a 200-page manual all at once. Chunking just means cutting documents into passages of a few paragraphs each. Bad chunking cuts in the middle of a thought — the first half of a procedure lands in one chunk, the second half in another, and neither one makes sense alone. Good chunking respects the structure of the document: one procedure per chunk, one policy clause per chunk, one troubleshooting step per chunk. For most company documents, cutting by section heading with a bit of overlap works well and needs no special tools.

Second, metadata. A chunk of text floating on its own is hard to find. Attaching a few facts to each chunk — which document it came from, which department owns it, what date it was last updated, what product or topic it covers — makes retrieval dramatically more reliable. When someone asks about the warranty terms for last year's model, metadata is what stops the system from answering with this year's terms. Most companies already have this information; it just needs to be recorded when documents are added.

Third, matching meaning as well as words. Traditional search matches keywords: if you type "refund," it finds documents containing "refund." But your support tickets might say "money back" and your policy might say "reimbursement," and keyword search misses both. Modern retrieval also converts passages into a numerical form — called embeddings — that captures meaning, so "refund," "money back," and "reimbursement" all match each other. The practical setups that work best combine both: keyword matching for exact terms like product codes and invoice numbers, plus meaning matching for everything else. That combination is sometimes called hybrid search, and you don't need to understand the mathematics — just know that a good setup does both, and a cheap one does only keywords.

Fourth, reranking and filtering. The first pass of retrieval usually returns twenty or thirty candidate passages. A second, more careful pass re-reads them and keeps the best five or six for the model to actually use. It also throws out passages that are too old, from the wrong document type, or that the asker isn't allowed to see. This step is invisible to users, but it's where a lot of accuracy comes from.

You can judge all of this without being technical. Ask the vendor or the team building it: show me the exact passages the system retrieved for these ten questions. If the passages are the ones a knowledgeable employee would have opened, retrieval is working. If they're from the wrong documents, the wrong year, or cut mid-sentence, it isn't — and no amount of model tuning will compensate.

The evaluation habit: your 100-question test set

You cannot improve what you don't measure, and you cannot trust what you don't test. Before a RAG assistant goes anywhere near real users, build a test set: roughly a hundred real questions your team actually gets asked, each paired with the answer a knowledgeable human would give and the documents that answer should come from.

This is the single most valuable artifact in the whole project. It costs a few days of a senior person's time — they already know the questions and the answers — and it is what makes every later decision measurable instead of a matter of opinion. Run every candidate setup against the test set. Score each answer simply: right, wrong, or "couldn't find it." A system that says "I don't know" when the answer genuinely isn't in the documents is behaving correctly; a system that invents an answer is failing, even if the answer sounds plausible.

Keep the test set alive after launch. Every time the assistant gives a wrong answer in production, add that question to the test set with the correct answer. Every month, re-run the full set and check that accuracy hasn't slipped. Documents change, new products launch, old policies expire — the test set is how you catch the assistant quietly going stale. Companies that skip this step always end up with the same story: it worked great at launch, and nobody noticed when it stopped working great.

One more thing the test set gives you: a fair way to compare options. If you're choosing between two vendors or two approaches, run both against your hundred questions and look at the scores. Marketing claims evaporate; the test set doesn't lie.

Access control: who may see what

This is the section most RAG demos skip and most real deployments regret skipping. Your company documents are not all meant for everyone. HR letters, salary structures, finance workings, legal correspondence, and customer complaints with names in them all sit in the same shared drives as the SOPs and manuals. A chatbot that can read everything will eventually answer a question with something it should never have revealed.

The rule is simple: the assistant must only ever retrieve documents the person asking is allowed to see. In practice this means the retrieval step filters by permission before the model ever sees the passages — not after, and not by asking the model nicely to keep secrets. A manager asking about leave policy should get the manager's version; a junior asking the same question should get the employee version. The same question, different answers, based on who asked. That is correct behaviour.

Setting this up is mostly an organisational task, not a technical one. Somebody has to go through the document sources and mark which groups may read which folders. Start with a simple assumption and adjust: most day-to-day documents can stay open to everyone, while a smaller set — HR documents for HR and managers, finance documents for finance, customer data for the support team — needs simple rules. Do this classification before launch, not after an incident forces it.

Also decide what happens with customer data in tickets and emails. Past support tickets are some of the most valuable training material for a knowledge assistant — they contain real questions and the answers that actually worked. But they also contain customer names, phone numbers, and order details. Either strip personal details from tickets before they're added, or restrict ticket-derived answers to the support team. Both approaches work; picking neither is what causes trouble.

Keeping documents fresh without heroics

A knowledge assistant is only as good as its newest document. The moment your price list, your leave policy, or your product manual changes, the assistant starts giving yesterday's answers with today's confidence. Freshness is not a one-time migration task; it's an ongoing habit, and it has to be cheap enough that people actually do it.

The pattern that works is boring: one owner per document source. The person who owns the leave policy owns the leave-policy document in the knowledge base. When they update the policy, updating the assistant's copy is part of the same task — not a separate ticket for later. If updating the assistant takes more than a few minutes, the process is broken and needs simplifying, because anything harder than that will be skipped under deadline pressure.

Versioning matters too. When a document is updated, the old version should be retired from the searchable set, not left sitting next to the new one. Two versions of the same policy in the knowledge base is how the assistant starts quoting the expired one — and it will always be the expired one that sounds more authoritative, because it was written more carefully. Keep one current version searchable; archive the rest where humans can still find them but the assistant can't.

Schedule a quarterly review. Once every three months, somebody with authority walks through the document list and confirms each source is still current and still has an owner. It takes an afternoon. This single habit prevents the slow rot that kills most knowledge assistants in their second year. And add a feedback button to the assistant itself — "this answer looks outdated" — so users become your early-warning system instead of quietly losing trust.

A worked example: the field-service knowledge assistant

Concrete beats abstract, so here's a full example: a knowledge assistant for a company with forty field technicians who install and service equipment at customer sites. Today, when a technician hits an unfamiliar fault code, they call the senior engineer. The senior engineer answers the same twenty questions every week, usually while standing on another site.

The source documents: installation manuals for each equipment model, the troubleshooting guide, the safety procedures document, the warranty terms, and two years of past service tickets with the fault codes, symptoms, and fixes. That's a few hundred documents — a perfect starting size. Not the entire shared drive; just the documents technicians actually reach for.

The test set: the senior engineer writes down the hundred questions they get asked most — "What does fault code E-47 mean on the 2023 model?", "What's the torque setting for the compressor bolts?", "Is this failure covered under the standard warranty?" — each with the correct answer and the document it comes from. This takes them two or three days, and it doubles as the acceptance test: the assistant doesn't launch until it scores well.

The rollout, week by week: weeks one and two, the pipeline — documents collected, chunked by section, tagged with model numbers and dates, access rules set so technicians see technical documents but not the finance workings that share the drive. Weeks three and four, evaluation — run the hundred questions, fix the retrieval misses, add the missing troubleshooting notes the senior engineer keeps in a personal notebook. Week five, pilot with five technicians, with a big visible disclaimer that answers should be double-checked against the manual for safety-critical steps. Weeks six to eight, expand to all forty, with the feedback button collecting every "this looks wrong" for the monthly test-set update.

What changes on the ground: the average time to resolve an unfamiliar fault drops because the technician gets the right manual page in seconds instead of waiting for a callback. The senior engineer gets their afternoons back. New hires can become productive faster because the company's accumulated field knowledge is finally searchable — they ask the assistant instead of waiting for a senior to free up. Nothing about this required inventing new technology — it required organising documents, writing a test set, and rolling out in stages.

A realistic rollout timeline

Anyone who promises a company-wide knowledge assistant in a week is selling you the demo, not the deployment. Here's what a realistic timeline looks like for a first RAG assistant over a few hundred documents, done properly.

Weeks one to two are document work: gathering the sources, deciding what goes in and what stays out, assigning owners, and setting access rules. This is the least glamorous phase and the one most likely to be skipped. Don't skip it — everything downstream depends on it.

Weeks three to four are pipeline and evaluation: chunking the documents, adding metadata, building the hundred-question test set, and iterating until accuracy is good. Expect the first run against the test set to be disappointing. That's normal; the test set is doing its job. Each round of fixes — better chunking here, a missing document there — visibly moves the score.

Weeks five to six are the pilot: a small group of real users, a visible disclaimer, and a feedback channel. Watch what people actually ask, because it will differ from what you predicted. Add the surprises to the test set. Fix the access-control gaps the pilot reveals — it always reveals at least one.

Weeks seven to eight are the wider rollout, plus the handover of the maintenance habits: document owners know their job, the quarterly review is on someone's calendar, and the test set re-runs monthly. Done right, a knowledge assistant goes live in about two months and earns its place — not because the AI is impressive, but because finding the right answer finally takes seconds instead of a phone call.

When RAG is the wrong tool

Honesty requires saying when not to build this. RAG is the wrong tool when the knowledge doesn't exist in documents at all — if the expertise lives entirely in people's heads and was never written down, no retrieval system can retrieve it. Write the documents first; the assistant comes second.

It's the wrong tool when the questions are few and stable. If your team gets asked the same fifteen questions and the answers never change, a well-organised FAQ page or a one-page cheat sheet beats a chatbot. Don't build a pipeline to answer questions a printed page could answer.

It's the wrong tool when answers need judgement rather than lookup. "Should we extend this customer's credit?" is not a retrieval question — it's a decision that needs context, experience, and accountability. An assistant can fetch the credit policy; it shouldn't make the call.

And it's the wrong tool when something off the shelf already does the job. If a ₹5,000/month SaaS tool already answers these questions for your team, use that instead of building your own pipeline. Building makes sense when the knowledge is yours — your SOPs, your tickets, your manuals — and no generic product can hold it. Buy the commodity; build only the thing that's genuinely yours.

The practical checklist

Start with your fifty to a hundred most-asked questions and a few hundred source documents — not your entire drive. Write the test set before you build anything, and measure answer accuracy against human-reviewed answers before expanding.

Spend your effort on the pipeline: chunk documents by section, attach metadata like owner and date, and combine keyword and meaning-based matching. Verify retrieval by reading the actual passages the system pulls for sample questions — if a knowledgeable employee would have opened the same pages, you're in good shape.

Enforce access controls from day one so the assistant only retrieves what the asker is permitted to see, and classify the sensitive subset of your documents before launch. Put a human review loop on anything customer-facing, and strip personal details from tickets or restrict ticket answers to the support team.

Keep documents fresh with one owner per source, retire old versions from the searchable set, run the test set monthly, and review the document list every quarter. Pilot with a small group first, watch what they actually ask, and expand only when the scores hold up.

The honest bottom line

RAG works, but it works because of unglamorous discipline, not because of AI magic. Good documents, a real test set, honest access control, and a maintenance habit — that's the whole secret, and none of it requires a research lab. Companies that do these four things end up with an assistant their teams genuinely rely on. Companies that skip them end up with a chatbot nobody trusts. The technology is the easy part; the checklist is the product.