You have probably used ChatGPT and noticed that it sometimes gets things wrong, confidently. That is because it answers from what it learned during training - which ended months ago, does not include your company's documents, and sometimes produces plausible-sounding but incorrect information.
For personal curiosity, that is fine. For business decisions based on your own data, it is a problem.
RAG - Retrieval-Augmented Generation - is the fix. Here is what it actually means.
What RAG is in plain words
RAG is a technique that gives an AI access to a specific set of documents before it answers a question.
When you ask a question, the system first searches through your documents to find the relevant sections. It then hands those sections to the AI along with your question. The AI answers based on what it just read - not what it learned during training - and tells you where the answer came from.
The result: answers grounded in your actual documents, with citations. If the document says something, the AI can say it. If the document does not say something, a well-built RAG system says "I don't know" instead of guessing.
Why it beats a generic chatbot for business
A generic chatbot trained on the open internet cannot answer questions about your contracts, your internal policies, your product catalog, or your support tickets. It does not have that information.
A RAG system built on your documents can. And it answers with a reference to the source, so you can verify it.
The difference matters most when the stakes are real: legal review, compliance checks, customer support, procurement decisions. In those situations, a confident wrong answer is worse than no answer.
Real examples of where it works
Contracts and legal documents. "What are the payment terms in our agreement with Supplier X?" The system reads your contracts and returns the exact clause, with a reference.
Internal policies. "What is our leave policy for new employees?" Instead of asking HR, staff ask the system - which reads the HR handbook and returns the relevant section.
Product catalogs. A sales team that works across hundreds of SKUs can ask the system in plain language and get accurate specs with a source reference.
Support tickets and knowledge bases. A support team that handles repetitive questions can route common queries through the system, which pulls from past resolved tickets and documentation.
The data-safety angle
One of the most common questions: does my data go to OpenAI? Does it leave our systems?
It depends on how you build it - and this matters.
Many off-the-shelf tools do send your documents to a third-party API. The document goes to the cloud, the API processes it, and the answer comes back. For some businesses and some data types, that is acceptable. For others - contracts, medical records, HR data, financial information - it is not.
A properly built RAG system can run entirely on your infrastructure, or on-device, so your documents never leave. We have built fully on-device AI extraction before (see Kept, our iOS app). The same principles apply to document RAG.
Self-hosted models have improved dramatically. The gap in quality between a self-hosted model and a cloud API has narrowed enough that for many use cases, the privacy benefit of self-hosting is worth it.
What good vs bad input data means for results
RAG quality depends heavily on input quality. This is not a disclaimer - it is a practical planning point.
Good input: consistently formatted documents, clear language, minimal contradictions, decent file quality if scanned.
Bad input: scanned PDFs with poor OCR, documents written inconsistently, contradictory information across files (which happens in every organisation), extremely long documents with no clear structure.
A well-built system handles bad input better than a poorly built one - but no system produces good answers from bad data. Part of what the sprint assessment covers is an honest look at your data quality before you commit to a build.
When RAG is and is not the right tool
RAG works well when:
- You have a specific set of documents you want to query
- Questions have answers that exist in those documents
- You need source references, not just an answer
- Your data is text-based and reasonably organised
RAG is not the right tool when:
- The task is numerical analysis or forecasting (different problem)
- You need real-time data (RAG works on a fixed corpus)
- Your "documents" are actually complex structured data (database, not documents)
- The question cannot be answered by reading - it requires reasoning from first principles
Knowing which tool fits is part of what the scoping sprint delivers.
If RAG sounds like it might fit a problem you have, start with the sprint - it will tell you if it will work on your data before you commit a budget.