Retrieval-augmented generation (RAG) is a method that makes an AI answer a question by first retrieving the most relevant passages from a chosen body of documents, then generating a response grounded in those passages, with the sources attached. Instead of relying only on what a model memorised during training, it reads your material at the moment of the question. That grounding is what stops it from inventing answers about your business. The approach was introduced by Lewis et al. (2020), who showed that combining a neural retriever with a generative model improved accuracy on knowledge-intensive tasks and let the system point to the evidence behind its answers.
Why plain chatbots hallucinate
A language model on its own is a probability engine: it predicts fluent continuations of text. Ask it about your internal pricing, a past contract, or a niche policy, and it will produce a confident answer whether or not it has any basis for one. This tendency to generate fluent but unsupported content is well documented; a comprehensive survey by Ji et al. (2023) catalogues hallucination across natural-language generation systems and identifies a recurring cause: models generating beyond what their inputs actually support. A consumer chatbot has no access to your documents and no obligation to cite anything, so the risk is structural, not occasional.
What RAG changes
RAG adds two steps around the model.
Retrieve. Your documents are converted into a searchable index, typically vector embeddings that capture meaning rather than only keywords. When a question arrives, the system pulls the passages most relevant to it.
Ground and cite. Those passages are supplied to the model as context, and the answer is constructed from them, with links back to the source so a person can verify each claim. The result is an answer your team can trust because it can be checked, not merely believed. Lewis et al. (2020) demonstrated that this retrieval step both improved factual accuracy and made outputs more specific and attributable than a comparable model generating from memory alone.
RAG is only as good as the data beneath it
This is the caveat that decides success in practice. Retrieval over a messy, duplicated, contradictory document store returns messy, contradictory passages, and the generated answer inherits that. Good RAG is perhaps 20% model and 80% the quality, structure, and access model of the corpus it retrieves from. That is why classification and clean-up come first, a point we develop in Why most AI pilots fail. Retrieval quality, chunking strategy, and the freshness of the index matter more to real-world reliability than the choice of frontier model.
Where RAG fits in a company
For most organisations, RAG is the difference between an AI demo and an AI system people actually use. A well-built retrieval layer turns a scattered document estate into a copilot that answers in plain language, cites its sources, and stays current as documents change. It also constrains risk: because answers are grounded in approved material and are traceable, they are far easier to review, audit, and govern than free-form generation.
Building that retrieval layer, on a foundation that is genuinely ready for it, is part of our Knowledge Systems service.
References
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023). Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 1–38. https://doi.org/10.1145/3571730
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33, 9459–9474. https://arxiv.org/abs/2005.11401