- Retrieval-augmented generation helps when an assistant must rely on specific, changing organisational knowledge and show the source of each answer.
- Success depends on the quality and versions of the files, on retrieval, and on applying permissions before answering, not on the strength of the model alone.
- It does not guarantee every answer is right or justify opening every file; start with a limited scope and test against reference answers and clear acceptance criteria.
What does this approach actually do?
When an employee asks about an internal procedure, the system searches the sources that person is allowed to access, retrieves suitable passages, then asks the model to answer on that basis. This is the general idea of retrieval-augmented generation (RAG). Implementation details vary, but separating search from generation helps you see where an error happens: the search may fail to find the right text, or the model may misread a text that was found. [1, 2]
This does not mean the model now permanently knows all of the organisation’s policies, or that the files have automatically become correct facts. What goes into the index may be out of date, contradictory or unapproved. So the project needs a content owner, not just a technical team. Efficiently indexing a cancelled policy can produce a confident answer that is no longer valid for the work.
When do you need it, and when is something simpler enough?
Start by defining the questions you want to serve. If users need to find a particular document, a good search engine may be enough. If the answer comes from a structured record, such as the status of a request or an invoice, querying the system directly may be better than searching files. If users need to combine information spread across documents and get an answer grounded in them, this approach may fit.
This is a suggested design comparison, not a rule that the newest technology is always better. Test the value of the use case before expanding the architecture: How much time do staff spend now? What errors happen today? Are the documents usable in the first place? If the real problem is that the organisation has never approved a clear procedure, the assistant will not create one. Resolving unclear content comes before trying to automate answers about it.
How do you prepare files without losing their meaning?
Organise documents by owner, version, effective date and status, and link each to a source that can be opened. Check how text is extracted from scanned images, tables and footnotes, especially in Arabic and mixed-language documents. Splitting a file into chunks makes searching easier, but it can separate a condition from its exception or a heading from its table; so the team reviews representative samples rather than settling for a successful technical import. [2]
Set up a way to update content, retire old material and delete what may no longer be kept. If one policy is spread over several files, decide which prevails in case of conflict. A useful suggestion is for each answer to show the document name, its version or effective date and where the citation sits, so users can check the meaning instead of trusting a general link that does not reach the right paragraph.
How do you stop information reaching an unauthorised user?
Access permissions are applied at retrieval according to the user’s identity, and file permissions are kept up to date inside the index. It is not enough to tell the model in its instructions “do not reveal confidential information” after it has already received it. Documentation on document-level access control describes mechanisms to filter results before they are used in an answer, with availability and details varying between components and versions. [3]
Test accounts with different roles, including a user who has lost a permission they previously had. Ask about documents that should not appear for them, not only about those that are available. Also review temporary logs, conversations, saved snippets and backups, since data can travel beyond its original source. This is a joint design and security review, not a setting left to the developer alone once the pilot is finished.
How do you deal with wrong answers and instructions hidden in files?
The model may generate unsupported information or mix up sources, risks discussed in NIST’s 2024 Generative AI Profile, NIST AI 600-1. The system should be able to say that the evidence is insufficient, and distinguish retrieved text from a decision that needs a specialist. A citation does not prove that the sentence is actually based on it; the match must be tested. [4]
File contents are treated as reference data, not as instructions that give the system new powers. If a file contains text asking it to send secrets or ignore the rules, that text must not be acted on. Where the assistant is connected to external tools, the authority to answer is kept separate from the authority to carry out an action, with reviews suited to the impact. This separation matters when the project moves from questions and answers to creating transactions or changing records.
How do you build a test that reflects the organisation’s work?
Collect real questions with reference answers reviewed by process owners, and add cases with no answer, conflicting documents and missing permissions. Assess whether the right source was found, the accuracy of the answer, the suitability of the citation, refusal when appropriate, and time and cost. Do not just demo easy questions to management, and do not use the model’s own answer as the only reference for judging it.
Keep the test set as separate as possible from development examples, and review it whenever the model, the content or the search method changes. The pilot can start with a narrow scope, such as one approved service manual, before adding HR and contracts. This practical suggestion makes it easier to find the cause when something goes wrong, and gives the content team a chance to fix documents before the circle of users widens.
What does a hypothetical Gulf application look like?
A hypothetical organisation wants to help staff find out which documents an internal service request needs. It selects a set of approved policies and excludes personal employee files from the pilot. A user asks about an exception that is not mentioned anywhere, and it turns out the right reply is to say there is no basis for it and refer them to the person with authority, not to invent a convincing exception. The case illustrates a suggested test and does not claim any performance result for a particular system.
Before scaling up, the organisation reviews where data is processed, the provider’s terms, retention, logs, and the national and sector controls that apply in its country. Buying a cloud service or running the system locally does not achieve compliance automatically; both need appropriate design and controls. A clear channel is set up for reporting a wrong answer, linking the report to the source and context without posting sensitive data in public channels.
What goes into the pilot acceptance record?
Record the scope of documents, users and questions tested, the versions of the system and content, and the success and failure criteria. Log errors and classify them: missing source, unsuitable retrieval, wrong interpretation, unauthorised disclosure, or an answer without support. Classification helps choose the right fix, instead of treating every error by rewording the instructions.
Define what will not be allowed at launch, and who reviews questions that fall outside the scope. Do not write the limits in a technical file users never see; they need to understand when to use the assistant and when to go back to the responsible person or the original source. Also define the channel for correcting a document when the error lies in the content rather than the model.
Test a policy update, a revoked permission and a deleted document, and check that the results reflect them within the time the system was designed for. These are normal operating events; a demo that worked on one day does not prove quality will hold as files change. Acceptance is therefore tied to an operating and monitoring plan and a content owner. Agree, too, how to stop using the assistant or fall back to traditional search if a significant problem appears. A clear fallback stops staff relying on a doubtful answer simply because the assistant has become the usual channel. These are suggested implementation arrangements, to be adapted to the service’s impact and the organisation’s needs.
Frequently asked questions
Do we need to train a new model on our files?
Not necessarily. The approach retrieves content at question time and places it in the model's context. There may be other reasons to customise a model, but they do not follow automatically from the need to answer from documents.
Does it eliminate hallucinations completely?
No. Grounding answers in evidence may help, but errors in retrieval, interpretation and generation remain possible. Testing, review and clear limits of use are needed. [4]
Can we open all files to make the project easier?
Do not let ease of implementation override permissions. Define a suitable scope, apply access per user, and check that it keeps working as files and roles change.
How should two conflicting documents be handled?
The content owner reviews them and decides which one is in force. Until the conflict is resolved, the system should show it or decline to give a definitive answer, rather than letting the model choose silently.
What is a good first pilot?
Choose a limited set of approved documents and frequent, low-impact questions, and define reference answers and acceptance criteria before building the demo.
References
Recommended programmes
