Internal knowledge assistant with RAG: when it works and when it does not

An assistant does not repair a weak knowledge base

In a growing company, the answer to 'How do we approve a discount?' may sit in Notion, an old PDF policy, a Slack thread and the sales director's memory. People lose time searching, while new joiners interrupt the same experts with the same questions. An internal AI assistant sounds like the obvious answer: ask in natural language and receive a concise response with a source.

The risk is that a fluent response may use the wrong version, combine rules from two countries or reveal HR information to someone without access. RAG does not decide which knowledge is true. It retrieves pieces from the sources available to it and supplies them to a model as context. Disorder at the input becomes more relevant-sounding disorder at the output.

The project therefore starts with one operational domain, defined users, repeated questions and authoritative sources—not with a vector database. When the correct way of working is not documented, the first intervention is an SOP and an owner, not a chatbot.

RAG shortens the distance to knowledge. It does not create ownership, freshness or access rights for you.

How RAG works in operational terms

Retrieval-Augmented Generation joins two processes: search and generation. During indexing, documents are extracted, cleaned, split into meaningful chunks, described with metadata and added to a searchable index. When a question arrives, the system retrieves relevant chunks, adds them to the prompt and asks the model to formulate an answer that should point back to the evidence.

Semantic search can match meaning even when the user and the document use different words. That helps when someone asks about 'holiday for a remote employee' while the policy says 'paid time off' and 'telework'. Names, contract IDs, product codes and exact phrases often benefit from keyword or hybrid search. A sound retrieval design uses the query type rather than treating vector search as the answer to every lookup.

Every stage needs an operational owner. The process owner approves the content. The system owner maintains connectors, indexing and access. Domain reviewers decide whether the answer and evidence are acceptable. Without these roles, the assistant may run technically while no one remains accountable for the knowledge.

Choose the simplest approach that solves the problem

'Build RAG' is not a business requirement. The requirement might be for a new employee to find the current expense policy within two minutes, or for a service team to check a procedure without interrupting an operations manager. Compare options by knowledge volume and variety, rate of change, access rules, need for citations and the capacity available for maintenance.

Work from left to right in the table. Move to a more complex option only when the simpler one cannot answer the real questions or meet governance requirements. Managed products remove much of the ingestion and retrieval engineering. Custom RAG buys control, but it also creates more components to observe, secure and maintain.

Use a knowledge-readiness gate before prototyping

Score the first corpus from zero to two on five dimensions. Authority: is there one approved source for each rule? Freshness: does it have an owner, version and review date? Structure: can it be split into chunks that remain meaningful alone? Metadata: do you know its department, country, audience, type and status? Access: can you enforce the same rights in retrieval?

A score of 8–10 provides a reasonable base for a limited pilot. At 5–7, repair the weakest dimensions and use a small corpus. Below 5, the prototype will mostly measure the quality of the disorder. This is a diagnostic, not an industry standard; weight it according to the cost of error in the domain.

Do not index the entire shared drive just in case. Start with 50–200 approved pages from one domain, remove duplicates and archived versions, and add source URL, title, owner, last reviewed, access group and effective date. A document without an owner may be a clue for a person, but it should not automatically become an authority for the assistant.

Scanned PDFs, complex tables and slide decks need extraction QA. Inspect the text that was actually extracted, not only the original file. A missing column, header or footnote can change meaning before the model ever sees the content.

• An authoritative source and business owner for every policy or procedure.
• One active version and a visible archive rule.
• Self-contained sections with descriptive headings rather than one monolith.
• Metadata for version, effective date, locale, audience and access group.
• An update trigger when a process, system, product or regulation changes.

Evaluate retrieval separately from the polished answer

When the assistant fails, first ask whether it found the correct evidence. If it did not, prompt changes rarely solve the root problem. Inspect parsing, chunk boundaries, metadata filters, query rewriting, keyword/vector balance and ranking. When the correct passage was retrieved but the answer is wrong, then review the instructions, context assembly and model.

Create a golden set of 50–100 real questions: simple, ambiguous, cross-document, stale, out-of-scope and phrased in users' own words. Record the expected source, permitted audience, acceptable answer and required refusal for each. A domain expert approves the set; the technical team should not define business truth on its own.

Track retrieval hit rate, evidence precision, citation correctness, grounded-answer rate, correct refusals, stale-answer rate, access violations, latency and cost per accepted answer. One average accuracy score hides the dangerous failures. Access to a restricted document is a hard fail even when the overall result looks excellent.

Store confirmed corrections as regression cases. A feedback button without a queue, owner and response time does not improve the system. Each issue should lead to one of four explicit changes: source content, metadata and permissions, retrieval configuration or answer instruction.

First test whether the right evidence was retrieved. Then test whether the model used it correctly.

Permissions and corpus protection

The search layer must not expand access. If an employee cannot open a compensation document in the source system, they must not receive a chunk or summary from it. Apply permission filters before retrieval rather than asking the model to keep a secret. Where departments, clients or countries share infrastructure, use document-level security trimming and test every role separately.

OWASP identifies data leakage, cross-context leakage and data poisoning as material risks in vector and embedding systems. Use an allowlist of sources, validate files during ingestion and do not automatically move external attachments into a trusted corpus. Hidden instructions in a CV, ticket or client file are not policy and must not change assistant behaviour.

Log a query ID, user role, applied filters, document IDs, retrieval scores, model and configuration version, and feedback outcome. Minimise sensitive content in logs and define retention and access. A vector store is not a safe zone by default; it contains a derivative form of company knowledge and deserves the same governance attention.

• Inherited identity or a reliable mapping between users and access groups.
• Permission-aware retrieval before context reaches the model.
• Approved ingestion connectors and quarantine for new or external files.
• Tests for cross-team, cross-client and archived-content leakage.
• A defined delete and re-index process when access or versions change.

When RAG will not solve the problem

RAG is unnecessary for a one-off question about one short document; a file upload or ordinary search may be enough. It is also the wrong layer for an exact live status, inventory value, price or balance. Those answers should come from the system of record through a query or governed tool. Last week's document is not a reliable substitute for today's CRM or ERP record.

RAG is not a cure for weak policies, conflicting SOPs or decisions that were never documented. Nor is it a substitute for fine-tuning or prompt design when the actual goal is a consistent output format, tone or specialised behaviour rather than access to changing knowledge.

For legal, financial, employment or safety decisions, retrieval can surface the relevant policy, but an authorised person should assess the case. The phrase 'according to document X' does not turn general guidance into a professional decision or remove the need for escalation.

• There are no authoritative sources or accountable knowledge owners.
• The corpus is small enough for ordinary search to solve the task.
• The answer is an exact transaction value from a structured system.
• Permissions cannot be enforced at document or chunk level.
• There is no domain reviewer, eval set or correction process after launch.

Worked example: a 30-day pilot for a service team

Consider a fictional 35-person engineering services firm. Project coordinators repeatedly interrupt senior engineers with questions about handover, site access, report templates and warranty procedures. The shared drive contains 480 files, but the first pilot does not index everything. The team selects the handover domain, makes the operations lead its owner and approves 120 active pages.

In week one, it removes 34 duplicate or archived files, adds metadata and checks the extraction of PDF tables. In week two, it builds 70 golden questions from real chat searches and interviews. They include missing information, old terminology, a different site type and roles without access.

In week three, a managed retrieval prototype answers only when it can show at least one visible source link. Illustrative go/no-go criteria are: the correct document appears in the top results for at least 90% of the golden questions; zero access violations; 95% citation correctness across answers the system chooses to provide; and a correct refusal for every question outside the corpus. These are example thresholds, not a universal benchmark.

In week four, five coordinators use the assistant alongside the current process. The team measures accepted answers, time to a verified answer, interruptions to senior engineers, corrections and cost. The decision after the pilot is one of three: expand to a second domain, complete more content cleanup or stop. A demo that answers five selected questions beautifully is not a fourth option.

The example, volumes and thresholds are illustrative. Design your pilot around real questions, roles, sources and cost of error.

The next decision: a knowledge-readiness review

Bring together the process owner, knowledge owner, IT or security and two real users. Select one domain and review the questions, authoritative sources, versions, metadata, permissions and correction loop. The outcome should be one of three decisions: ready for a managed pilot, content cleanup first, or RAG is not the appropriate approach.

1→10 can include this review in a Systems Audit or AI implementation scope. We map the knowledge flow, select the simplest viable architecture, define the corpus and permission model, and build a measurable pilot backlog. The objective is not another chatbot. It is fewer interruptions and faster decisions with visible evidence.