Retrieval-augmented generation (RAG) gives a model selected source material to use when answering a question. A private RAG system keeps each processing step within the approved data boundary. That includes parsing, OCR, embeddings, retrieval, generation, and evaluation.
This reference design assumes an internal document collection with known access rules. It does not assume that every document should be available to every employee.
Define the document contract
For each source, identify who owns it, which version is authoritative, who may read it, and how changes are detected. Preserve a stable document identifier through ingestion, indexing, and deletion.
An illustrative record might contain:
{
"document_id": "manual-104",
"version": "3",
"source_uri": "internal://manuals/manual-104",
"page": 12,
"allowed_groups": ["service-team"],
"text": "A passage extracted from the source document."
}These fields describe a possible application schema, not a universal format. Derive permissions from the source system and trusted identity data. Do not let an uploaded document or the caller assign itself access groups.
Parse before choosing a chunk size
Inspect extracted text from actual files. Multi-column PDFs, scanned forms, tables, and repeated headers can produce poor passages even when extraction reports success.
Keep headings with the text they qualify. Preserve table structure when cells depend on column labels. Record page or section references so the user can inspect the source. Treat OCR quality as an evaluated component rather than a hidden preprocessing step.
Start with a simple chunking approach, then adjust using retrieval failures. A fixed token length is a parameter to test, not a guarantee of useful context.
Apply authorization before model access
Retrieval should return only passages the authenticated user may access. Enforce that restriction in the retrieval query or in a trusted authorization stage before any passage is sent to a reranker or generation model.
The order matters. Retrieving everything, generating an answer, and hiding its citations afterward does not remove the information from the answer. A shared answer cache can also cross access boundaries unless its scope accounts for the user’s effective permissions and source version.
For multi-tenant systems, evaluate tenant isolation as an explicit acceptance requirement. Include attempts to reference another tenant’s document identifier.
Evaluate retrieval separately
Build a set of questions with expected supporting passages. Include exact identifiers, paraphrases, rare terminology, conflicting document versions, and questions with no answer.
Measure whether the expected evidence appears in the retrieved set before asking whether the generated answer sounds good. Compare keyword retrieval, embedding-based retrieval, and a combined approach when appropriate. Keep the test conditions stable enough to identify what changed.
If the correct passage is absent, switching the generator may not address the underlying failure.
Generate answers with checkable references
Provide selected passages with stable reference labels. Ask for an answer grounded in those passages and a clear statement when the evidence is insufficient.
Validate that cited labels exist and point to the material supplied to the model. Then evaluate whether that material actually supports the claim. A syntactically valid citation can still be attached to an unsupported sentence.
Treat retrieved text as untrusted content. A document that tells the assistant to ignore its rules should not gain authority over the application. Keep any tool actions separately authorized.
Design updates and deletion
When a document changes, update its passages, embeddings, and index entries together or use a versioned cutover. Avoid temporarily combining old and new passages as if they were one authoritative source.
When access changes, the retrieval policy and relevant caches must reflect it. When a document is removed, track deletion through the index, source copy, cache, and applicable retention process. Define what happens to backups separately.
Fit the pilot to a real workflow
Choose one collection, one user group, and a question set they recognize. Measure answer support, source retrieval, latency, refusals, and corrections. Have subject-matter reviewers assess the difficult examples.
The evaluation guide explains how to structure the comparison. A document search engagement can apply the design to your files and identity systems.