A RAG demo can look convincing long before the system is ready for production. It can answer a few sample questions, cite a document, and impress a planning meeting. The hard part starts when real users ask messy questions, source documents change, permissions matter, costs rise, and leadership wants to know whether the answers can be trusted.
Retrieval-augmented generation is a system design problem, not just a model choice. The retrieval layer, source data, chunking strategy, metadata, access controls, prompt design, evaluation harness, monitoring, and support ownership all influence whether the feature helps or creates risk.
1. Data readiness
- Do the source documents answer the questions users will actually ask?
- Are documents current, deduplicated, owned, and organized by source of truth?
- Are there content types that need special handling, such as tables, PDFs, images, code, or policy documents?
- Is there a clear update process when source content changes?
RAG cannot reliably answer questions that the source corpus does not support. Teams often discover that the AI feature is exposing a knowledge-management problem. That is not a failure; it is useful evidence. But it needs to be handled before the system is treated as production-ready.
2. Retrieval quality
Chunking, metadata, embeddings, reranking, filters, and query rewriting all shape what the model sees. If retrieval is weak, prompt changes only hide the problem. A production system needs representative test questions and a way to inspect whether the right source material is being retrieved.
- Can the system retrieve the right source for common questions?
- Can it handle ambiguous, incomplete, or multi-part questions?
- Can users see or verify the source behind the answer?
- Are retrieval failures logged and reviewed?
3. Permissions and privacy
A RAG system should not make restricted information easier to access by accident. If the source corpus includes private, customer, employee, legal, financial, or confidential material, access boundaries need to be enforced before retrieval and generation.
- Can retrieval filter by role, user, group, tenant, project, or document permission?
- Are sensitive documents excluded or handled through a separate path?
- Does logging avoid storing secrets or private content unnecessarily?
- Is there a clear data retention and deletion approach?
4. Evaluation and regression testing
If the team cannot measure whether the system improved or regressed, it is not ready to scale. RAG evaluation should include retrieval quality, answer accuracy, refusal behavior, citation quality, latency, and cost. The eval set should include routine, edge-case, and failure-mode examples.
5. Observability and ownership
Production RAG needs a support model. Teams should know who reviews failures, who updates source content, who owns evals, who approves prompt and retrieval changes, and what happens when users report bad answers. Without ownership, the system quietly decays.
The readiness question is not whether a RAG system can answer a sample prompt. The real question is whether the team can operate it responsibly when the content, users, workflows, and risks become real.