RAG Architecture for Enterprise: Beyond the Tutorial

The Tutorial vs. Reality Gap

Every RAG tutorial follows the same pattern:

1. Split documents into chunks
2. Generate embeddings
3. Store in a vector database
4. Query with semantic search
5. Pass context + question to an LLM
6. Return the answer

This works for demos. It breaks catastrophically in production.

Enterprise RAG systems deal with messy documents, adversarial queries, stale data, access controls, hallucination detection, and user expectations shaped by years of Google-quality search. The gap between a tutorial RAG and a production RAG is the same as the gap between a TODO app and an enterprise CRM.

Architecture Overview

A production enterprise RAG system has five layers, not two:

┌─────────────────────────────────────────┐
│           Query Understanding           │
│  (intent classification, query rewrite) │
├─────────────────────────────────────────┤
│             Retrieval Layer             │
│  (hybrid search, reranking, filtering)  │
├─────────────────────────────────────────┤
│           Context Assembly              │
│  (deduplication, ordering, truncation)  │
├─────────────────────────────────────────┤
│            Generation Layer             │
│  (prompt construction, LLM call, cache) │
├─────────────────────────────────────────┤
│         Answer Verification             │
│  (grounding check, citation, feedback)  │
└─────────────────────────────────────────┘

Let's walk through each layer.

Layer 1: Query Understanding

Raw user queries are messy. "what's our policy on remote work for contractors in Germany" contains multiple intents, a geographic filter, and an employment-type filter. Passing this directly to a vector search returns mediocre results.

What production systems do:

Intent Classification

Before searching, classify the query:

• Factual lookup — "What is the PTO policy?" → single-document retrieval
• Comparative — "How does our UK policy differ from US?" → multi-document retrieval with comparison logic
• Procedural — "How do I submit an expense report?" → step-by-step retrieval with ordering
• Analytical — "What are our biggest cost centers?" → may need structured data, not documents

Query Rewriting

Transform the user query into a better search query:

// Original: "what's our policy on remote work for contractors in Germany"
// Rewritten queries (multi-query retrieval):
const queries = [
  "remote work policy contractors",
  "Germany employment regulations contractors",
  "work from home policy international contractors",
];

Multi-query retrieval generates 2–4 reformulations of the original question and retrieves results for each, then merges and deduplicates. This consistently outperforms single-query retrieval by 15–25% on our benchmarks.

Metadata Extraction

Extract structured filters from natural language:

// Extracted from: "Q3 2025 financial reports for the APAC region"
const filters = {
  documentType: "financial_report",
  timePeriod: { quarter: 3, year: 2025 },
  region: "APAC",
};

These filters are applied as pre-filters on the vector store, dramatically reducing the search space and improving precision.

Layer 2: Retrieval

This is where most tutorials live — but production retrieval is significantly more sophisticated.

Hybrid Search

Pure vector (semantic) search misses exact matches. Pure keyword (BM25) search misses semantic similarity. Production systems use both:

// Hybrid retrieval: combine semantic + keyword scores
const semanticResults = await vectorStore.search(embedding, { topK: 20 });
const keywordResults = await bm25Index.search(query, { topK: 20 });

// Reciprocal Rank Fusion (RRF) to merge results
const merged = reciprocalRankFusion(semanticResults, keywordResults, {
  semanticWeight: 0.6,
  keywordWeight: 0.4,
});

In our benchmarks, hybrid search with RRF consistently outperforms either approach alone by 20–30% on recall@10.

Reranking

The initial retrieval returns 20–50 candidates. A reranking model (cross-encoder) re-scores each candidate against the original query with much higher accuracy than the initial bi-encoder retrieval:

// Rerank top 30 results down to top 5
const reranked = await crossEncoder.rerank(query, candidates, { topK: 5 });

Reranking is the single highest-impact improvement you can make to a RAG pipeline. It typically improves answer quality by 15–30% with minimal latency cost (50–100ms for 30 candidates).

Access Control Filtering

Enterprise documents have access controls. A junior analyst shouldn't see board meeting minutes. A contractor shouldn't see salary data.

This must be enforced at the retrieval layer, not the generation layer:

const results = await vectorStore.search(embedding, {
  topK: 20,
  filter: {
    accessLevel: { $lte: user.clearanceLevel },
    department: { $in: user.departments },
  },
});

Layer 3: Context Assembly

You have 5–10 relevant chunks. Now you need to assemble them into a coherent context for the LLM.

Deduplication

Multiple chunks from the same document section often appear in results. Deduplicate by source document and section, keeping the highest-ranked version.

Contextual Ordering

Order chunks by relevance, but also by logical sequence. If chunks come from the same document, preserve their original order. Users expect answers that follow a logical flow, not a random collage of excerpts.

Token Budget Management

LLMs have context window limits. Even with 128K+ windows, more context isn't always better — irrelevant context degrades answer quality.

// Allocate token budget
const TOKEN_BUDGET = 4000; // for context
let usedTokens = 0;
const selectedChunks = [];

for (const chunk of rankedChunks) {
  const chunkTokens = countTokens(chunk.text);
  if (usedTokens + chunkTokens > TOKEN_BUDGET) break;
  selectedChunks.push(chunk);
  usedTokens += chunkTokens;
}

Our experiments show that 3,000–5,000 tokens of highly relevant context outperforms 20,000 tokens of loosely relevant context in answer accuracy.

Layer 4: Generation

Prompt Construction

The system prompt matters enormously. A well-constructed prompt includes:

1. Role definition — "You are a knowledge assistant for [Company]. Answer based strictly on the provided context."
2. Context block — the assembled chunks with source metadata
3. Instructions — how to handle ambiguity, missing information, conflicting sources
4. Output format — structured response with citations

You are a knowledge assistant for Acme Corp.
Answer the user's question using ONLY the provided context.

Rules:
- If the context doesn't contain enough information, say so explicitly
- Cite sources using [Source: document_name, page X]
- If sources conflict, present both viewpoints
- Never fabricate information not present in the context

Context:
[Source: remote-work-policy-2025.pdf, page 3]
"Contractors based in EU countries are eligible for remote work
arrangements subject to local employment regulations..."

[Source: germany-compliance-guide.pdf, page 12]
"German labor law requires contractors working remotely to..."

Question: {user_query}

Response Caching

Identical or near-identical questions appear frequently. Cache responses keyed by a hash of the query + retrieved chunk IDs:

const cacheKey = hash(normalizedQuery + chunkIds.sort().join(','));
const cached = await cache.get(cacheKey);
if (cached && cached.age < MAX_CACHE_AGE) return cached.response;

Caching reduces LLM API costs by 30–50% in enterprise deployments where teams ask similar questions.

Model Selection

Not every query needs GPT-4 or Claude Opus. Route queries by complexity:

• Simple factual lookups → smaller, faster model (Haiku, GPT-4o-mini)
• Complex analytical queries → larger, more capable model (Opus, GPT-4o)
• Multi-step reasoning → chain-of-thought with the strongest available model

This reduces average latency by 40% and costs by 50% compared to routing everything through the largest model.

Layer 5: Answer Verification

The most overlooked layer — and the one that separates production systems from prototypes.

Grounding Check

After the LLM generates an answer, verify that every claim in the answer can be traced back to the provided context. This catches hallucinations before they reach the user:

// Use a smaller model to verify grounding
const groundingCheck = await verifier.check({
  answer: generatedAnswer,
  context: providedChunks,
  question: originalQuery,
});

if (groundingCheck.confidence < GROUNDING_THRESHOLD) {
  return { answer: generatedAnswer, warning: "Low confidence — verify with source documents" };
}

Citation Extraction

Every factual claim should link back to its source document and page/section. This lets users verify answers and builds trust in the system.

Feedback Loop

Users can flag incorrect answers. This feedback feeds back into:

• Fine-tuning the reranker
• Adjusting chunk boundaries for problematic documents
• Identifying gaps in the document corpus

Document Ingestion: The Unglamorous Foundation

The quality of your RAG system is bounded by the quality of your document processing pipeline.

Chunking Strategy

Fixed-size chunks (500 tokens) are the default in tutorials. They're terrible for production.

Better approaches:

• Semantic chunking — split at paragraph boundaries, keeping semantic units intact
• Hierarchical chunking — store both fine-grained chunks and parent sections, retrieve the parent when multiple child chunks match
• Sliding window with overlap — 400 tokens with 100-token overlap, better than no overlap but still naive

Document-Specific Processing

PDFs with tables need table extraction. Slides need text + visual layout. Spreadsheets need column header context. Legal documents need section hierarchy preservation.

One-size-fits-all ingestion pipelines produce one-size-fits-none results.

Freshness

Enterprise documents change. Policies get updated. Reports are published quarterly. Your ingestion pipeline needs:

• Change detection (hash comparison, modification timestamp)
• Incremental re-indexing (update changed documents, not the entire corpus)
• Version tracking (which version of a document was used to generate an answer?)

Performance Benchmarks

From our production deployments:

| Metric | Tutorial RAG | Production RAG | |---|---|---| | Answer relevance (human eval) | 62% | 89% | | Hallucination rate | 18% | 3% | | Average latency (p50) | 2.1s | 1.4s | | Average latency (p99) | 8.5s | 3.2s | | Cost per query | $0.08 | $0.03 |

The production system is more accurate, faster, and cheaper — because it routes queries intelligently, caches aggressively, and uses smaller models where possible.

When to Build vs. Buy

Build your own RAG when:

• Your documents contain sensitive/regulated data (healthcare, finance, legal)
• You need custom access controls that SaaS tools can't enforce
• Your domain requires specialized chunking or retrieval logic
• Query volume justifies the infrastructure investment (1,000+ queries/day)

Use a managed solution when:

• You're experimenting with RAG for a small team
• Your documents are general-purpose (public knowledge bases, FAQs)
• You don't have ML engineering resources to maintain the pipeline

What We Build

We've deployed enterprise RAG systems that process 50,000+ daily queries across document corpora spanning hundreds of thousands of pages — for healthcare organizations, financial institutions, and enterprise knowledge management.

If you're building a RAG system that needs to work at enterprise scale with real security, accuracy, and performance requirements, this is our specialty.