RAG Architecture Diagram: Naive vs. Advanced RAG Explained

RAG reduces LLM hallucination rates significantly compared to vanilla prompting - because the model generates responses anchored to documents you provide, not patterns baked into training data. That's the case for RAG in one sentence. This guide describes what the architecture actually looks like: naive RAG (the version you start with), advanced RAG (the version production systems often need), and the decisions that separate them.

What is RAG architecture?

RAG stands for retrieval-augmented generation. It is a pattern for connecting a large language model to an external knowledge base so it can answer questions based on your specific documents, not just its training data.

The architecture has three layers:

Retrieval - find content from your knowledge base that is relevant to the user's query.

Augmentation - take the retrieved content and inject it into the prompt you send to the LLM.

Generation - the LLM reads the query plus the retrieved content and generates a grounded response.

Without retrieval, LLMs answer from training data alone - which does not include your internal documents, your product knowledge, or anything updated after the model's training cutoff. RAG fixes that. AWS Prescriptive Guidance on RAG documents how grounding LLM responses in retrieved external knowledge reduces hallucinations and increases response relevance, because the model is generating answers anchored to retrieved facts it can cite rather than drawing from statistical patterns in training data alone.

The architecture lives between your documents and your LLM. The complexity of that middle layer is what distinguishes naive from advanced RAG.

Naive RAG architecture

Naive RAG is the simplest implementation. Five components, one pass.

Naive RAG Architecture
─────────────────────────────────────────────────────────

Documents
    │
    ▼
[Chunker]           Split documents into fixed-size chunks
    │               (e.g. 512 tokens, 50-token overlap)
    ▼
[Embedding Model]   Encode each chunk as a vector
    │               (e.g. text-embedding-3-small, BGE-M3)
    ▼
[Vector Database]   Store and index all chunk vectors
                    (e.g. Pinecone, Qdrant, pgvector)

── At query time ──────────────────────────────────────

User Query
    │
    ▼
[Embedding Model]   Encode the query as a vector
    │               (same model as ingestion)
    ▼
[Vector Search]     Retrieve top-k most similar chunks
    │               (cosine similarity against the index)
    ▼
[Prompt Builder]    Inject retrieved chunks into prompt
    │               "Context: [chunks] Question: [query]"
    ▼
[LLM]               Generate a response based on context
    │               (GPT-4o, Claude 3.5, Gemini 1.5)
    ▼
Response to user

The five components in detail

Chunker - splits your source documents into pieces the embedding model can encode and the LLM can use. Typical starting point: 512 tokens with 50-100 token overlap. The overlap prevents answers that span a chunk boundary from being missed.

Embedding model - converts text into a vector (a list of numbers representing semantic meaning). Similar text produces similar vectors. At query time, the query is embedded with the same model. The vector database then finds chunks whose vectors are nearest to the query vector. Common choices: text-embedding-3-small (OpenAI, managed), BGE-M3 (BAAI, open source), E5-large-v2 (Microsoft, open source).

Vector database - stores the chunk vectors and their associated text. Supports fast approximate nearest-neighbor search (typically using HNSW index for performance). Common choices: Pinecone (fully managed), Qdrant (open source, self-hostable), Weaviate (open source, hybrid search built in), pgvector (PostgreSQL extension).

Prompt builder - assembles the final prompt sent to the LLM. The retrieved chunks are injected as context before the user's question. Most implementations include a system prompt instructing the LLM to answer from the provided context only, or to say "I don't know" if the answer is not in the retrieved material. How precisely those instructions are scoped - the citation format, refusal conditions, and output structure - is a prompt engineering decision that directly affects answer accuracy.

LLM - the generation layer. Takes the assembled prompt (query + retrieved chunks) and generates a response. The LLM's job at this stage is relatively constrained: it does not need to recall facts from training, only reason over what is provided.

When naive RAG is enough

Naive RAG handles most business RAG use cases. If your knowledge base is:

• Well-structured (clean headings, short paragraphs, consistent formatting)

• Relatively uniform in document type

• Queried with specific, focused questions

• Under 50,000 documents

...then naive RAG with well-tuned chunking will get you 80-85% accuracy on representative test queries. That is often sufficient for internal tools, FAQ bots, and first-generation customer support applications.

Advanced RAG architecture

Advanced RAG adds three stages around the core retrieval step: pre-retrieval (before the search), retrieval improvements (during the search), and post-retrieval (after the search, before the LLM).

Advanced RAG Architecture
─────────────────────────────────────────────────────────

Documents (ingestion pipeline)
    │
    ▼
[Chunker]           Split into chunks (size tuned per doc type)
    │
    ▼
[Embedding Model]   Encode chunks as dense vectors
    │
    ├──────────────────────────────────────────┐
    ▼                                          │
[Vector Index]      Dense semantic index        │
(e.g. HNSW)                                    │
    │                                          │
    │                                          ▼
    │                                   [Keyword Index]
    │                                   BM25 / inverted
    │                                   index for sparse
    │                                   retrieval
    │                                          │
    └────────────────────────────────────────┘

── At query time ──────────────────────────────────────

User Query
    │
    ▼
[Query Rewriter]    Reformulate the query for better retrieval
    │               Techniques: HyDE, step-back prompting,
    │               multi-query expansion
    │
    ▼
[Hybrid Retriever]  Run dense vector search + BM25 in parallel
    │               Merge results with Reciprocal Rank Fusion (RRF)
    │               Retrieve top-20 to 50 candidates
    │
    ▼
[Reranker]          Score all candidates with a cross-encoder
    │               (Cohere Rerank, BGE Reranker, MS MARCO)
    │               Select top-3 to 5 for the prompt
    │
    ▼
[Context Compressor] Strip irrelevant sentences from chunks
    │               Keep only sentences that directly address the query
    │
    ▼
[Prompt Builder]    Assemble final prompt with compressed context
    │
    ▼
[LLM]               Generate response with citations
    │               (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro)
    │
    ▼
[Citation Extractor] Map response claims to source chunks
    │
    ▼
Response + sources to user

Pre-retrieval: query rewriting

The core problem naive RAG does not solve: users write queries that are semantically distant from the way answers are written in documents.

A user asks: "What's the refund window?" The relevant document chunk says: "All return requests must be submitted within 30 days of the original purchase date." The vocabulary is completely different. Dense vector search struggles with this mismatch.

Three techniques improve pre-retrieval quality:

HyDE (Hypothetical Document Embeddings) - instead of embedding the query directly, ask the LLM to generate a hypothetical answer document ("a passage that would answer this question"). Embed the hypothetical document instead of the query. This moves the embedding into the answer space rather than the question space, dramatically improving semantic alignment with the actual knowledge base.

Step-back prompting - ask the LLM to reformulate the query at a higher level of abstraction before retrieval. "What is the company's return policy?" becomes "What are the general policies governing product returns at this company?" This retrieves broader context that often contains the specific answer.

Multi-query expansion - generate 3-5 rephrased versions of the query and run retrieval for each. Merge the result sets. This trades compute for coverage: if one phrasing misses the relevant chunk, another phrasing likely catches it.

Retrieval: hybrid search and reranking

Hybrid search - dense vector search (semantic similarity) is good at finding conceptually related content. Keyword search (BM25) is good at finding content with exact term matches. Neither alone is optimal. Hybrid search runs both in parallel and merges results using Reciprocal Rank Fusion (RRF), which ranks documents appearing high in both lists ahead of documents appearing only in one.

Weaviate includes hybrid search natively. For Pinecone and pgvector, BM25 search runs separately (typically via Elasticsearch or a simple inverted index) and the results are merged in application code before reranking.

Reranking - after retrieval, you have 20-50 candidate chunks. Vector similarity scores are a reasonable proxy for relevance but not a precise measure. A reranker is a cross-encoder model that takes each (query, chunk) pair and produces a precise relevance score. The top-3 to 5 chunks by reranker score are selected for the prompt.

Research using the BEIR benchmark - the standard heterogeneous IR evaluation suite - consistently shows that two-stage retrieval with cross-encoder reranking outperforms single-stage vector similarity search, with cross-encoder reranking achieving the highest nDCG@10 scores across domains. It is the highest-ROI upgrade from naive to advanced RAG.

Common rerankers: Cohere Rerank (managed API, strong out of the box), BGE-Reranker-v2-M3 (open source, self-hostable), cross-encoder/ms-marco-MiniLM-L-6-v2 (lightweight, good for latency-sensitive systems).

Post-retrieval: context compression and citation

Context compression - even after reranking, retrieved chunks often contain irrelevant sentences. If a 512-token chunk answers only 50 tokens worth of the user's question, the other 462 tokens are noise that reduces the signal-to-noise ratio for the LLM. A context compressor (often a small LLM or a cross-encoder scoring individual sentences) extracts only the sentences directly relevant to the query.

This matters because context window utilization is inefficient in most RAG systems. Most RAG implementations use only 20-40% of the available context window efficiently - the rest is overhead from irrelevant context. Compression lets you fit more useful signal into the same token budget, either improving accuracy or reducing cost.

Citation extraction - production RAG systems need to tell users where an answer came from. Citation extraction maps each claim in the generated response back to the chunk that sourced it. This is done either by prompting the LLM to include citations in its output (e.g., "[Source: policy-doc-section-3]") or by post-processing the response against the retrieved chunks.

Naive vs. Advanced RAG comparison

Architecture decisions that matter most

Chunk size and overlap

This is the most impactful decision in RAG architecture - more than the embedding model, more than the LLM. Wrong chunk size causes retrieval failures that no amount of reranking can fix.

Starting recommendations:

• General business documents: 512 tokens, 50-100 token overlap

• Legal or technical documents with dense information: 256-384 tokens, 10-20% overlap

• Conversational content, FAQs: 128-256 tokens, minimal overlap

• Long narrative documents: 512-1024 tokens, 100-200 token overlap

The diagnostic: run 20-30 representative test queries. For each failed answer, check whether the right chunk was retrieved (retrieval failure) or whether the right chunk was retrieved but the answer was wrong (generation failure). Retrieval failures usually point to chunking or embedding issues. Generation failures usually point to chunk quality or LLM instruction problems.

Embedding model selection

The embedding model determines the quality of your semantic index. A better embedding model means better retrieval without any other changes.

For most business RAG systems: start with text-embedding-3-small. If you need self-hosted for data privacy, BGE-M3 is the strongest open source option as of mid-2026.

Index type

HNSW (Hierarchical Navigable Small World) is the index type used by most production vector databases. It provides approximate nearest-neighbor search with strong recall at millisecond-range latency. For most RAG applications, HNSW is the right choice with no tuning needed.

Flat (exact) indexes guarantee recall but scale poorly beyond 1 million vectors. IVF (Inverted File Index) indexes trade some recall for speed at very large scale. For under 10 million documents, HNSW with default settings outperforms both alternatives.

When to use each approach

Use naive RAG when:

• Your knowledge base is under 50,000 well-structured documents

• User queries are specific and vocabulary-aligned with your documents

• Latency is critical (under 500ms requirement)

• You are validating the RAG approach before committing to infrastructure

• Cost per query needs to stay low

Use advanced RAG when:

• Naive RAG evaluation shows retrieval failures on more than 20-25% of test queries

• User queries are conversational, vague, or use different vocabulary than your documents

• Your knowledge base is large, varied, or frequently updated

• Answers need to be citable with source attribution

• Accuracy improvement of 10-15 percentage points justifies 2-4x higher per-query cost

The typical path: build naive RAG first, evaluate it on 50 representative questions, then add advanced components in order of impact. Reranking first (highest ROI, moderate latency cost). Hybrid search second. Query rewriting third. Context compression when you are optimizing for cost or context window utilization.

RAG is not a single architecture - it is a spectrum. The right point on that spectrum depends on your knowledge base, your query patterns, and your accuracy requirements. Most teams that start with advanced RAG on day one are over-engineering a system they have not yet validated. Most teams that stop at naive RAG have left 10-15 percentage points of accuracy on the table.

Evaluate first. Build what the evaluation tells you the system needs. When you are ready to build, our RAG pipeline development service covers ingestion, retrieval, hybrid search, and production monitoring.