The Production RAG Guide: Retrieval, Reranking and Evaluation

In NKK's view, teams often debug RAG backwards: a wrong answer appears, so they rewrite the prompt or swap the model, when the evidence the model needed was never in front of it. Barnett and colleagues, drawing on three case studies, list seven failure points, and the first three (missing content, missed documents, evidence not in context) start before generation ^[1]. Their abstract's first lesson is that "validation of a RAG system is only feasible during operation" ^[1]. We read that as: build a starter evaluation set up front, then grow it from production failures.

This guide takes a position. In NKK's view, production RAG is a chain of stages whose pass rates multiply, so the cheapest fix is usually the earliest broken stage. We found no failure-rate data that says how often that is the case, and say so below. And authorization, the stage that is easiest to leave untested because it fails silently, is a property of retrieval, not of the prompt. We walk the pipeline in debugging order, cite what the evidence supports, say where it is thin, and finish with a decision flow and a debugging table.

Debug in pipeline order: the model is the last place to look

The first number is a vendor measurement: Anthropic's own evaluation across codebases, fiction, ArXiv and science papers, with embeddings-only retrieval (Gemini Text 004) as the baseline. The page does not state who built the datasets or whether they are public, so treat it as indicative. The mechanism is general, though: if the relevant chunk is not retrieved, nothing downstream recovers it ^[2]. The second is a 2023 result on 2023 models, in which GPT-3.5-Turbo scored 75.8 with the answer first and 53.8 mid-context; the paper itself says "more than 20%". Liu et al. found accuracy highest when relevant information sits at the beginning or end of the input and lower in the middle, even for long-context models ^[3]. Retrieval can succeed and the answer can still fail. The third is one 2026 preprint: Recall@5 rose from 0.587 for dense-only retrieval to 0.816 for hybrid retrieval plus a reranker on 23,088 finance queries ^[17].

Stages also multiply. Suppose five stages pass 95%, 90%, 92%, 93% and 97% of the time. End to end that is about 71%, even though no single stage looks alarming. The five rates in this chart are invented to show the arithmetic, and real stages are not independent, but the lesson holds: a weak early stage caps everything after it.

The NKK Retrieval Debug Ladder orders the checks so effort never goes into the prompt while the evidence is missing. It builds on the failure points above ^[1], Microsoft's retrieval metrics ^[4] and its generation diagnostics ^[5].

How to use it: take 50 failing examples (NKK's rule of thumb, not an evidence-based number), label each with the lowest rung that fails, and fix the most common rung first. Where it breaks: rungs interact (a new chunk size changes recall and rank at once), so treat the order as triage, not causal proof, and rungs 2 and 3 cannot be measured without labelled evidence.

Build the evaluation set before the pipeline

Microsoft's RAG design guidance defines a test case as a query, the context that answers it (actual text and page) and the expected answer, and prefers real content and real user questions over synthetic ones ^[6]. Questions generated per chunk by an LLM are acceptable if a subject-matter expert verifies them ^[6]. The least glamorous requirement matters most: gather queries the documents do not address, and decide in advance what the system should do, such as saying it does not know ^[6]. Models struggle here. The RGB benchmark found six LLMs showed some noise robustness but struggled with negative rejection, information integration and false information ^[7]. The SemEval-2026 multi-turn task includes unanswerable, underspecified and non-standalone questions, covering the same gap ^[8].

Measure each stage, then the whole. For retrieval use precision@K, recall@K and mean reciprocal rank; positive queries should score near 1 and negative queries near 0 ^[4]. For generation, Microsoft defines groundedness, completeness, utilization, relevance and correctness, and the pairings diagnose. High completeness with low utilization suggests lowering top-k. Low completeness with high utilization suggests larger chunks or higher top-k. High groundedness with low correctness means the answer is grounded in the wrong, or wrongly combined, context ^[5].

Automated judges need calibration. Ragas defines faithfulness as the share of answer statements supported by the context; on its WikiEval set, agreement with human annotators was 0.95 for faithfulness but 0.70 for context relevance ^[9]. That dataset is small and built by the authors, so treat it as indicative. ARES uses a few hundred human annotations to correct judge error ^[10]. Label a sample by hand before you trust any automated score.

Chunking: let document structure choose the method

Microsoft calls the chunking approach "a semipermanent choice" in the overall design, because changing it forces downstream changes, and matches methods to structure: layout analysis for semi-structured documents, custom code for law and patents, sentence-based or fixed-size chunks with overlap for plain prose ^[11]. There is no consensus size. OpenAI's managed file search defaults to 800 tokens with 400 overlap ^[12]; Amazon Bedrock's default is about 300 tokens, sentence-aware ^[13].

Evidence for any one answer is thin and mixed. In Chroma's vendor study, the 800/400 token splitter that Chroma assumes matches OpenAI's default (it ran TokenTextSplitter, not OpenAI's own implementation) scored lowest: precision 1.4 (IoU 1.4), against 1.5 for recursive splitting at the same 800/400 setting, 7.0 (IoU 6.9) for 200-token recursive chunks and 8.0 for 200-token cluster-semantic chunks, with the top 5 chunks and text-embedding-3-large. The gap at equal size is small; the large differences are against 200-token chunks. These are token-level metrics on LLM-generated queries, not answer accuracy ^[14]. On semantic chunking, a controlled comparison (Qu, Tu and Bao) found gains mainly on artificially stitched datasets; on original datasets fixed-size was usually equal or better (HotpotQA F1@5 90.59 against 87.37), and the authors call it the more efficient and reliable choice ^[15]. A 2026 preprint adds that the best strategy depends on the task: structure-based chunking (its term for fixed-size, sentence and paragraph splitting) beat LLM-guided chunking for in-corpus retrieval, while contextualized chunking helped there and hurt in-document retrieval ^[16].

The better-supported lever is restoring context a chunk loses. Anthropic prepends LLM-written context to each chunk; in its own test, failed top-20 retrievals fell from 5.7% to 3.7%, a 35% relative reduction ^[2]. Independent checks are smaller. On a 20% subset of NFCorpus, with Jina-V3 and a small open model (Phi-3.5-mini) writing the prefixes, nDCG@5 moved from 0.303 to 0.312 ^[18], the only independent chunk-level check we found. A finance preprint prepended summaries to whole, unchunked documents of about 920 tokens (Recall@5 0.587 to 0.615 dense, 0.695 to 0.717 hybrid) ^[17]; that is indirect evidence, and its authors caution that results may differ for chunked corpora. Late chunking, which embeds the full text before splitting, is the authors' claim and an alternative ^[19]. In the same NFCorpus study, late chunking scored 0.309 nDCG@5 on the subset, against 0.303 for the plain fixed-window baseline and 0.317 for the full contextual pipeline; the authors summarise late chunking as one that "offers higher efficiency but tends to sacrifice relevance and completeness" ^[18].

Two cautions. Layout-rich PDFs are a weak spot for text extraction; ColPali proposes embedding page images instead, but the evidence is the authors' own ^[20]. And permissions must travel with every chunk, a point we return to below.

Hybrid search: BM25 is the floor, not the enemy

The 2021 BEIR benchmark, covering 18 datasets and 10 retrieval systems, found BM25 "a robust baseline"; re-ranking and late-interaction models had the best average zero-shot performance at high computational cost, and dense and sparse models often underperformed ^[21]. Those are 2021 models, but a 2026 benchmark points the same way for numeric, table-heavy finance documents. On 23,088 finance queries, BM25 beat a strong dense model on Recall@5 (0.644 against 0.587), hybrid retrieval with reciprocal rank fusion reached 0.695, and adding a Cohere reranker reached 0.816 ^[17]. That is one preprint, one domain and whole documents of about 920 tokens, so treat it as a data point, not a law. Vendors agree on the mechanism: keyword search wins for product codes, jargon, dates and names because it finds exact matches ^[22].

Embedding choice needs the same humility. MTEB found that no embedding method dominates across tasks ^[23]. BRIGHT showed the top MTEB model scoring 59.0 nDCG@10 on MTEB but 18.3 on its reasoning-heavy queries ^[24]. Use the leaderboard to shortlist, then decide on your own eval set.

For fusion, start with reciprocal rank fusion. It sums 1/(k + rank) across result lists; Cormack et al. fixed k at 60 in a pilot, found it near-optimal but not critical, and reported average gains of 4% to 5% over Condorcet, CombMNZ and the best single system ^[25]. Azure's documentation also uses a k of about 60 ^[26]. Weaviate moved its default to score-based fusion and advises setting alpha explicitly ^[27]. We found no independent comparison, so switch only if your eval set shows a gain.

Reranking and how much context to pass

Retrieval optimizes for recall; reranking optimizes for precision. Microsoft describes a common four-step pattern: retrieve broadly (for example the top 50), merge with RRF, rerank with a cross-encoder, semantic ranker or LLM, then truncate to the top 5 to 10, starting the candidate set at 20 to 50 ^[4]. A cross-encoder scores each query-document pair jointly, which is too slow for a whole corpus, hence the two stages ^[28].

The gains are the largest in our sources. Reranking lifted hybrid Recall@5 by 12.1 points and MRR@3 by 17.2 points in the finance benchmark ^[17]. In Anthropic's test, adding contextual BM25 and reranking cut failures from 5.7% to 1.9%, a 67% relative reduction, from 150 candidates reranked to a top 20; passing 20 chunks beat passing 10 or 5 ^[2]. LLM rerankers are a credible alternative: a distilled 440M model outperformed a 3B supervised model on BEIR ^[29]. We found no primary-source latency figures for rerankers, only the warning that reranking adds latency to every query ^[4]. Measure it.

Two rules prevent common mistakes. First, use reranker scores for ordering, not thresholds: "Cross-encoder scores are relative, not absolute" ^[4]. If you want abstention, calibrate a cutoff on positive and negative test queries. Second, more context is not free. Going from 20 to 50 documents improved accuracy by only about 1.5% for GPT-3.5-Turbo and about 1% for Claude-1.3 while cost rose ^[3]. Tune candidate depth (recall) and final context size (precision) as separate dials.

Query rewriting and agentic retrieval: gate by query type

Rewriting is not a free upgrade. HyDE has an LLM write a hypothetical answer and searches near its embedding; the authors concede the document "may contain false details" ^[30]. Rewrite-Retrieve-Read trains a small rewriter with reinforcement learning from reader feedback and shows gains, but against a web search engine, so transfer to private corpora is unshown ^[31]. Self-RAG and Corrective RAG make retrieval adaptive, deciding when to retrieve or scoring retrieved documents; both are research systems ^[32][33].

The counter-evidence matters. In the finance benchmark HyDE (0.544) scored below plain dense retrieval (0.587), and multi-query fusion (0.640) did not beat BM25 (0.644) ^[17]. Precise numeric lookups do not need paraphrase. Where rewriting helps is conversational, vague or reasoning-heavy input: BRIGHT reports up to 12.2 points from explicit reasoning about the query ^[24], and multi-turn questions are non-standalone by nature ^[8]. Microsoft says to keep the original query and add context rather than replace it ^[4].

Agentic retrieval plans and decomposes queries with an LLM, runs subqueries in parallel and reranks each. Azure's documentation says it adds latency, bills tokens from two services, and, on the page we read (2026-09-16), LLM query planning is in preview ^[34]. GraphRAG targets a narrow case, whole-corpus questions such as "What are the main themes in the dataset?" ^[35].

When not to build RAG at all

Li et al. compared RAG with long-context models across public datasets and three models: long context won on average when resources allowed, but RAG was much cheaper, and a router, Self-Route, let the model choose per query and cut compute while keeping comparable performance ^[36]. Anthropic notes that a knowledge base under about 200,000 tokens (roughly 500 pages) can skip RAG and sit in the prompt ^[2]. NKK would add prompt caching to the trial to control cost. Every study here uses 2023-2024 models, so treat the thresholds as starting points and re-test per model generation.

The NKK RAG Fit Matrix turns this into a placement exercise. The points are NKK's judgement, illustrative, not measurements.

Where it breaks: the axes are qualitative, and we found no independent numbers for the bottom-right quadrant. Permissions are deliberately off the chart, because they would push almost every enterprise corpus upward: if different users may see different content, long-context stuffing needs the same filtering as retrieval.

Security and permissions are retrieval-layer problems

OWASP's 2026 Top 10 for LLM applications (published 2026-08-03 ^[37]) gives vector and embedding weaknesses their own entry, LLM09:2026. The 2025 edition numbered it LLM08 ^[38], and the 2026 LLM08 is now Hidden Context Exposure, so check which edition a citation means. The 2026 entry names RAG-relevant risks: cross-tenant leakage when similarity search runs across the full index before access control is applied, embedding inversion, retrieval-time data poisoning, retrieval jamming and membership inference through raw similarity scores. Its mitigations include enforcing tenant scoping inside the index query rather than as a post-retrieval filter, applying access control at the chunk level, tracking provenance for every embedding, withholding raw similarity scores from clients and keeping immutable logs of retrieval activity ^[37]. Prompt injection is a separate risk: the 2026 entry lists retrieved RAG passages as a delivery surface for indirect injection ^[37], and the 2025 entry states that RAG and fine-tuning do not fully mitigate prompt injection ^[39]. The PDF we read, linked from the 2026 resource page, still shows "Publication date to be set" on its cover.

The threats are not hypothetical. EchoLeak, CVE-2025-32711 in Microsoft 365 Copilot, enabled remote, unauthenticated data exfiltration through a single crafted email ^[40]; the NVD record, which describes it as AI command injection in M365 Copilot allowing an unauthorized attacker to disclose information over a network and does not use the name EchoLeak itself, shows a 9.3 rating from Microsoft and 7.5 from NVD, and does not state a patch date ^[41]. PoisonedRAG reached about 90% attack success by injecting five malicious texts per target question in a research setting, and the authors found the defences they tried insufficient ^[42]. Vectors are not anonymous either: Vec2Text recovered 92% of 32-token inputs exactly from their embeddings ^[43].

Enforcement therefore belongs inside the search. Azure AI Search applies the caller's identity at query time, and its documentation warns that permission changes reach results only after metadata is synchronized to the index; with chunking, permission metadata must be projected onto each chunk row or chunk-level references are not filtered ^[44]. On the page we read (2026-08-08), only security filters were generally available; ACL, Purview and SharePoint enforcement were preview. Elastic's document-level security shows the pattern and its limit: aggregate information about the whole index can still leak through crafted requests ^[45]. One preprint argues that prompt sanitisation and output filtering are probabilistic, and that retrieved content must be authorized for every participant in the interaction, deterministically ^[46]. The authors say Microsoft deploys it, a vendor claim.

The NKK Fail-Closed Retrieval Stack turns this into five layers. Each names the threat it answers and needs its own test cases in the eval set.

What the evidence does not show

A guide built on public sources has blind spots, and you should know them before you copy a number.

• No large independent replication of contextual retrieval. The 35% to 67% figures (the upper figure includes reranking) come from one vendor's own evaluation, whose dataset provenance the page does not state ^[2]; the only independent chunk-level check we found is small and reports a modest gain ^[18], and one document-level result is indirect ^[17].
• No independent comparison of RRF against score-based fusion, and none of prompted query rewriting on enterprise corpora.
• No primary-source reranker latency or cost figures. Budget by measuring, not by quoting.
• Dated models. The long-context, lost-in-the-middle and BEIR results use 2021-2024 models ^[3][36][21].
• No adoption or failure-rate statistics. We found none from a source that meets our standard, so this guide makes no claim about how many RAG projects fail.
• Thin security quantification. PoisonedRAG and Vec2Text numbers come from research settings ^[42][43], and the OWASP entries describe risks and mitigations, not how often they occur ^[37].

The decision flow

The NKK models fit together as follows. The Fit Matrix scopes the build, the Decision Flow sequences what to build, the Debug Ladder diagnoses what went wrong, the Debugging Table maps symptoms to ladder rungs, and the Fail-Closed Stack details flow step 1 and ladder rung 6. Run the seven flow steps in order; each defaults to the best-supported choice in the sections above.

Debugging table: symptom, rung, test, fix

Match the symptom to a rung of the Debug Ladder, run the test, and try the first fix before changing anything else.

What to do next

1. Build the evaluation set first. Use the sizing rule and readiness scorecard above, and have an expert verify the expected answers.
2. Baseline with BM25, then climb the decision flow. Record recall@K and MRR for keyword search alone, then add dense retrieval, RRF and a reranker one at a time, keeping the metric for each stage.
3. Sweep chunk size and overlap in tokens. Treat vendor defaults as hypotheses, and add contextual prefixes before semantic or LLM-guided chunking.
4. Gate query rewriting and agentic retrieval by query type. Keep the original query and measure latency and token cost.
5. Move permissions into the search. Put permission fields on every chunk, filter at query time, fail closed, and measure sync lag.
6. Make the eval set a regression gate. Run it on every model, prompt, index or chunking change, and use the debugging table to triage failures.

Conclusion

RAG quality is the product of stages, so fix the earliest broken one. Hybrid retrieval, a reranker and an honest evaluation set have the best support in the evidence we read; much of the rest, from chunk size to query rewriting, is hypothesis to test on your own corpus. Authorization is the stage you cannot afford to find in production.