Section 01 · Framing
RAG hallucinations are not like general LLM hallucinations
A general LLM hallucinates from its training distribution. A RAG system can fail differently: the model is given a context and still generates claims the context does not support.
Quick answer
How do you detect RAG hallucinations? The production-grade approach is a layered defense: improve retrieval grounding first (40 to 71 percent reduction before any post-generation work), then apply faithfulness scoring via NLI to flag ungrounded sentences, then use claim decomposition for high-stakes outputs to label each claim as Entailed, Contradicted, or Baseless. Route answers below your faithfulness threshold to guardrails or human review. No single method is sufficient on its own — the layers compound.
RAG hallucinations split into three root causes. Retrieval failure: the retrieved passages are irrelevant to the question. The model confabulates from parametric knowledge or produces a hedged non-answer. The fix is at the retrieval layer, not the generation layer. Faithfulness failure: the retrieved context is relevant, but the model generates claims that go beyond or contradict what the context actually says. The fix is post-generation faithfulness scoring. Over-abstention: the model refuses to answer a question that is actually supported by the retrieved context. This is often invisible in standard benchmarks but is a real production problem.
Each failure type has a different detection method. The sections below address them in order of highest to lowest leverage.
Section 02 · Architecture
The layered defense
The production defense is not a single technique. It is a stack of four layers, each catching failures the layer above it misses.
| Layer | What it catches | When to stop here |
|---|---|---|
| Retrieval grounding | Retrieval failures | High-precision retrieval already achieved |
| Faithfulness scoring (NLI) | Most faithfulness failures | Low-risk domains with adequate eval coverage |
| Claim decomposition | Subtle contradictions, multihop failures | High-stakes outputs with known failure patterns |
| Guardrails + HITL | Residual edge cases | Never omit for regulated, medical, legal outputs |
Not every system needs all four layers. A customer support agent over a well-structured product FAQ can stop at faithfulness scoring. A medical information system should run all four.
Section 03 · Layer 1
Retrieval grounding
The single most effective intervention is improving retrieval quality. Better grounding reduces hallucination rate by 40 to 71 percent before any post-generation check runs.
Semantic chunking over fixed-size chunking
Fixed-size chunking splits documents at arbitrary character boundaries, frequently separating a claim from its supporting evidence. Semantic chunking splits at logical boundaries — paragraph, section, or sentence group — keeping evidence intact. The model receives passages it can faithfully cite.
Reranking after initial retrieval
A first-stage retriever returns a broad candidate set. A cross-encoder reranker rescores the top candidates with a more expensive but more precise relevance model. Reranking consistently improves faithfulness because the model receives the most relevant passage, not just the most similar-looking one.
Evidence sufficiency scoring
If the top-ranked passage does not contain the answer, the system should know that and abstain rather than force the model to generate from incomplete context. Evidence sufficiency scoring assigns a confidence score to the retrieved set before generation. Below a threshold, the system returns a graceful abstention.
For detail on chunking choices and their effect on retrieval precision, the RAG chunking strategies guide covers the tradeoffs across fixed, semantic, and sentence-based chunking approaches.
Section 04 · Layer 2
Faithfulness scoring
After generation, check whether each sentence in the model output is supported by the retrieved context. This is faithfulness scoring, and the standard implementation uses natural language inference.
An NLI model takes two inputs: a premise (the retrieved passage) and a hypothesis (a sentence from the model output). It outputs one of three labels: Entailment (the passage supports the claim), Contradiction (the passage refutes the claim), or Neutral (the passage neither supports nor refutes the claim).
Faithfulness score for the full answer is computed as the number of entailed sentences divided by the total sentence count. A threshold of 0.85 to 0.90 is a common starting point. Answers below the threshold are flagged for abstention, reformulation, or escalation.
Off-the-shelf options for faithfulness scoring
cross-encoder/nli-deberta-v3-small on HuggingFace is fast, small, and good for production throughput. RAGAs wraps the NLI check with a RAG-specific interface. Trulens provides faithfulness, context relevance, and answer relevance in one evaluation pass.
The faithfulness check adds latency. For synchronous user-facing applications, run it in parallel with the main response generation and use the score to decide whether to surface the answer or return a graceful abstention. For asynchronous batch workflows, run it as a post-processing step.
For the full set of metrics — faithfulness, context relevance, answer relevance, context recall — and how they interplay in a production eval rig, see RAG evaluation metrics.
Section 05 · Layer 3
Claim decomposition
Faithfulness scoring at the sentence level catches most failures. It misses subtle cases: a sentence may be largely entailed but contain one phrase that contradicts the source.
Claim decomposition addresses this by breaking the model output into atomic claims — the smallest meaningful assertions — before running the NLI check.
Decompose into atomic claims
Use a small LLM to extract assertions from the model output. “The context window limit is 200,000 tokens and it was released in June” decomposes into two claims: a context window claim and a release date claim.
Retrieve targeted passages for each claim
Run a second retrieval pass targeted at the specific claim, not the original query. This surfaces the passage most relevant to each individual assertion.
Label each claim
Run NLI against the targeted passage. Label each claim as Entailed (passage supports it), Contradicted (passage refutes it), or Baseless (no relevant passage found). Score the answer as the fraction of Entailed claims.
Use claim decomposition selectively
Claim decomposition is 3 to 5 times more expensive than sentence-level faithfulness scoring. Use it for high-stakes answers, flagged responses, or domains where a single Contradicted claim has significant consequences. Most production systems use sentence-level scoring as the first pass and claim decomposition as the escalation layer.
For production AI systems where hallucination consequences are significant, claim decomposition pairs with the guardrail layer described in AI agent guardrails to create a complete detection and interception stack.
Section 06 · Layer 4
Guardrails and human-in-the-loop routing
Automated detection catches most failures. HITL routing catches the rest and is the appropriate backstop for domains where a missed hallucination has real consequences.
The routing logic uses two thresholds. Answers at or above the high threshold are auto-approved. Answers between the two thresholds surface the response with a “based on available information” caveat. Answers below the low threshold route to human review.
Thresholds are domain-specific
In a regulated context (legal, medical, financial), the high threshold may be 0.95 and anything below routes to human review. In a customer support context, the low threshold may be 0.70 and partial answers surface a caveat automatically. There is no universal setting.
Before routing to a human reviewer, run a guardrail check that screens for known false claim patterns in your domain, topic drift (the answer addresses a topic not covered by the retrieved context), and answer completeness (the answer is hedged in a way that suggests the model lacked context). Guardrails catch obviously wrong answers before human reviewers see them, reducing reviewer burden.
Track the human approval rate on flagged answers
If reviewers approve more than 60 to 70 percent of flagged answers, the threshold is too aggressive and you are routing unnecessary volume to human review. The approval rate is the calibration signal for your thresholds — revisit them quarterly as your retrieval quality improves.
Section 07 · Measurement
Evaluation metrics to track in production
The layered defense only improves if you are measuring the right things. Four metrics cover the RAG hallucination problem adequately.
Faithfulness
Fraction of model output sentences entailed by the retrieved context. The primary measure of hallucination rate. Target above 0.85 for most domains; above 0.95 for regulated outputs.
Context relevance
Fraction of retrieved passages that are relevant to the question. Measures retrieval quality directly — the Layer 1 lever. A low score here explains high hallucination rates better than any post-generation check.
Answer relevance
Fraction of the answer that actually addresses the question asked. Catches over-abstention and topic drift — the failure modes that faithfulness scoring alone misses.
Human approval rate on flagged answers
The rate at which human reviewers approve answers the system flagged for low faithfulness. The calibration metric for your thresholds. Track this monthly and adjust thresholds as retrieval quality improves.
For the full eval setup including how to run these metrics at production scale, see LLM agent evaluation in production.