Section 01 · The Problem
Your RAG is built. Now why is it underperforming?
You shipped a RAG system. It retrieves documents, passes them to the LLM, and returns answers. And yet the answers are wrong, irrelevant, or inconsistent. The pipeline works. The results do not.
Quick answer
The short answer: Most RAG quality problems trace back to one of five root causes: poor chunking, a mismatched embedding model, no reranking pass, unoptimized queries, or missing groundedness filters. Fixing the right lever improves retrieval accuracy faster than rebuilding the system.
You have already done the hard part of getting a RAG pipeline running. Vectors are indexed, retrieval is wired, and the LLM generates answers. The problem is that somewhere between the user's question and the final response, something goes wrong. The retrieved chunks miss the point. The model drifts off document. The same question gives different answers on different days.
This is not a sign that RAG is broken as an approach. It is a sign that the pipeline has one or more tuning problems. Every production RAG system I have seen that underperforms has a specific, diagnosable cause — not a fundamental limitation.
The five levers below are the ones that matter most. They are ordered roughly by how often each one is the culprit, starting with the issue that breaks more RAG systems than any other.
Section 02 · Lever 1
Fix your chunking strategy first
The way you split documents before indexing is the single most impactful decision in your pipeline. Most teams get it wrong on the first pass.
Fixed size chunking — splitting documents into equal token or character windows with some overlap — is the default in nearly every getting started tutorial. It is also the fastest path to poor retrieval. A fixed size chunk may split a sentence in half, separate a question from its answer, or lump together paragraphs that have nothing to do with each other. The vector for that chunk becomes a blurry average of unrelated concepts, and the retriever cannot find it reliably.
Three approaches outperform fixed size chunking in practice:
Semantic chunking
Splits documents at natural semantic boundaries rather than at a token count. LangChain's SemanticChunker and LlamaIndex's SemanticSplitterNodeParser both implement this by embedding the text and splitting when adjacent sentences diverge in embedding space beyond a threshold. The result is chunks where each one covers a coherent idea.
Hierarchical chunking
Stores two granularities: large parent chunks that cover full sections, and smaller child chunks embedded for retrieval. When a child chunk is retrieved, the system returns its parent for context. This approach fixes the common case where the right passage is retrieved but stripped of the surrounding context the LLM needs to answer correctly.
Document structure aware chunking
Uses the document's own structure (headings, paragraphs, list items, table cells) to define chunk boundaries. PDF and HTML parsers that understand document structure — Unstructured, LlamaParse — produce dramatically better chunks than naive text splitters.
The metric to watch after changing your chunking strategy is context precision: what fraction of your retrieved chunks are actually relevant to the question. You can measure this with RAGAS or a manual sample review. A context precision above 0.7 is a reasonable production target for most use cases.
Section 03 · Lever 2
Upgrade or tune your embedding model
The embedding model turns your chunks into vectors. If the model was not trained on text that looks like your corpus, the vector space does not reflect the relationships that matter for your retrieval task.
Most teams start with text-embedding-ada-002 or a similar general purpose model and never revisit the choice. That is fine for retrieval over general knowledge text. It is not fine for specialized corpora.
Legal documents, scientific literature, clinical notes, and software documentation all have vocabulary distributions that general purpose embedding models handle poorly. A query about fiduciary duty in the context of delegated portfolio management and a chunk about portfolio management standards may end up far apart in the embedding space of a model that mostly saw web text, even though they are semantically close for a lawyer.
Three options to evaluate when general purpose embeddings underperform:
OpenAI text-embedding-3-large
The current strongest general purpose embedding from OpenAI and outperforms ada-002 on most benchmarks. Start here if you have not already upgraded from ada-002.
Voyage AI domain models
Offers domain specific models for legal, finance, code, and general multilingual text. Their voyage-law-2 and voyage-finance-2 models consistently outperform generalist alternatives on retrieval benchmarks for those domains.
Fine tuning on your own data
The highest ceiling option and the most expensive. If you have a corpus of query and relevant passage pairs, fine tuning with contrastive learning can produce an embedding model that outperforms every general purpose option on your specific task. The sentence-transformers library makes this accessible without building infrastructure.
Watch context recall after an embedding model change: are the right documents appearing in the top ten retrieved chunks? If recall is high but precision is low, that is a reranking problem. If recall itself is low — the right documents are not in the retrieved set at all — that points to the embedding model or the chunk boundaries.
Section 04 · Lever 3
Add a reranker to your retrieval step
Vector similarity is good at finding roughly relevant chunks quickly. A reranker is good at deciding which of those roughly relevant chunks is actually the most useful. They solve different problems.
The standard RAG retrieval flow returns the top K chunks by cosine similarity to the query embedding. Bi encoder models are fast, but they score query and document independently. They do not look at how the query and document interact. A cross encoder model does exactly that: it takes a query and a candidate chunk together and produces a relevance score based on their joint representation. Cross encoders are slower but dramatically more precise.
The practical setup adds a reranking step after your initial vector retrieval: (1) retrieve the top 20 to 40 chunks by vector similarity, (2) pass each query and chunk pair to a cross encoder reranker, (3) resort by the reranker scores, (4) pass only the top 5 to 8 reranked chunks to the LLM.
Cohere Rerank
The most widely deployed production option. It is fast enough for real time use, exposes a clean API, and performs well across most English language domains.
Jina Reranker v2
An open source alternative that runs locally, which matters when data privacy prevents sending chunks to a third party API.
cross-encoder/ms-marco-MiniLM-L-6-v2
A smaller, faster open source option from Hugging Face that works well when the query volume is high and latency is tight.
Reranking consistently improves the quality of the context window passed to the LLM. The metric to watch is answer relevance — does the LLM's answer actually address what the user asked? RAGAS measures this directly. One tradeoff to know: reranking adds latency. For a typical setup retrieving 30 candidates and reranking with Cohere, you are adding roughly 100 to 300ms.
Section 05 · Lever 4
Rewrite the query before retrieval
The question your user asks is often not the best query for finding the right documents. Query rewriting closes that gap before the vector search even runs.
Users phrase questions in natural language that does not always match how the relevant information is phrased in your documents. A user asks: “Why did the system fail last Tuesday?” Your runbook chunk says: “Root cause analysis for timeout errors during peak load.” The semantic distance between those two is larger than you want.
Three query transformation techniques that address this in different ways:
HyDE (Hypothetical Document Embeddings)
Asks the LLM to generate a hypothetical answer to the query, then uses that hypothetical answer as the search query instead of the original question. The embedding of a hypothetical answer tends to be much closer to the embedding of a real answer chunk than the embedding of a question is.
Step back prompting
Rewrites a specific question into a more general one before retrieval. A question like “What is the dosage of metformin for a patient with stage 3 CKD?” becomes “What are the dosage guidelines for metformin in patients with kidney disease?” The broader formulation retrieves a wider set of relevant context.
Query expansion
Runs the original query plus two or three LLM-generated variants in parallel, merges and deduplicates the retrieved chunks, and passes the union to the reranker. This improves recall at the cost of some added latency and token use.
For production systems, HyDE and step back prompting add one LLM call before retrieval — roughly 100 to 500ms depending on your model and latency configuration. The quality improvement is usually worth it for knowledge intensive tasks. For simple lookup tasks where queries and documents share vocabulary, query rewriting adds cost without much benefit — measure first.
Section 06 · Lever 5
Filter for response groundedness
Even with well retrieved chunks, an LLM can still hallucinate. Groundedness filtering is the check that catches responses where the model drifts off the retrieved context.
The first four levers all improve what goes into the LLM. This one monitors what comes out. A RAG system without groundedness checking has no safety net for the case where retrieved chunks are ambiguous, the query is unusual, or the LLM overrides the context with its parametric knowledge.
Groundedness (also called faithfulness) measures whether the claims in the response can be traced back to the retrieved context. A response that states a fact not present in any of the retrieved chunks is hallucinating, regardless of whether that fact is correct.
RAGAS
The most widely adopted open source evaluation framework for RAG. It provides four core metrics: faithfulness (is every claim grounded in the retrieved context?), answer relevance (does the response answer the question?), context precision (are the retrieved chunks relevant?), and context recall (did retrieval find the right documents?). Works as both an offline evaluation tool and an online monitor for sampling live traffic.
TruLens
Provides a similar metric suite with a UI for inspecting individual traces and a persistence layer for tracking metric trends over time. It integrates directly with LangChain, LlamaIndex, and raw OpenAI calls. If you want a dashboard for monitoring RAG quality in production, TruLens is the faster path to that.
For production systems where hallucination has real consequences — medical information, legal guidance, financial advice — add an inline groundedness check. Before returning a response to the user, score it against the retrieved chunks with a small, fast model. If the faithfulness score drops below a threshold (commonly 0.6 to 0.7), either retry the generation with a stricter system prompt or return a fallback response that acknowledges the system could not find a confident answer.
If you want to go deeper on building and auditing RAG systems, the AI systems architecture service covers the full stack from retrieval design through observability.
FAQ
Common questions about RAG pipeline optimization
What is the most common reason a RAG system returns irrelevant results?
Fixed size chunking is the most frequent culprit. When chunks are split at arbitrary token boundaries rather than semantic ones, the vector for each chunk represents a blurry average of unrelated sentences. The retriever then returns chunks that are topically adjacent but not actually relevant to the question. Switching to semantic or hierarchical chunking resolves this in most cases without any other changes.
How do I know which RAG optimization lever to try first?
Start with context precision: run a sample of 20 to 30 representative queries, retrieve the top five chunks for each, and manually score how many are genuinely relevant. If precision is below 0.6, the problem is in chunking or embedding. If precision is reasonable but answer quality is still poor, the problem is in reranking, query formulation, or the LLM prompt. That two step diagnostic points you to the right lever before you invest time in the wrong one.
What is retrieval reranking and when do I need it?
Reranking adds a cross encoder model after your initial vector search. The vector search returns roughly relevant candidates quickly. The cross encoder then scores each candidate against the query jointly, catching relevance signals the vector similarity missed. You need reranking when your initial retrieval recalls the right documents but they appear in the wrong order, or when a large candidate pool dilutes the quality of what reaches the LLM.
How do I measure whether my RAG pipeline optimization actually worked?
Use RAGAS or TruLens to establish a baseline before you change anything. Run a fixed evaluation set of 50 to 100 queries with known good answers, record the four core metrics (faithfulness, answer relevance, context precision, context recall), then rerun the same set after each change. A lever that genuinely helped will show a measurable improvement in at least one metric without degrading the others.
Building a RAG pipeline that performs reliably in production is an iterative process. For a systematic review, the production RAG implementation guide covers the foundational design decisions that set up these optimizations to land cleanly. For a deeper look at how to measure whether your optimizations are actually working, the RAG evaluation metrics post covers RAGAS, TruLens, and custom evaluation harnesses in detail.