RAG Context Window Optimization: Four Levers Explained

Most RAG debugging effort goes to retrieval failure — the case where the relevant document was never surfaced. Packing failure is at least as common in production: the right document was retrieved but the model did not use it well because of how it was presented in the context window.

Packing failure shows up in a specific pattern. Faithfulness scores on your evaluation set look acceptable — the model is not hallucinating outright — but relevancy scores are lower than expected, and users report that answers miss nuance clearly present in the documents. When you inspect the context window at inference time, you see the same passage repeated across three chunks from slightly different document sections, a key sentence buried at position 15 in a 20 chunk context, and raw document text that runs to 400 tokens when 80 would have carried the information.

The four levers below address each of these patterns in turn. None of them require changing the retrieval model or reindexing the corpus. They operate on the output of retrieval, before it reaches the model.

When documents share structural repetition — subsections that share sentence level text, a FAQ that restates the main article, or multiple versions of a document in the same index — the first three chunks from a retrieval result of five may all carry the same core sentence with minor variation. The model reads them in sequence, finds no new information in chunks 2 and 3, and either ignores them or treats the repetition as emphasis and gives it disproportionate weight.

Maximal Marginal Relevance (MMR) solves this. It scores each candidate chunk with a combined criterion: query relevance minus a penalty proportional to the similarity of that chunk to the chunks already selected. Formally, each candidate c is scored as:

score(c) = λ × relevance(c, query) − (1 − λ) × max_similarity(c, already_selected)

The λ parameter controls the relevance and diversity tradeoff. At λ = 1.0 you get pure similarity (same as top K). At λ = 0.0 you get maximum diversity with no regard for relevance. In practice, values between 0.6 and 0.8 give a retrieval result that covers the relevant information space without sacrificing query specificity.

Most vector database clients expose MMR retrieval as a parameter rather than a code change. LangChain, LlamaIndex, and most vector store SDKs have an mmr option on the retriever or similarity search call.

One tradeoff worth noting: MMR adds a second retrieval pass over the candidate pool. For a corpus where redundancy is low, the gain in answer quality is marginal and the extra latency may not justify it. Profile your corpus for chunk similarity before committing MMR as the default mode.

If your RAG pipeline uses a reranker after retrieval, MMR and reranking address different problems and work well together when applied in the right order: MMR diversifies the candidate pool, then the reranker rescores it for relevance.

Language models process context sequentially. Their attention mechanism means that information at the beginning and end of the prompt is more reliably encoded into the internal representation used for answer generation than information in the center. In a context spanning 20 chunks, positions 1 and 20 are recalled reliably. Chunks at positions 8 through 14 are significantly more likely to be missed or given lower weight even when the model appears to process the full context.

This pattern is called lost in the middle. It was characterized in published work on long context retrieval and has been replicated across GPT-4, Claude, and Llama family models. The effect is less pronounced in models specifically trained for long context tasks, but it does not disappear entirely.

The practical rule is simple: place your highest relevance evidence at the top of the injected context, supporting detail in the middle, and any critical constraint or instruction at the bottom. Do not rely on the reranker score alone to determine position — a chunk ranked first by the cross encoder should also appear first in the prompt.

One useful pattern for multi document queries is the “sandwich” layout: place the most relevant chunk first, fill the middle with supporting evidence sorted by descending relevance score, and repeat the most critical sentence as the last element before the query. This costs a small number of additional tokens but materially improves answer faithfulness for queries where a single key fact must be located and cited.

What is lost in the middle in RAG?

Lost in the middle is a positional bias in language models where information placed in the center of a long context is less reliably recalled than information at the start or end. In RAG, this means that high priority retrieved chunks should appear near the beginning or end of the injected context, not in the middle — regardless of their retrieval rank score.

It carries the answer, several sentences of surrounding context that help the model understand it, and a number of sentences that are relevant to the document but not to this specific query.

Extractive compression identifies and removes the low value sentences before injection. The most common approach uses a small encoder model or a term frequency scorer to rank every sentence in the chunk by its relevance to the query, then keeps the top N sentences that fit within a target per chunk token budget. The removed sentences are not summarized — they are dropped entirely. This avoids the hallucination risk of abstractive compression, where a secondary language model generates a shorter version of the chunk and may introduce facts not present in the source text.

In production, extractive compression is most valuable in two scenarios. The first is when the corpus contains long form documents: technical specifications, legal agreements, or product documentation where the relevant fact is one sentence embedded in a 2,000 token section. The second is when per query token cost has become a budget constraint and you cannot reduce chunk count without harming recall.

Abstractive compression — asking a small model to summarize each chunk — can reduce token count more aggressively than extraction but requires a separate model in the inference path and a faithfulness gate to catch summaries that introduce errors. For teams using RAG in regulated contexts, the risk of a compressor introducing an unsupported fact is a compliance concern, not just an accuracy concern. Extractive compression only selects and drops — it does not add information, which matters in financial, medical, and legal workflows.

The guide to RAG hallucination detection in production covers how to measure faithfulness degradation across pipeline changes, which makes it a useful companion when adding any compression step.

Teams add more chunks to catch edge cases. Rerankers start returning longer documents. Compression is applied inconsistently. The average context at inference time grows from 2,000 tokens to 8,000 tokens across several months of incremental changes, and nobody notices until the inference bill does.

A per tier budget policy assigns a maximum token count to the injected context for each query type, enforced before the context reaches the model. A simple three tier structure covers most use cases:

• Simple factual queries (lookup, extraction): 512–1,024 tokens per retrieval call. These queries need one or two facts. More context adds noise.
• Analytical queries (compare, explain, synthesize): 2,048–4,096 tokens. These benefit from multiple chunks covering different facets of the question.
• Multi document queries (summarize across sources, audit against policy): 4,096–8,192 tokens. These require breadth but should still compress individual chunks before packing.

The fallback path matters as much as the limits. When retrieval returns more tokens than the tier limit allows, the pipeline should apply extractive compression first, then truncate remaining chunks from lowest to highest relevance rank until the context fits. Hard truncation without prior compression drops evidence that could have been preserved.

Query classification for tier routing can be as simple as a zero shot prompt to a small model or a keyword based heuristic. The classification call is much cheaper than running the full context through a frontier model, and the explicit routing decision makes the tradeoff auditable.

The deeper value of an explicit budget policy is that it forces the team to be intentional. A team that sets a limit and then measures faithfulness at that limit is doing something different from a team that lets context size float. The former has a defensible engineering decision with data behind it.

Budget policy is one component of a broader production retrieval architecture that also covers indexing strategy, query routing, and latency budgeting — topics covered in depth elsewhere in this series.

Apply one, measure against your evaluation set, confirm the signal is positive, then proceed to the next. Applying all four simultaneously makes it impossible to attribute any observed improvement or regression.

A safe order: start with ordering, because it requires no model changes and no additional latency — it is a sort step on the retrieved list before prompt construction. Then apply MMR, which changes the retrieval call and can be tested on a sample of representative queries before deployment. Add extractive compression third — this requires a sentence scorer and a compression loop in the retrieval post processing step, so test faithfulness scores before and after. Implement budget policy last, because it depends on query classification, which adds a service dependency.

A compression ratio above 60 percent begins to risk dropping load bearing sentences; 30 to 50 percent is a safer starting range when first deploying compression.

For chunk level decisions like splitting strategy and embedding model choice, see the RAG chunking strategies guide — those decisions interact with compression, because tightly scoped chunks extract less cleanly than verbose ones.

Faithfulness measures whether the generated answer is grounded in the retrieved context. Track it before and after any compression adjustment, because compression can drop supporting sentences the model needed to stay grounded.

Context recall measures whether the retrieved context contained the answer to the question. Chunk selection changes (top K versus MMR) affect context recall directly. If MMR improves faithfulness but hurts context recall, the diversity parameter λ is too low.

Answer relevancy measures whether the answer addresses the question asked. Positional ordering changes affect answer relevancy when the key chunk was previously buried in the middle.

Latency and cost per query are operational metrics. A lever that improves faithfulness by two percentage points but doubles per query cost requires a business tradeoff, not just a technical one.

Context window optimization solves the packing problem, not the retrieval problem. If you have applied all four levers and quality is still below acceptable levels, the root issue has likely shifted back to retrieval coverage or chunking strategy.

Working through a production RAG system that handles high query volume at low latency is the kind of architecture engagement I take on through AI Systems Architecture consulting. If you are at the stage where these levers matter, that typically signals a system ready for a structured architecture review.