Section 01 · Definition
What is reranking in RAG?
Most RAG pipelines retrieve a pool of candidates by embedding similarity, then pass the top few directly to the generator. Reranking adds a precision stage between retrieval and generation.
Quick answer
What is reranking in RAG? Reranking in RAG is a second retrieval stage where a cross-encoder model scores each candidate chunk against the query jointly — not as separate embeddings — then reorders the list by relevance. It typically delivers 10–25% better precision at top 5 at the cost of 50–500ms of added latency.
Embedding models (bi-encoders) work by encoding the query and each document chunk separately, then comparing the resulting vectors by cosine similarity. That approach is fast and scales to millions of chunks, but it has a ceiling: the model never sees the query and the chunk together, so it cannot catch fine-grained term interactions that only matter in context.
A cross-encoder reranker takes each (query, candidate chunk) pair as a single input and produces a scalar relevance score. Because it processes both texts jointly, it can attend to the specific words in the query when reading the chunk. The result is a ranking that reflects true relevance far more precisely than cosine similarity — at the expense of having to run one forward pass per candidate.
Section 02 · The pattern
How the two-stage retrieval pattern works
The two-stage pattern is the canonical production approach. You retrieve a larger-than-needed pool cheaply, then rerank it expensively but accurately before cutting to your context window.
Stage one uses your existing bi-encoder vector search. Instead of fetching the top 5 or top 10, you fetch the top 20 to 100 candidates. This is cheap because the similarity calculation is just a dot product against stored vectors. The key insight is that recall at this stage matters more than precision — you want the right chunk to be somewhere in the pool, even if it is not at the top.
Stage two passes each of those candidates, paired with the original query, through a cross-encoder. The cross-encoder scores them and the pipeline keeps only the top 3 to 5 to send to the language model. This is where latency lives: you run a forward pass for each candidate, so reranking 50 candidates costs roughly 2.5 times more than reranking 20. The tradeoff is that the top 5 you pass to the generator are far better ranked than any bi-encoder could produce.
| Property | Bi-encoder | Cross-encoder |
|---|---|---|
| Input | Query and doc encoded separately | Query + doc concatenated as one input |
| Scoring | Cosine similarity of two vectors | Scalar relevance score from classifier |
| Speed | Milliseconds (precomputed vecs) | 50–500ms per batch of candidates |
| Precision | Good for top 50–100 recall | Excellent for final top 3–5 ranking |
| Scale | Millions of chunks | 20–100 candidates only |
The two stages are complementary. Bi-encoders are not being replaced — they are being used for what they are good at: fast, high-recall filtering of a large index. Cross-encoders are added for what bi-encoders are weak at: precise ordering of a small candidate pool.
Section 03 · Model choices
Which reranking model should you pick?
Four options dominate production RAG systems. The right choice depends on your latency budget, infrastructure constraints, and whether the query domain matches general web text or a specialized corpus.
MS-MARCO trained cross-encoders are the most common starting point. Models like cross-encoder/ms-marco-MiniLM-L6-v2 are fine-tuned on 500k Microsoft Bing query-passage pairs and run locally on CPU or GPU. They are free and fast but calibrated to general web retrieval — specialized domains like medical, legal, or financial text can see degraded performance without further fine-tuning.
| Model | Latency (20 candidates) | Cost | Best for |
|---|---|---|---|
| MS-MARCO MiniLM L6 | 30–80ms (local GPU) | Free | General text, tight latency budgets |
| MS-MARCO MiniLM L12 | 60–150ms (local GPU) | Free | Better accuracy, moderate latency tolerance |
| Cohere Rerank | 100–300ms (API) | $0.001 per 1k tokens | High accuracy, no infra overhead |
| FlashRank (MiniLM) | 20–50ms (local CPU) | Free | Edge or serverless, ultralow latency |
| Custom fine-tuned | Varies by size | Training cost | Specialized domains: medical, legal, finance |
Cohere Rerank is the fastest path to production if your team does not want to manage model serving. It accepts up to 1,000 candidates per call and returns ranked results via API. The accuracy on general English text is strong — comparable to larger cross-encoders — and the API abstracts away hardware concerns. The downside is API latency (the network round trip adds 80–150ms before inference) and data privacy considerations for regulated industries.
FlashRank is worth knowing about if your RAG pipeline runs on serverless infrastructure or at the edge. It is an inference engine optimized for speed on small cross-encoders, running MiniLM variants at roughly 20–50ms for 20 candidates on CPU. That is fast enough to stay inside a 200ms total retrieval budget for most applications.
Section 04 · Decision framework
When does reranking justify the latency cost?
The decision is empirical, not architectural. Run the measurement first — the signals below tell you whether the numbers will support adding a reranker.
Quick answer
The one-sentence rule: Add a reranker when your evaluation shows that recall at top 20 is above 0.8 but precision at top 3 is below 0.6. That gap is exactly what cross-encoders close.
Your retrieval pool has the right answer, but it lands too low
If your evaluation shows that the correct chunk appears in positions 6–20 for more than 30% of queries, a reranker will move it up. This is the strongest signal that reranking helps — the right content is reachable by the first stage but imprecisely sorted.
Your query set mixes conceptual and exact-token queries
A query pool that blends paraphrase-style questions with exact identifier lookups often suffers from imprecise vector ranking on the identifier queries. Reranking helps here even when hybrid search is already in place, because cross-encoders score the pair jointly and weight term overlap without requiring a separate BM25 index.
Hallucination rate is high despite good recall
Hallucinations often trace back to the generator being given irrelevant or weakly related chunks alongside the correct one. When the context window contains noise, even a strong language model fabricates details. A reranker that filters to the top 3 most relevant chunks reduces noise before generation without touching the generation step at all.
Your latency budget allows an extra 100–500ms per query
Reranking makes sense for asynchronous pipelines, document search, and question-answering products where a 300ms addition is invisible to users. It is harder to justify for real-time chat or voice applications where total latency is already near its ceiling.
Section 05 · Counter-signals
When to skip the reranker
Adding a reranker to a pipeline where it is not needed costs latency and adds an inference dependency without producing any measurable improvement in generation quality.
First-stage retrieval already places the right chunk at position one or two
If precision at top 3 is already above 0.75 on your evaluation set, a reranker will not move the needle. Spend the engineering time on chunking strategy or context window management instead.
You are retrieving from a narrow, well-structured index
A small corpus of 500 to 5,000 curated chunks tends to have low ambiguity. Bi-encoder retrieval is already accurate because there are few competing candidates. Reranking 20 candidates from a corpus where only two or three are plausible matches is overhead with no benefit.
Your latency ceiling is under 150ms end to end
Real-time voice assistants and edge inference pipelines often cannot absorb the cost of a cross-encoder. If your total retrieval budget is 50ms, even FlashRank on CPU is too slow. In these cases, invest in chunking and embedding quality rather than reranking.
Section 06 · Measurement
How to measure whether reranking actually helps
Every reranking decision should start with a retrieval evaluation. Without baseline numbers, you are guessing — and reranking can make things marginally worse if your first-stage recall is already low.
Build a retrieval evaluation set of 50 to 100 query-answer pairs drawn from real user queries if you have them, or from your document corpus if you do not. For each query, the “correct” chunk is the one that contains the grounding information needed to answer it. This does not require LLM judgment — you can annotate it manually in an afternoon for 50 queries.
Measure two things before and after adding a reranker. First: recall at top 20 — does the correct chunk appear in the first 20 retrieved candidates? This tells you whether your first-stage retrieval is reaching the right content at all. Second: precision at top 5 — is the correct chunk in the five you pass to the generator? This tells you whether ranking quality is the problem. If recall at top 20 is below 0.6, the issue is in your embedding model or chunking strategy, not your ranking. A reranker will not fix it.
Run the evaluation twice: once with your current pipeline (no reranker) and once with a reranker inserted between retrieval and generation. The difference in precision at top 3 and top 5 is your expected improvement in generation quality. A lift of 0.10 or more at top 5 typically justifies the added latency. A lift of 0.03 or less is noise — the reranker is not helping and you should look elsewhere.
Track NDCG at k alongside precision and recall if you want a single metric that captures both whether the right chunk is retrieved and how highly it is ranked. NDCG at 5 penalizes cases where the right chunk lands in position 4 or 5 more heavily than position 1 or 2, which maps directly to generator context quality.
If you are working with hybrid search already, add the reranker after the RRF fusion step — not before. You want the cross-encoder to see the merged, deduplicated candidate pool, not just one retriever's output. This gives it the broadest possible view of what is available before it makes its final ranking call.
If you are evaluating whether your overall RAG architecture is production ready, the AI Systems Architecture service covers retrieval evaluation design, reranking integration, and the full pipeline audit end to end.