Hybrid Retrieval and Re-ranking Systems
Hybrid search for RAG: dense embeddings plus BM25, reciprocal rank fusion, cross-encoder re-ranking, metadata filters, latency budgets, failure modes, and retrieval evals.
- Read time
- 16 min
- Level
- Intermediate
- Updated
- 2026-08-25
- Components
- Dense retriever · Sparse BM25 retriever · Rank fusion (RRF)
Hybrid retrieval is a search architecture that runs dense embedding search and sparse lexical search (typically BM25) in parallel, fuses the two ranked lists — usually with reciprocal rank fusion — then re-ranks a shortlist with a cross-encoder before the generator sees any passages. It exists because vector search misses exact identifiers and keyword search misses paraphrases; enterprise queries need both.
Key takeaways
- 01
Vector-only RAG fails on SKUs, error codes, policy numbers, and rare proper names; BM25 is what catches those tokens.
- 02
Reciprocal rank fusion is the default merger: no score calibration, cheap, and stable when the two retrievers use incompatible score scales.
- 03
A cross-encoder on the fused top 50–100 is where most quality is won, and also where most of the latency budget is spent.
- 04
Apply structured filters (tenant, ACL, date, source type) on candidates before fusion, not after the model has already read a passage.
- 05
Score retrieval independently of generation: hit@k, nDCG, and MRR on a golden query set, then score faithfulness on the generator.
Why hybrid beats either retriever alone
Dense retrievers embed the query and each chunk into the same vector space and return nearest neighbors. They are strong on paraphrase, synonym, and “this is about the same thing” queries. They are weak on tokens that barely appeared in the embedding model’s training mix: ticket IDs, SKUs, error strings, statute numbers, internal acronyms. Sparse retrievers (BM25, and learned sparse models such as SPLADE) invert the token index and score exact and stemmed overlap. They are strong on those identifiers and weak when the user never uses the document’s wording.
Enterprise question logs are a mixture of both shapes in the same hour. Hybrid retrieval is the production default because it does not require predicting which shape a given query is. Run both retrievers, fuse, then spend the expensive model only on a shortlist that already contains the right neighborhood.
Data flow: query to fused shortlist
A query hits a rewrite step first — optional query expansion, spelling, and hyphen/ID normalization — then fans out to the dense index and the sparse index with the same filters. Each retriever returns its own top-k (typically 50–200). Rank fusion merges those lists into one ordering. A cross-encoder re-ranker scores query–passage pairs on the fused head (typically 50–100) and emits the final 5–10 passages the generator is allowed to see, each with source URL, heading path, and chunk id.
Keep the generator on a short, cited context. Dumping the fused 100 into the prompt raises cost, raises contradiction risk, and does not raise hit rate once the re-ranker has done its job. The retrieval stack owns “what is relevant”; the generator owns “what to say given these passages.”
- Rewrite: normalize IDs, expand rare acronyms, keep the raw query too
- Fan-out: dense k and sparse k under the same metadata filters
- Fuse: reciprocal rank fusion (k≈60) or a learned mixer
- Re-rank: cross-encoder on the fused head; take top 5–10
- Generate: answer only from the shortlist; refuse on empty retrieval
Rank fusion and the re-ranker
Do not add raw dense scores to raw BM25 scores. The scales are not comparable and the merger will silently pick a winner based on units, not relevance. Reciprocal rank fusion scores each document as the sum of 1/(k + rank) across lists, with k commonly 60. It ignores score magnitudes, handles missing documents, and is one line of code. Weighted RRF (a coefficient per retriever) is worth tuning on a labeled set; it is not worth guessing.
The cross-encoder jointly encodes query and passage and outputs a relevance logit. It is too slow to run on the corpus, which is why it sits after fusion. Budget it: a 100-pair re-rank on a small encoder is often 30–80ms; a large encoder on 200 pairs will blow a 200ms retrieval SLO. Cache re-rank scores for repeated query–chunk pairs. If you cannot afford a cross-encoder, take the RRF head as-is rather than inserting a second dense bi-encoder that mostly duplicates the first retriever.
Filters, indexes, and latency budgets
Structured predicates — tenant, ACL, language, document type, updated-after — belong in the retrievers, not in a Python loop after the fact. Post-filtering a top-50 that was retrieved without the predicate under-recalls: the relevant authorized chunk never entered the list. Push filters into the vector store and the inverted index so both lists are already legal. Permissioned retrieval (query-time ACL, no existence leak) is a sibling architecture; hybrid search is the ranking layer that sits on top of it.
Split the latency budget in writing. A typical interactive RAG SLO is 200–400ms for retrieval including re-rank, leaving the rest of the turn for generation. Dense ANN (HNSW, IVF) and BM25 should each finish well under 50ms at corpus sizes of tens of millions of chunks. The re-ranker is the lever: reduce pairs, use a smaller encoder, or skip it on queries whose fused lists already agree on the top documents.
Failure modes
The common outage is not “search is down”; it is silent wrong ranking. Embedding-model or chunker version drift — queries embedded with model B against an index built with model A — looks like a quality regression and will not throw. Sparse and dense indexes that ingest on different clocks return different documents for the same version of a page. A re-ranker that prefers long, generic chunks will bury the one-line policy that actually answers the question. Score-sum fusion without RRF lets whichever retriever shouts louder win every query.
Operational failures cluster around filters and timeouts. A filter applied only to one retriever produces fused lists that re-introduce forbidden documents. A re-ranker timeout that falls back to an empty list is worse than falling back to the RRF order. Debug retrieval with the retrieved ids, ranks, and filters logged per query — not by reading the generated sentence.
- Index/embed version skew after a model or chunker change
- Raw-score addition instead of rank fusion
- Filter on one retriever only, or filter after generation
- Re-ranker timeout emptying the context instead of using RRF
- Verbose-chunk bias in the cross-encoder
Evals, and when not to use hybrid
Build a golden query set from real questions with labeled relevant chunk ids (not just “a good answer”). Report hit@5, hit@10, nDCG@10, and MRR, sliced by query shape: identifier, paraphrase, multi-hop, and filtered. Ablate: dense only, sparse only, RRF, RRF plus re-ranker. If dense-only already hits 0.9 on a FAQ corpus of 200 pages, hybrid is overhead. Keep retrieval metrics in CI; a generator eval that stays flat while hit@5 drops is how teams ship a nicer prompt over a broken index.
Skip hybrid when the corpus is tiny and fully keyword-addressable, when the SLO is tens of milliseconds and a cross-encoder will not fit, or when a single strong learned-sparse index already matches hybrid on your slices. Do not skip hybrid on an enterprise wiki, ticket corpus, or policy set — those query mixes are exactly the ones that punish vector-only systems.