Hybrid Search for RAG: Dense + Sparse Done Right

Dense vector search understands meaning but fumbles exact identifiers. Sparse keyword search nails identifiers but misses intent. Production RAG needs both — done right. This is a practical guide to hybrid retrieval.

Why Single-Modality Retrieval Fails

Real query logs are a mix:

• "What's our parental leave policy?" — semantic, dense wins
• "Error QX-4412 on invoice ingestion" — exact match, sparse wins
• "Latest amendment to the Vora master services agreement" — needs both

Dense-only systems miss financial terminology, product codes, and names. Sparse-only systems fail on paraphrase and intent. In my enterprise deployments, hybrid retrieval improved recall by 30–45% over either alone.

The Architecture

query ──► dense retriever (top 25) ──┐
      └─► sparse retriever (top 25) ─┤──► RRF fusion ──► cross-encoder rerank ──► top 6

Reciprocal Rank Fusion

Never try to normalize and average raw scores — BM25 and cosine similarity live on incompatible scales. RRF sidesteps this by fusing on rank:

def rrf(rankings: list[list[str]], k: int = 60) -> list[str]:
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
    return sorted(scores, key=scores.__getitem__, reverse=True)

The constant k=60 is remarkably robust across domains — tune it last, if ever.

Weighting the Modalities

When one modality should dominate (e.g., a code-search product), use weighted RRF or run a lightweight query classifier that routes: identifier-looking queries lean sparse, natural-language questions lean dense. Route, don't average, when the query type is confident.

Reranking: Where Precision Comes From

Fusion gives you good recall in the top ~50. A cross-encoder (BGE reranker, Cohere Rerank, or a fine-tuned MiniLM) then buys precision:

pairs = [(query, doc.text) for doc in fused[:50]]
scores = reranker.predict(pairs)
top = [d for d, _ in sorted(zip(fused, scores), key=lambda x: -x[1])][:6]

Expect P95 latency cost of 80–150 ms — usually the best latency purchase in the whole pipeline.

Evaluation Setup

Build a labeled set of (query, relevant-passages) pairs and track per-modality and fused metrics separately: hit rate@k, MRR, nDCG. The interesting insight is always in the segmentation — which query classes does each modality win? That tells you where to invest next.

Implementation Notes

• OpenSearch and Elasticsearch give you BM25 + kNN in one engine; pgvector + tsvector works well in Postgres shops
• Keep permission filters inside both retrievers so fusion never sees unauthorized documents
• Cache the sparse index in memory — it's small and hot
• Log fused ranks per result; you'll need them to debug "why did this passage rank #1?"

Hybrid retrieval is the highest-ROI upgrade path for most RAG systems: no new models to train, meaningful recall gains, and it composes cleanly with everything downstream.