RAG Reranker and Hybrid Retrieval in Python: BM25, Vectors, RRF and a Cross-Encoder, Measured

Whether hybrid search or a reranker improves your RAG retrieval depends on your queries, and you can find out in about 80 lines of Python. I measured BM25, vector search, reciprocal rank fusion (RRF) of the two, and a cross-encoder reranker on top, using recall@k on two synthetic sets. On the easy set, plain vector search won and both upgrades slightly hurt. On the hard set, vector search collapsed and hybrid plus rerank hit 1.00. That split is the useful finding.

This takes a different angle from the earlier hybrid search post, which measures BM25 plus embeddings on exact IDs versus paraphrases. Here I add a reranker stage and a stress test. For where retrieval sits in the whole pipeline, see RAG pipeline architecture, stage by stage.

What did I run, and what is synthetic?

Run on 2026-10-23 in a fresh virtual environment: pip install rank_bm25 fastembed numpy.

• Embeddings: BAAI/bge-small-en-v1.5, local, through fastembed.
• Reranker: Xenova/ms-marco-MiniLM-L-6-v2, a small cross-encoder, local, through fastembed.
• BM25: the rank_bm25 package (Okapi), tokenised with \w+ after lowercasing.
• No LLM was involved. Retrieval only, so there is nothing to stub.
• All data is synthetic, about a made-up file-sync product called Fernwick. I wrote it, so the "right answer" labels are mine.

Set A (easy): 40 hand-written passages and 43 queries in four kinds: 10 exact error codes ("FW-4012"), 10 paraphrases of those errors with no shared rare words, 12 plan-specific questions where three near-identical passages differ by plan name ("how long does Starter keep deleted files"), and 11 how-to questions phrased differently from the docs ("read my files on a plane" for "make available offline"). One correct passage per query. (The Set A script is the one below plus a loader and a per-kind table; the data file is not reproduced here.)

Set B (hard): 120 generated passages of the form Error FW-5331: the token refresh step failed during restore. Retry in a few minutes. with random fillers, so passages differ mostly by their code. 60 queries, each just the bare code. The full script is further down.

Both are small. Treat the numbers as an illustration of a method, not as a benchmark of these models.

How do the retrievers work?

1. BM25 scores passages by overlapping words, weighting rare words more.
2. Vector ranks by cosine similarity between normalised embeddings.
3. Hybrid (RRF) takes the top 20 from each and sums 1 / (60 + rank) per passage. Rank-based, so you never reconcile BM25 scores with cosine scores.
4. Rerank: take the top 10 candidates from vector (or from hybrid) and re-score each with the cross-encoder, which reads the query and passage together.

That gives five methods in the tables: bm25, vector, hybrid_rrf, vector+rerank, hybrid+rerank.

Set A: when vector search alone wins

Real output:

40 passages, 43 queries (synthetic)
method            R@1   R@3   R@5    MRR
bm25             0.67  0.84  0.86  0.763
vector           0.91  0.98  1.00  0.947
hybrid_rrf       0.84  0.91  0.93  0.881
vector+rerank    0.88  0.98  0.98  0.926
hybrid+rerank    0.88  0.93  0.93  0.903

Recall@1 by query kind
method               howto         id paraphrase       plan
bm25                  0.55       1.00       0.50       0.67
vector                0.91       0.90       0.90       0.92
hybrid_rrf            0.73       1.00       0.70       0.92
vector+rerank         0.82       1.00       0.70       1.00
hybrid+rerank         0.82       1.00       0.70       1.00

Read it row by row.

Vector alone got the best MRR (0.947). Even on exact IDs it scored 0.90 at rank 1: with only 40 passages, "FW-4012" still sits closest to the passage that contains it, and the single miss was "how to fix FW-4103", ranked second.

Hybrid was worse than vector (0.84 vs 0.91 at rank 1). BM25 did fine on IDs (1.00) but poorly on paraphrases (0.50) and how-tos (0.55), and because RRF gives BM25's list an equal voice, its bad ranks pulled correct vector answers down. Fusion is not free: a weak retriever in the mix is not harmless.

The reranker helped one kind and hurt another. On plan questions it reached 1.00 at rank 1; it reads "Team" in the query next to "Team plan" in the passage, which is what a cross-encoder is good at. On paraphrases it fell from 0.90 to 0.70. In the hybrid-plus-rerank run, two of the misses had rank 999, meaning the right passage was not among the 10 candidates, so no reranker could rescue it:

rank 999 [howto] 'the app is eating all my wifi' -> wanted p_bandwidth
rank 999 [howto] 'exclude node_modules from syncing' -> wanted p_ignore

The second query is the same vocabulary gap that made the reranker drop the right section in the pipeline post. The passage never mentions node_modules, and a keyword retriever cannot invent the link.

Set B: when vector search collapses

The stress test, as one standalone script. The data are generated by the code.

import random
import re
from collections import defaultdict

import numpy as np
from fastembed import TextEmbedding
from fastembed.rerank.cross_encoder import TextCrossEncoder
from rank_bm25 import BM25Okapi

# SYNTHETIC: 120 near-identical error-code passages for a made-up product.
random.seed(7)
nouns = ["thumbnail", "index", "checksum", "manifest", "chunk upload", "permission sync", "preview", "rename", "delta patch", "token refresh"]
stages = ["upload", "download", "background scan", "restore", "share", "migration"]
fixes = ["Retry in a few minutes.", "Restart the client.", "Contact support with the log file.", "Re-link the device."]
codes = random.sample(range(5000, 5999), 120)
texts = [f"Error FW-{c}: the {random.choice(nouns)} step failed during {random.choice(stages)}. {random.choice(fixes)}" for c in codes]

tok = lambda s: re.findall(r"\w+", s.lower())
bm25 = BM25Okapi([tok(t) for t in texts])
emb = TextEmbedding("BAAI/bge-small-en-v1.5")
ce = TextCrossEncoder("Xenova/ms-marco-MiniLM-L-6-v2")
D = np.array(list(emb.embed(texts)))
D = D / np.linalg.norm(D, axis=1, keepdims=True)


def vector_rank(q):
    v = next(iter(emb.query_embed(q)))
    v = v / np.linalg.norm(v)
    return list(np.argsort(-(D @ v)))


def bm25_rank(q):
    return list(np.argsort(-bm25.get_scores(tok(q))))


def rrf(rankings, k=60, depth=20):
    s = defaultdict(float)
    for r in rankings:
        for pos, i in enumerate(r[:depth], start=1):
            s[i] += 1 / (k + pos)
    return sorted(s, key=lambda i: -s[i])


def rerank(q, cand):
    sc = np.array(list(ce.rerank(q, [texts[i] for i in cand])))
    return [cand[j] for j in np.argsort(-sc)]


methods = {
    "bm25": bm25_rank,
    "vector": vector_rank,
    "hybrid_rrf": lambda q: rrf([bm25_rank(q), vector_rank(q)]),
    "vector+rerank": lambda q: rerank(q, vector_rank(q)[:10]),
    "hybrid+rerank": lambda q: rerank(q, rrf([bm25_rank(q), vector_rank(q)])[:10]),
}
queries = [(f"FW-{c}", i) for i, c in enumerate(codes[:60])]  # query is the bare code
print(len(texts), "passages,", len(queries), "code-only queries")
for name, fn in methods.items():
    ranks = []
    for q, t in queries:
        r = fn(q)
        ranks.append(r.index(t) + 1 if t in r else 999)
    n = len(ranks)
    print(f"{name:14} R@1 {sum(x <= 1 for x in ranks)/n:.2f}  R@5 {sum(x <= 5 for x in ranks)/n:.2f}  R@10 {sum(x <= 10 for x in ranks)/n:.2f}")
print(texts[0])

Real output:

120 passages, 60 code-only queries
bm25           R@1 1.00  R@5 1.00  R@10 1.00
vector         R@1 0.55  R@5 0.85  R@10 0.90
hybrid_rrf     R@1 0.97  R@5 1.00  R@10 1.00
vector+rerank  R@1 0.90  R@5 0.90  R@10 0.90
hybrid+rerank  R@1 1.00  R@5 1.00  R@10 1.00
Error FW-5331: the token refresh step failed during restore. Retry in a few minutes.

Vector search found the right passage first for only 55 percent of bare codes, and 10 percent were not in its top 10 at all. BM25 was perfect, because the code is the only rare token. RRF recovered nearly all of that (0.97), and the reranker over the hybrid candidates finished the job. Note that vector+rerank stayed at 0.90 at every k: the reranker reordered the 10 candidates but could not add the 10 percent the vector stage never surfaced. A reranker is a second-stage sharpener, not a rescue for missing candidates.

This set is deliberately unfair to embeddings. Real codes, SKUs, function names and error strings sit in real corpora with other text around them, which helps embeddings somewhat. The point is the direction: the more your queries depend on rare literal tokens, the more BM25 earns its place.

When does hybrid beat vector-only?

From these two runs, not as a law:

Mixed corpora are the usual case in production, which is why people run hybrid by default. Just count it against vector-only on your own queries before you keep it.

How do I test this on my own data?

You need 30 to 50 real questions, each with the passage that answers it. Take them from support tickets, search logs or colleagues, not from the passages themselves, because questions written while reading the passage share its vocabulary and flatter keyword search.

1. Tag each query with a kind (ID, paraphrase, near-duplicate, how-to).
2. Run all five methods and print recall@1, @5, @10 by kind.
3. List the misses with their ranks. Rank 999 means a candidate-stage failure.
4. Pick the simplest method whose misses you can live with.

Latency also matters. The cross-encoder scores every candidate against every query, so 10 candidates is 10 model passes per question. I did not time it here, so measure it on your hardware before you put it in a request path.

Swapping in hosted embeddings and rerankers

The only changes are emb.embed and ce.rerank. Any provider with an embeddings endpoint works, and hosted rerank endpoints exist from Cohere, Voyage and Jina, among others (I did not call any of them here). Disclosure: the AICredits link is an affiliate link and costs you nothing. If you are in India and want to pay for an OpenAI-compatible API by UPI, AICredits lets you start from Rs 50 with the base URL https://api.aicredits.in/v1; see the review and the base URL setup guide. I have not tested embeddings or reranking through it, so check the catalog first. Direct OpenAI, Voyage and Cohere accounts are the non-affiliate alternatives.

Whichever you pick, re-embed the whole corpus when you change the embedding model, and rerun the measurement above. Then add RAG evaluation metrics for the generation side.

What I did not test

• Larger corpora. Forty and 120 passages are tiny, and approximate-nearest-neighbour indexes change behaviour at scale.
• Other embedding models or larger rerankers. A bigger cross-encoder may behave differently on the paraphrase misses.
• RRF constants other than 60 and weighted fusion.
• Latency, memory and cost.
• Real user queries. Mine are written by one person.