Hybrid search for RAG: combining BM25 and vector search
Vector search finds text with similar meaning even when the words differ, and it often misses exact terms like product codes, error codes, names and numbers. Keyword search, usually BM25, finds exact terms reliably and misses paraphrases and synonyms. Hybrid search runs both and merges the results, most simply with reciprocal rank fusion, which ranks a chunk higher when both methods rank it high. For most production RAG systems, hybrid search plus reranking retrieves noticeably better than vector search alone.
Two kinds of search, two kinds of failure
Keyword search matches words. Search "refund" and you get documents containing "refund". It is precise, fast and predictable, and it fails when the user's words differ from the document's: "my money back" doesn't match "refund", and a spelling mistake can return nothing.
Vector search matches meaning. "My parcel never arrived" finds "missing shipments", because their embeddings are close. It understands context and intent, and it fails on exact identifiers: "ERR-4012" and "ERR-4021" mean almost the same thing to an embedding model, even though only one is right.
Real questions contain both kinds of information. "Why does the X200 show error E-17 after the firmware update?" needs exact matching on "X200" and "E-17" and semantic matching on the rest.
How BM25 works
BM25 is the standard keyword ranking algorithm, used in Elasticsearch, OpenSearch, Lucene and many others. It scores how well a document matches a query from three ideas:
- Term frequency: a document that mentions the query word more often is more relevant, but with diminishing returns, so the tenth mention adds less than the second.
- Inverse document frequency: rare words matter more. A match on "E-17" means much more than a match on "the".
- Length normalisation: a match in a short chunk counts more than the same match buried in a very long one.
It needs no training and no model, it is fast, and it is very good at exactly what embeddings are weak at.
Combining the results
Hybrid search runs keyword search and vector search separately, each returning a ranked list, and then merges them.
Reciprocal rank fusion
The simplest and most widely used method is reciprocal rank fusion (RRF). Each chunk gets a score of 1 / (k + rank) from each list it appears in, the scores are added, and the chunks are sorted by the total. k is a constant, often 60, that stops the top ranks from dominating completely.
RRF uses only the ranks, not the raw scores, which is useful because BM25 scores and cosine similarities are on completely different scales. A chunk that is ranked well by both methods ends up at the top.
import math
import re
from collections import Counter
def tokenize(text: str) -> list[str]:
return re.findall(r"[a-z0-9-]+", text.lower())
class BM25:
def __init__(self, docs: list[str], k1: float = 1.5, b: float = 0.75) -> None:
self.docs = [tokenize(d) for d in docs]
self.k1, self.b = k1, b
self.avg_len = sum(len(d) for d in self.docs) / len(self.docs)
df = Counter(term for d in self.docs for term in set(d))
n = len(self.docs)
self.idf = {t: math.log(1 + (n - f + 0.5) / (f + 0.5)) for t, f in df.items()}
def scores(self, query: str) -> list[float]:
out = []
for doc in self.docs:
tf = Counter(doc)
s = 0.0
for term in tokenize(query):
if term in tf:
num = tf[term] * (self.k1 + 1)
den = tf[term] + self.k1 * (1 - self.b + self.b * len(doc) / self.avg_len)
s += self.idf[term] * num / den
out.append(s)
return out
def rrf(rankings: list[list[int]], k: int = 60) -> list[int]:
total: dict[int, float] = {}
for ranking in rankings:
for rank, doc_id in enumerate(ranking, start=1):
total[doc_id] = total.get(doc_id, 0.0) + 1 / (k + rank)
return sorted(total, key=total.get, reverse=True)
docs = [
"Error E-17 on the X200 means the battery firmware failed to update.",
"If your laptop will not charge after an update, reinstall the power driver.",
"The X300 supports fast charging up to 65 watts.",
]
query = "X200 error E-17 after firmware update"
bm25_scores = BM25(docs).scores(query)
keyword_ranking = sorted(range(len(docs)), key=lambda i: bm25_scores[i], reverse=True)
vector_ranking = [1, 0, 2]
print("keyword:", keyword_ranking)
print("hybrid:", rrf([keyword_ranking, vector_ranking]))
The vector ranking here is written by hand to show a typical case: the embedding model prefers the general "won't charge after an update" chunk, while BM25 puts the exact "E-17" chunk first. Fusion keeps both strong candidates at the top and pushes the X300 chunk, which neither method liked, to the bottom, and a reranker then decides between the top two by reading them against the question. Without the keyword side, the E-17 chunk would depend entirely on the embedding model ranking it highly. In practice, use a library like rank_bm25 or your database's built-in BM25 rather than this version.
Weighted score fusion
The other approach normalises both scores to the same range and combines them with a weight, for example 0.7 for vectors and 0.3 for keywords. It gives you a dial to tune, and it needs that tuning on your data, because the right weight depends on your documents and queries.
Where to run hybrid search
- Many vector databases support hybrid search directly, like Qdrant, Weaviate, Milvus and Pinecone, often with sparse vectors for the keyword side.
- Elasticsearch and OpenSearch combine their strong BM25 with vector search.
- In PostgreSQL, you can combine pgvector with full-text search.
- For small projects, run BM25 in Python and merge with RRF, as above.
Sparse embeddings
There is also a middle option: learned sparse models like SPLADE, which produce keyword-like vectors where the model has also learned related terms, so "refund" can activate "money back". They behave like a smarter BM25, and some systems use them as the keyword side of hybrid search.
When hybrid search matters most
- Product names, model numbers, SKUs and part numbers.
- Error codes, ticket IDs and invoice numbers.
- Names of people, companies and places.
- Legal and medical terms, where the exact term matters.
- Short queries, where there is little meaning for an embedding to work with.
- Languages or jargon the embedding model handles poorly.
Test it
Add questions with exact identifiers to your evaluation set, and compare recall for vector-only, keyword-only and hybrid retrieval. Hybrid usually wins, but measure it, and tune the fusion on your data.
From the floor
For Neural Vault, the offline RAG app for sensitive documents in AI Engineer HQ, the retrieval requirement was hybrid search from the start: lexical search for exact keyword matches and vector search for meaning, the same combination search engines use. A keyword search for a city name doesn't need a model or a vector database, and a question asked in other words, or with spelling mistakes, does, so you integrate both.