How to set threshold to ensemble retriever and filter hybrid search results by score?

[ak4hcl]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

# Run hybrid chain
self.ensemble_retriever= self.create_hybrid_retriever(self.retriever1,self.bm25_retriever)
self.history_aware_retriever = create_history_aware_retriever(
            self.llm, self.ensemble_retriever, contextualize_q_prompt
        )
rag_chain = create_retrieval_chain(self.history_aware_retriever, question_answer_chain)
# rag chain with context history
self.conversational_rag_chain = RunnableWithMessageHistory(
            rag_chain,
            self.get_session_history,
            input_messages_key="input",
            history_messages_key="chat_history",
            output_messages_key="answer",
        )
result=self.conversational_rag_chain.invoke({"input": question})

Description

I want to avoid RAG for casual message by user or greetings like "hey there!", "hi", hello", "What's the time now?" and so on.
I can only do it by setting the threshold to filter the results by their score. But -neither ensemble retriever nor BM25 retriever are returning score while I search. Please suggest some ideas.

System Info

# pip freeze | grep langchain
langchain==0.2.5
langchain-chroma==0.1.1
langchain-community==0.2.5
langchain-core==0.2.9
langchain-openai==0.1.8
langchain-text-splitters==0.2.1
langchainhub==0.1.20

Windows/Linux
Python version: 3.11.9

[dosubot[bot]]

Hey there, @ak4hcl! 👋 I'm here to assist you with bugs, questions, and becoming a contributor. Let's squash those bugs together!

To set a threshold for an ensemble retriever and filter hybrid search results by score, you can modify your retrievers to return scores and then filter the results based on these scores. Here’s how you can achieve this:

1. Modify the bm25_retriever to return scores:

   Wrap the underlying vector store's .similarity_search_with_score method to include scores in the document metadata.

   from typing import List
   from langchain_core.documents import Document
   from langchain_core.runnables import chain
   
   @chain
   def bm25_retriever(query: str) -> List[Document]:
       docs, scores = zip(*vectorstore.similarity_search_with_score(query))
       for doc, score in zip(docs, scores):
           doc.metadata["score"] = score
       return docs

2. Integrate score filtering into the create_hybrid_retriever function:

   Modify the function to ensure both retriever1 and bm25_retriever return scores and filter the results based on a score threshold.

   from typing import List, Tuple
   from langchain_core.documents import Document
   from langchain_core.runnables import chain
   
   @chain
   def create_hybrid_retriever(query: str) -> List[Document]:
       # Retrieve documents and scores from retriever1
       docs1, scores1 = zip(*retriever1.similarity_search_with_score(query))
       for doc, score in zip(docs1, scores1):
           doc.metadata["score"] = score
   
       # Retrieve documents and scores from bm25_retriever
       docs2, scores2 = zip(*bm25_retriever.similarity_search_with_score(query))
       for doc, score in zip(docs2, scores2):
           doc.metadata["score"] = score
   
       # Combine results from both retrievers
       combined_docs = list(docs1) + list(docs2)
       
       # Filter results by score threshold
       score_threshold = 0.5  # Set your desired threshold here
       filtered_docs = [doc for doc in combined_docs if doc.metadata["score"] >= score_threshold]
       
       return filtered_docs

3. Modify the create_history_aware_retriever function to incorporate score filtering:

   Add a step to filter the retrieved documents based on their scores.

   def create_history_aware_retriever(
       llm: LanguageModelLike,
       retriever: RetrieverLike,
       prompt: BasePromptTemplate,
       score_threshold: float = 0.5  # Add a score threshold parameter
   ) -> RetrieverOutputLike:
       if "input" not in prompt.input_variables:
           raise ValueError(
               "Expected `input` to be a prompt variable, "
               f"but got {prompt.input_variables}"
           )
   
       def filter_by_score(documents):
           return [doc for doc in documents if doc.metadata.get("score", 0) >= score_threshold]
   
       retrieve_documents: RetrieverOutputLike = RunnableBranch(
           (
               lambda x: not x.get("chat_history", False),
               (lambda x: x["input"]) | retriever | filter_by_score,
           ),
           prompt | llm | StrOutputParser() | retriever | filter_by_score,
       ).with_config(run_name="chat_retriever_chain")
       return retrieve_documents

4. Update your code to use the modified functions:

   self.ensemble_retriever = create_hybrid_retriever(self.retriever1, self.bm25_retriever)
   self.history_aware_retriever = create_history_aware_retriever(
       self.llm, self.ensemble_retriever, contextualize_q_prompt, score_threshold=0.5
   )
   rag_chain = create_retrieval_chain(self.history_aware_retriever, question_answer_chain)
   self.conversational_rag_chain = RunnableWithMessageHistory(
       rag_chain,
       self.get_session_history,
       input_messages_key="input",
       history_messages_key="chat_history",
       output_messages_key="answer",
   )
   result = self.conversational_rag_chain.invoke({"input": question})

By following these steps, you can set a threshold to filter the results by their score, ensuring that casual messages or greetings are filtered out ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[ak4hcl]

@dosu I saw your last message but what is "vectorstore" in you code results = self.vectorstore.similarity_search_with_relevance_scores(query, k=self.k) ? What is it doing here while we are talking about keyword search (bm25) ?

[dosubot[bot]]

The term vectorstore refers to a storage mechanism used to store and retrieve documents based on their vector representations. In the context of BM25 keyword search, vectorstore can be used to store documents and perform similarity searches to retrieve documents that are most relevant to a given query.

However, the BM25Retriever class in LangChain does not inherently support returning relevance scores. To achieve this, you need to modify the BM25Retriever to include scores in its output. Here’s how you can do it:

1. Modify the BM25 Retriever to return scores:

   from langchain_community.retrievers import BM25Retriever
   
   class BM25RetrieverWithScores(BM25Retriever):
       def get_relevant_documents_with_scores(self, query):
           processed_query = self.preprocess_func(query)
           scores = self.vectorizer.get_scores(processed_query)
           docs_and_scores = [(doc, score) for doc, score in zip(self.docs, scores)]
           sorted_docs_and_scores = sorted(docs_and_scores, key=lambda x: x[1], reverse=True)
           return sorted_docs_and_scores[:self.k]
   
   # Initialize the modified BM25 retriever
   bm25_retriever = BM25RetrieverWithScores.from_texts(
       doc_list_1, metadatas=[{"source": 1}] * len(doc_list_1)
   )

2. Modify the Ensemble Retriever to filter results by scores:

   from langchain.retrievers import EnsembleRetriever
   
   def filter_by_score(docs_and_scores, min_score):
       return [doc for doc, score in docs_and_scores if score >= min_score]
   
   # Create the ensemble retriever
   ensemble_retriever = EnsembleRetriever(
       retrievers=[bm25_retriever, faiss_retriever], weights=[0.5, 0.5]
   )
   
   # Retrieve documents with scores
   docs_and_scores = bm25_retriever.get_relevant_documents_with_scores("apples")
   
   # Filter documents by score
   min_score = 0.5  # Set your threshold score
   filtered_docs = filter_by_score(docs_and_scores, min_score)

3. Integrate the modified retriever into your existing code:

   self.ensemble_retriever = EnsembleRetriever(
       retrievers=[self.retriever1, bm25_retriever], weights=[0.5, 0.5]
   )
   self.history_aware_retriever = create_history_aware_retriever(
       self.llm, self.ensemble_retriever, contextualize_q_prompt
   )
   rag_chain = create_retrieval_chain(self.history_aware_retriever, question_answer_chain)
   self.conversational_rag_chain = RunnableWithMessageHistory(
       rag_chain,
       self.get_session_history,
       input_messages_key="input",
       history_messages_key="chat_history",
       output_messages_key="answer",
   )
   result = self.conversational_rag_chain.invoke({"input": question})

By following these steps, you can modify the BM25Retriever to return scores and filter the results in your ensemble retriever based on these scores ^[1][2].

To continue talking to Dosu, mention @dosu.

[lingvisa]

@dosu, Similarily, if I have a faiss_vectorstore embedding, how to return the score? Because I need both to be scored

[lingvisa]

When you use ensemble_retriever, you have a unified interface to work with both retrievers, so how to make both return scores so that you can use the ensemble_retriever().

[skier-song9]

I also think @dosu is wrong about BM25Retriever things. But @ak4hcl it's possible to get scores from BM25Retriever. though it's still tricky to get scores from EnsembleRetriever.

BM25Retriever doesn't use similarity score to search documents. It uses BM25 score (ref : rank_bm25.py). So it doesn't make sence to get similarity score of ensemble retriever. Ensemble retriever works by "weighted_reciprocal_rank" not "cosine similarity". (ref : langchain-EnsembleRetriever)

Simply if you want to get bm25 score from BM25Retriever, just access to vectorizer and call get_score() function.
But if you want to get score from EnsembleRetriever, you may figure it out...
In my case, I just calculate cosine similarity again after invoking documents by EnsembleRetriever.

Example code:

from langchain_community.retrievers import BM25Retriever
bm25_retriever = BM25Retriever.from_documents(docs, **params)
print(bm25_retriever.vectorizer.get_scores("foo")) # this will return BM25 score of the query for each documents

I took some notes (in Koreanㅎㅎ) in my notion about how BM25Retriever works. >> skiersong9-notion