limit amount(20) documents return after using filter in retriever

[BarryC7]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

sources = {'b.txt','c.txt'}
all_sources = {'a.txt','b.txt','c.txt'}

retriever = vectordb.as_retriever(search_kwargs={'k': 25,'filter': {'source':list(all_sources)}})
print(len(retriever.invoke(question))) #return 20
retriever.invoke(question)

retriever = vectordb.as_retriever(search_kwargs={'k': 15,'filter': {'source':list(all_sources)}})
print(len(retriever.invoke(question))) #return 15
retriever.invoke(question)

retriever = vectordb.as_retriever(search_kwargs={"k": int(25)})
print(len(retriever.invoke(question))) #return 25
retriever.invoke(question)

Description

Since i want to use a filter to only retrieve documents from some specific papers, i saw some references:
#21965
https://api.python.langchain.com/en/latest/vectorstores/langchain_community.vectorstores.faiss.FAISS.html#langchain_community.vectorstores.faiss.FAISS.as_retriever

but i fund that it will only return 20 documents when i using filter in retriever, i try to set filter with all papers or increase the value k, but it still return 20 documents.

System Info

Python Version: 3.10.14
langchain_core: 0.2.32
langchain: 0.2.14
langchain_community: 0.2.7
langsmith: 0.1.85
langchain_experimental: 0.0.63
langchain_google_genai: 1.0.8
langchain_google_vertexai: 1.0.7
langchain_huggingface: 0.0.3
langchain_text_splitters: 0.2.2

[dosubot[bot]]

Hey there, @BarryC7! I'm here to help you with any bugs, questions, or contributions you have in mind. Let's get to the bottom of this together!

I found a similar issue that might be relevant to your problem. The default value of the fetch_k parameter in the as_retriever method is 20, which limits the maximum number of documents fetched regardless of the k value specified. To retrieve more documents, you should adjust the fetch_k parameter accordingly ^[1].

Here's how you can modify your code to set the fetch_k parameter:

sources = {'b.txt', 'c.txt'}
all_sources = {'a.txt', 'b.txt', 'c.txt'}

retriever = vectordb.as_retriever(search_kwargs={'k': 25, 'fetch_k': 50, 'filter': {'source': list(all_sources)}})
print(len(retriever.invoke(question)))  # Should return up to 25 documents
retriever.invoke(question)

retriever = vectordb.as_retriever(search_kwargs={'k': 15, 'fetch_k': 50, 'filter': {'source': list(all_sources)}})
print(len(retriever.invoke(question)))  # Should return up to 15 documents
retriever.invoke(question)

retriever = vectordb.as_retriever(search_kwargs={'k': 25, 'fetch_k': 50})
print(len(retriever.invoke(question)))  # Should return up to 25 documents
retriever.invoke(question)

By setting fetch_k to a value larger than k, you ensure that enough documents are fetched before filtering, allowing the retriever to return the desired number of documents ^[2].

To continue talking to Dosu, mention @dosu.

[BarryC7]

If I use fetch_k then the search type will switch from similarity (default) to mmr?

[dosubot[bot]]

Yes, if you use fetch_k, the search type will switch from similarity (default) to max marginal relevance (MMR) ^[1][2].

To continue talking to Dosu, mention @dosu.