Chunking based on key values from the first page

[Divakar-kumar]

How should we ensure not to lose context while chunking documents used in RAG? Say for ex., you need to track key properties in the first page over other chunks

[pamelafox]

Hm, good question. What kind of documents are you using? I could imagine either:

• Repeating the key properties from the first page in each chunk
• Adding the key properties as an additional field in the search index, like pageproperties, and fetching those when doing the search() call in the RAG

[Divakar-kumar]

Thanks for the response.

For example in the document analysed below using DocumentIntelligence Layout model , we have date in the first page and let's assume the other pages don't have it. Now if I ask query to LLM to fetch important news for a specific date. It might have missed the context, as there are lot of chunks now without the key property date.

[pamelafox]

Thanks for sharing the example, that's great.

So I think the minimal change would be to include the date in every chunk, since then the search index has a chance of retrieving the right chunks.

However, if it's going to be a common use case to retrieve by date, that's what we'd call a "structured query", versus an unstructured query. When you want to support structured queries, we often recommend using GPT function calling/tools:
https://platform.openai.com/docs/guides/function-calling

To go down that route, you would:

1. Add the date as an additional field in the index
2. Add a function to chatreadretrieveread.py like "search_by_date", to the initial ChatCompletion call that turns the user query into a keyword search.
3. If GPT decides to call that function, then you would pass in the date as a category filter to the search query.

For examples of function calling, see:
https://gist.github.com/pamelafox/a3fdea186b687509c02cb186ca203328#file-chatreadretrieveread-py-L123
which supports fetching GitHub issues (so it uses a completely different API to search in that case)
and Azure-Samples/azure-search-openai-demo#1176 (which uses functions to escalate to a human)

[Divakar-kumar]

Thanks for detailed response @pamelafox .

In my case If I want to use Azure Cosmos DB Mongo vCore as my vector database, then I should just include list of key properties in every chunks. Or I'm not sure if this keyword + semanitc search is available in Cosmos DB as well. If yes, then I could proceed with Function calling as you mentioned.

[pamelafox]

I believe CosmosDB Mongo vCore does support keyword search (i.e. a full text search), though am not sure if it's easy to do a hybrid search (combining vector search and text search results into one list of ranked results). It might be that vector search will work well enough for your use case, particularly if you're going to use structured queries to filter by date. Anthony Shaw will be presenting on Function Calling at a session next week, by the way.