core[patch]: improve index/aindex api when batch_size<n_docs

[federico-pisanu]

• Description: prevent index function to re-index entire source document even if nothing has changed.
• Issue: Bug in Indexing Function Causes Inconsistent Document Deletion #22135

I worked on a solution to this issue that is a compromise between being cheap and being fast.
In the previous code, when batch_size is greater than the number of docs from a certain source almost the entire source is deleted (all documents from that source except for the documents in the first batch)
My solution deletes documents from vector store and record manager only if at least one document has changed for that source.

Hope this can help!

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		Sep 30, 2024 8:52pm

[baskaryan]

could we add unit tests for this?

[federico-pisanu]

@baskaryan yeah sure!
I checked the current tests and noticed that they where not properly testing the functionality. Patching the timestamps but not changing the patch between the "index" calls it was not properly simulating consequent calls. The test were passing but they were not testing the right thing.
Updating the tests was sufficient to check that the new code works and fixes the old behavior.

[federico-pisanu]

Hi @eyurtsev can i do something to make it easier to merge?

[eyurtsev]

Standby we're working on the 0.3 release so there was an effective code freeze for the past 2 weeks

[eyurtsev]

Hi @federico-pisanu,

Thank you for the PR! In order to merge the changes, the code would need to do the following:

1. Apply similar changes to the aindex API
2. Revert modifications to existing unit tests (these modifications make it look like the PR is introducing a new bug!)
3. Add a new unit tests that cover the relevant scenario.

[eyurtsev]

@federico-pisanu let me double check the existing unit tests to make sure they were wrong

[eyurtsev]

@federico-pisanu OK confirmed -- no action is required from you at this time. I'll re-apply changes on aindex

[federico-pisanu]

@eyurtsev Isn't it already done?

[eyurtsev]

@federico-pisanu yep didn't notice -- looks good added unit tests to cover the optimization -- i.e., we get different results based on which batch was mutated