community : [bugfix] Use document ids as keys in AzureSearch vectorstore #25486
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Skipped Deployment
|
dosubot
bot
added
size:S
|
@eyurtsev please review or assign to someone! Thanks! |
|
Is there anything I can do to move this PR forward? Please let me know! |
|
Pretty please, could someone please take a look? We're using this fix in our pipeline and have to keep using forked version until the PR gets through. Please let me know if I can do anything to speed up the process! |
dosubot
bot
added
size:M
|
@ccurme added a unit test. Please take a look. Thanks! |
| else: | ||
| key = str(uuid.uuid4()) | ||
| # Encoding key for Azure Search valid characters | ||
| key = base64.urlsafe_b64encode(bytes(key, "utf-8")).decode("ascii") |
There was a problem hiding this comment.
right now if a user provides keys, we encode them. Here we're changing that. Is this a breaking change? Should we encode in all circumstances?
There was a problem hiding this comment.
The documents are passed to azure.search.documents.client.upload_documents() . Nowhere in the documentation is mentioned that the ids should be url encoded. My guess is that the keys were url encoded just as an extra precaution.
I left it there for the case when the keys are generated by uuid4() however in cases when keys/ids are passed in, url encoding them breaks stuff.
For instance when indexer is used to manage documents in the azure vector store, it passes ids to .add_texts() and expects that those ids are then used as-is (and stores them in the record manager under provided ids). When .add_embeddings() url-encodes them, the ids in vector store do not match ids in record manager. This breaks indexing.
In general when I pass ids/keys to a method I don't expect them to be silently changed in the background. Better alternative would be checking validity of provided keys/ids, however the documentation stats that the key is of type Edm.String which is defined here as <any UTF-8 character>' Note: See definition of UTF8-char in [RFC3629]
There is a limit of 1024 chars which could be checked, and the keys should be unique what could be checked but only for the documents currently being uploaded - and therefore I think it is better left for the user to ensure uniqueness.
There was a problem hiding this comment.
@ccurme sorry for late reply! I actually replied to your question 4 days ago but didn't hit "start review" (I assumed the comment would be still visible). Only today I found out that the comment does not show up if the review is not "started".
Please take a look and let me know whether it answers your questions and whether you agree. Thanks!
There was a problem hiding this comment.
@ccurme Please let me know if I can expand on the explanation I gave. Or please let me know what can I do to move forward. The issue I'm trying to fix is that as it stands right now, the indexer does not work with Azure at all. Thank you!
dosubot
bot
added
the
lgtm
Description
Vector store base class currently expects
idsto be passed in and that is what it passes along to the AzureSearch vector store when attempting toadd_texts(). However AzureSearch expectskeysto be passed in. When they are not present, AzureSearchadd_embeddings()makes up new uuids. This is a problem when trying to run indexing. Indexing code expects the documents to be uploaded using provided ids. Currently AzureSearch ignoresidspassed fromindexingand makes up new ones. Later whenindexerattempts to delete removed file, it uses theidit had stored when uploading the document, however it was uploaded under differentid.Twitter handle: @martintriska1