Pgvector - filters #257
Pgvector - filters #257
Conversation
| @pytest.mark.skip(reason="NOT operator is not supported in PgVectorDocumentStore") | ||
| def test_not_operator(self, document_store, filterable_docs): ... |
There was a problem hiding this comment.
This bothers me a bit.
Since we magically handle NULL values, as required by most of our base tests, it is difficult to properly support the NOT operator.
(For example, a != 3 becomes a IS DISTINCT FROM 3 to also include NULL values in the result.)
I also tried to support the NOT operator, but this involved inverting all nested conditions (which can also be logical) and started to get messy.
So the simplest solutions are:
- do not support the NOT operator
- drop magic handling of NULL values, consistently with Postgres behavior (but requires overriding many of our base tests).
Happy to discuss this...
There was a problem hiding this comment.
Let's start by not supporting NOT, if there's demand we can proceed with option number 2, knowing that the investment in patching the base tests is worth it.
| sql_query_str = sql_query.as_string(cursor) if not isinstance(sql_query, str) else sql_query | ||
| detailed_error_msg = f"{error_msg}.\nSQL query: {sql_query_str} \nParameters: {params}" | ||
| raise DocumentStoreError(detailed_error_msg) from e |
There was a problem hiding this comment.
Beware printing raw SQL queries, they can leak secrets and sensitive data.
| f"Could not write documents to PgvectorDocumentStore. \n" | ||
| f"SQL query: {sql_query_str} \nParameters: {db_documents}" | ||
| ) | ||
| raise DocumentStoreError(error_msg) from e |
There was a problem hiding this comment.
Same security considerations
integrations/pgvector/src/haystack_integrations/document_stores/pgvector/filters.py
Outdated
Show resolved
Hide resolved
integrations/pgvector/src/haystack_integrations/document_stores/pgvector/filters.py
Outdated
Show resolved
Hide resolved
integrations/pgvector/src/haystack_integrations/document_stores/pgvector/filters.py
Outdated
Show resolved
Hide resolved
integrations/pgvector/src/haystack_integrations/document_stores/pgvector/filters.py
Show resolved
Hide resolved
integrations/pgvector/src/haystack_integrations/document_stores/pgvector/filters.py
Outdated
Show resolved
Hide resolved
|
I have tried to address most of the comments and added some unit tests (some of these overlap with existing ones, but run very fast). |
| sql_query_str = sql_query.as_string(cursor) if not isinstance(sql_query, str) else sql_query | ||
| detailed_error_msg = f"{error_msg}.\nSQL query: {sql_query_str} \nParameters: {params}" | ||
| raise DocumentStoreError(detailed_error_msg) from e |
There was a problem hiding this comment.
We could print the SQL query in the debug logs and do something like:
detailed_error_msg = f"{error_msg}.\nYou can find the SQL query and the parameters in the debug logs"
It's not perfect but it should be ok for most of the cases
| @pytest.mark.skip(reason="NOT operator is not supported in PgVectorDocumentStore") | ||
| def test_not_operator(self, document_store, filterable_docs): ... |
There was a problem hiding this comment.
Let's start by not supporting NOT, if there's demand we can proceed with option number 2, knowing that the investment in patching the base tests is worth it.
Support for the Haystack filtering logic in PgvectorDocumentStore.