-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] function score query returned an invalid (negative) score with multi match cross fields query #7860
Comments
In this case, a term is in every field in all the documents: "red" and it also appears in the field "shape" which only exists in one doc. given n = 4 (number of documents containing the term) and N = 3 (total number of documents with the field), the IDF calculation: log(1 + (N - n + 0.5) / (n + 0.5)) where n = 4, documents containing term and N = 3: log(1 + (3 - 4 + 0.5)/( 4 + 0.5)) which produces -0.11778303. I also plugged this into another calculator and got -0.0511525224. We have two things to investigate:
extract from results with explain:
|
@snikoyo please do help us with a fix if you have time, making sure we're not copying any non-APLv2 code of course |
How do more documents have the term than have the field?! head-asplode |
Looking closing in the above shape_index example, I got some findings for the above two questions by validating the numbers from
So in this bug, the negative number is causing by N < n, which is the total number of documents with fields is smaller than the number of documents contains term when a document have more than one field containing the query term. Both N and n are not populated correctly. this is how it's calculated idf currently: ** for field To fix the idf calculated for matching multiple fields, is to have a logic to take idf calculation separate per field. ** for field In the same example of shape's idf score want some opinions about the proposed fix logic above... |
To simplify this bug, I tried getting rid of the function score query, and using the multi-match query with cross field and I can reproduce the negative score issue, so function score query is not related to this bug. I found out the negative score is due to when counting the Here I reuse @macohen 's documents
use multi-match query with cross field, getting rid of functional score query:
In the response for document id 2.
the explain is
The two fields, While looking at the DisjunctionMaxQuery, the cross field query is rewrite as ((color:red)^5 | (shape:red)^2), if the logic is working as expected, the document frequency should calculate by per field level. Will look further.. |
I think "Method 3: calculate idf per document, all fields sharing the same idf" makes the most sense. The BM25 calculation assumes that it is applied to terms from a single field. If a single term matches across multiple fields, the document and term frequencies should be calculated across those fields too, IMO. (Of course, if another term only matches on one of the fields, should we take the IDF of that term across all fields? Probably.) |
@msfroh This is an interesting problem which need careful design of algorithm. We need to consider kinds of edge cases, and it may impact existing users if the algorithm changed. |
One of the biggest con of " calculating idf per document, all fields sharing the same idf", the per field score will be skewed because of the weight. in the above query, the weight is different
If we apply the same idf per field, in this document, In PerFieldSimilarity, |
As far as I know, the real problem is less about the scores being "wrong" and more about the negative scores causing an exception. How about if somewhere around here, we say:
We should still offer a better cross-field score, but this would at least address the immediate pain point. The "real" solution is probably something like BM25F, as discussed in #3996 |
Agree with @msfroh on this, as addressing the negative scores and absorbing the exception could be the first step to smoothen out the experience and unblock users. This can be followed by better scoring implementations as a follow up step. |
It took a little while to produce a failing test, because the examples given by @macohen and @mingshl above don't return negative scores, because the dismax query that the Here's the YAML REST test that fails:
I'll post the cheap workaround (set the value to zero if negative) to make it succeed. |
Thanks @msfroh , sounds a plan to fix negative value first if no concern, then we can figure out a better way to calculate score. |
I actually found what I believe is a pretty safe fix for the underlying issue that should have no effect on any "normal" index, where the fields being searched are present in most documents: #13829 It only impacts scoring in the specific cases that may produce negative scores (where the maximum document frequency of any term exceeds the minimum number of docs containing a field, across all fields). |
Describe the bug
The cross_fields scoring type can produce negative scores when some documents are missing fields.
This bug was already reported for Elasticsearch and fixed with this MR for Elasticsearch >= 8.4.
We encounter it randomly on production for certain queries when we search a field that does not exist in all documents.
It appears deterministically for these search queries.
Error message:
{ "error": { "root_cause": [ { "type": "exception", "reason": "function score query returned an invalid score: -0.05202394 for doc: 1123896" } ], "type": "search_phase_execution_exception", "reason": "all shards failed", "phase": "query", "grouped": true, "failed_shards": [ { "shard": 0, "index": "my_index", "node": "my_node", "reason": { "type": "exception", "reason": "function score query returned an invalid score: -0.05202394 for doc: 1123896" } } ] }, "status": 500 }
The document number mentioned in the error message does not exist.
To Reproduce
The steps to reproduce are in the above mentioned bug report for Elasticsearch.
Expected behavior
The function score should never return a negative value.
Plugins
None.
Host/Environment (please complete the following information):
OpenSearch version: 2.5
The text was updated successfully, but these errors were encountered: