Document rescore window_size anomalies

[worleydl]

Some users have encountered the scenario where their model tanks document scores. When this happens a rescore window can actually sink documents outside of the original window and propagate lower scoring documents on top! This is dangerous as it's not always obvious when it's occurring.

Document the behavior and some possible workarounds.

• Always adding a flat boost
• Using the max value
• Others?

[rudibatt]

There is another bad anomaly related to that, that mostly becomes obvious if a smaller window_size is used and the documents after the window_size are fetched:
Inside ES only the topN documents (Math.max(window_size, from + size)) are considered when merging the rescored documents with the non-rescored documents. This is why the negative-rescored documents always get at the end of the topN list and always will get returned in the response, no matter what "from" / offset is requested.
The responsible code is org.elasticsearch.search.rescore.QueryRescorer.combine(TopDocs, TopDocs, QueryRescoreContext)

Example:

• Assume a model calculates negative scores for a certain query
• window_size=48
• Fetching products from=48 with size=12 will return the worst 12 products that were rescored by the model (the ones that were on position 35-47 in the paging-request before)
• Fetching products from=60 with size=12 will return the same 12 products

Not sure if this a bug inside elasticsearch, the ltr plugin or just a bad designed model. Some documentation for a workaround would be a first way to go, but I guess this should also be fixed somewhere. Any advice?

[worleydl]

Thanks for describing another case Rudi. This sounds like a potential bug so I'd like to write a specific test case around it to get a better understanding of where the blame lies.

In the meantime you could try shifting all of the outputs for your model by some value so that there are never any negative values. I'm also curious if the approach of adding a constant boost to all of the rerank docs helps to alleviate your particular case or not. In theory the boost + [model output (even if negative)] is always over the window_size + 1 doc's score so things don't get sunk.

[rudibatt]

I could reproduce this bug with standard elasticsearch rescorers. It always happens when a rescorers tanks the scores, e.g. by multiplying a score < 1 with the original score:

{
  "from": 20,
  "size": 10,
  "query": {...},
  "rescore": [
    {
      "window_size": 20,
      "query": {
        "rescore_query": {
          "constant_score": {
            "filter": {
              "term": { "user.id": "kimchy" }
            },
            "boost": 0.1
          }
        },
        "score_mode": "multiply"
      }
    }
  ]
}

I am about to open a ticket for that at Elasticsearch...

[rudibatt]

This anomalies can be avoided by boosting the documents within the desired window_size into a higher value range. This is the solution that works for my usecase:

  "rescore": [
    {
      "query": {
        "rescore_query": {
          "constant_score": {
            "filter": {
              "match_all": {}
            },
            "boost": 1
          }
        },
        "query_weight": 0.001,
        "rescore_query_weight": 10,
        "score_mode": "total"
      },
      "window_size": 100
    },
    {
      "window_size": 100,
      "query": {
        "rescore_query": {
          "sltr": {
            "model": "my_model",
            "params": {
              "keywords": "##query##"
            }
          }
        },
        "query_weight": 1,
        "rescore_query_weight": 1,
        "score_mode": "total"
      }
    }
  ]

[rlps]

The way I deal with this is using the parameter "query_weight": 0 and "score_mode": total.

  "rescore": [
    {
      "window_size": 100,
      "query": {
        "rescore_query": {
          "sltr": {
            "model": "my_model",
            "params": {
              "keywords": "##query##"
            }
          }
        },
        "query_weight": 0,
        "rescore_query_weight": 1,
        "score_mode": "total"
      }
    }
  ]

This way the final score is: 0*first_query + 1*model_rescore. What I previously didn't know is that all the documents from the first query get the 0 rescore weight. This way, everything that is outside the rescore window receives a score=0 and for the ones inside the rescore window they're left only with the model score.

[rudibatt]

@rlps as you mention yourself, the scores of all other documents outside of the "rescore window" will be 0 with this approach. At least in my case this is not desired, as the order of those documents will be random (or maybe even inconsistent?).