[bug fix] Fix async actions are left in neural_sparse query

[zhichao-aws]

Description

The bug exists in multi-nodes and multi-shards environment. For neural_search query, the cluster will throw error "async actions are left".

The search action contains 2 phase, query and fetch. In multi nodes & multi shards case, nodes will finish the query phase normally. But there will be a serialization and deserialization at the start of fetch shard phase, and after deserialization the doRewrite method of constructed QueryBuilder(i.e. NeuralSparseQueryBuilder) will be called. However for this rewrite the QueryBuilder can not have async action otherwise an exception will be thrown.

In our current implementation, the model inference result is not included in the serialization/deserialization process, and it will register an async action when rewritten. And this will break the search.

Add queryTokensSupplier to the process of serialization/deserialization, as well as doEquals and doHashCode

Issues Resolved

[List any issues this PR will resolve]

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[zhichao-aws]

should be in 2.11

[codecov]

Codecov Report

Merging #438 (462b614) into 2.11 (73cab3b) will increase coverage by 0.04%.
The diff coverage is 86.66%.

@@             Coverage Diff              @@
##               2.11     #438      +/-   ##
============================================
+ Coverage     80.71%   80.75%   +0.04%     
- Complexity      502      510       +8     
============================================
  Files            41       41              
  Lines          1576     1590      +14     
  Branches        241      247       +6     
============================================
+ Hits           1272     1284      +12     
  Misses          201      201              
- Partials        103      105       +2     

Files	Coverage Δ
...h/neuralsearch/query/NeuralSparseQueryBuilder.java	67.62% <86.66%> (+2.02%)	⬆️

[martin-gaievski]

nit: we need to change formatting in this method, every if case body must be in curly braces, plus for null/not null checks we can utilize Objects static methods. I'm fine with doing this is future, not a blocker for this PR

[heemin32]

Let's handle the issue when we backport it to 2.x

[martin-gaievski]

can we create a new instance of ByteStreamOutput instead of re-using existent one. That improves readability and lower chance of error. Same for line 306, instance of NamedWriteableAwareStreamInput. Can be done in followup PR

[heemin32]

Let's handle the issue when we backport it to 2.x or as a follow up pr.

[martin-gaievski]

is this a backward compatible change? I think for 2.10 and below versions there will be no boolean at all.
Plus the read - I think you can do same with readOptional method

[martin-gaievski]

to clarify my previous comment - that can be not backward compatible in terms of a rolling cluster upgrades. With that different cluster nodes may run different versions, say cluster is now on 2.8 and user wants to do rolling upgrade to 2.11 (some general documentation on this).
In such case query may be serialized at 2.8 node (without new field at all) and deserialized an a 2.11 node. If we just try to read the boolean there will be a runtime exception. We need to check minimal supported version before even reading boolean flag.
The reverse scenario is also possible: query serialized on 2.11 node and read on 2.8 node, in such case we shouldn't write to the stream.

[heemin32]

This class is newly introduced in 2.11. No issue with backward compatibility.

[martin-gaievski]

Looks good to me, BWC is not an issue as it's in new class, for others we can take care of them in next PRs

[opensearch-trigger-bot]

The backport to main failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-main main
# Navigate to the new working tree
cd .worktrees/backport-main
# Create a new branch
git switch --create backport/backport-438-to-main
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 51e6c00770d27fb4eabc20c38bdeff23c5c45997
# Push it to GitHub
git push --set-upstream origin backport/backport-438-to-main
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-main

Then, create a pull request where the base branch is main and the compare/head branch is backport/backport-438-to-main.