[ML] testGetDeploymentStats_WithStartedStoppedDeployments failure

[benwtrent]

Repro line:

./gradlew ':x-pack:plugin:ml:qa:native-multi-node-tests:javaRestTest' --tests "org.elasticsearch.xpack.ml.integration.PyTorchModelIT.testGetDeploymentStats_WithStartedStoppedDeployments" -Dtests.seed=5E2AFE7B6589657 -Dbuild.snapshot=false -Dtests.locale=ar-TN -Dtests.timezone=America/Campo_Grande -Druntime.java=17 -Dlicense.key="x-pack/license-tools/src/test/resources/public.key"

Reproduces locally?:
yes
Applicable branches:
master
Failure excerpt:

org.elasticsearch.xpack.ml.integration.PyTorchModelIT > testGetDeploymentStats_WithStartedStoppedDeployments FAILED
    org.elasticsearch.client.ResponseException: method [POST], host [http://127.0.0.1:64466], URI [/_ml/trained_models/foo/deployment/_infer], status line [HTTP/1.1 500 Internal Server Error]
    {"error":{"root_cause":[{"type":"status_exception","reason":"Error in inference process: [inference native process died unexpectedly with failure [[2:3] [pytorch_result] unknown field [request_id]]]"}],"type":"status_exception","reason":"Error in inference process: [inference native process died unexpectedly with failure [[2:3] [pytorch_result] unknown field [request_id]]]"},"status":500}
        at __randomizedtesting.SeedInfo.seed([5E2AFE7B6589657:157DC45ADCA763B5]:0)
        at app//org.elasticsearch.client.RestClient.convertResponse(RestClient.java:346)
        at app//org.elasticsearch.client.RestClient.performRequest(RestClient.java:312)
        at app//org.elasticsearch.client.RestClient.performRequest(RestClient.java:287)
        at app//org.elasticsearch.xpack.ml.integration.PyTorchModelIT.infer(PyTorchModelIT.java:752)
        at app//org.elasticsearch.xpack.ml.integration.PyTorchModelIT.testGetDeploymentStats_WithStartedStoppedDeployments(PyTorchModelIT.java:360)

[elasticmachine]

Pinging @elastic/ml-core (Team:ML)

[benwtrent]

OK, when build snapshot is false, the build provided is WAY too old. It doesn't have commit: elastic/ml-cpp@3a628c1 in it.

In fact its from commit: 6dde25e6ff38a3

[droberts195]

Hopefully https://elasticsearch-ci.elastic.co/view/Machine%20Learning/job/elastic+machine-learning+main/98 will fix this

[imotov]

Just failed in release tests in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request+release-tests/32/consoleText not sure if it was after or before the fix was applied.

[droberts195]

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+pull-request+release-tests/32/ says Build #32 (06-Jan-2022 20:05:49).

The upload of the artifacts that fixed this happened in https://elasticsearch-ci.elastic.co/job/elastic+machine-learning+main+combine-artifacts/90/, which was Build #90 (06-Jan-2022 21:55:01).

So that explains why the correct artifacts were not available at the required time.

I believe this is fixed now.