[serve] Remove log poll client from replicas

[edoakes]

Why are these changes needed?

In general, broadcasting changes to the replicas via the LongPollClient is hard to reason about (it circumvents our versioning semantics as there's no rolling update). Ideally we would only be using the LongPollClient to broadcast replica membership and nothing else.

Related issue number

Closes #19145

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[simon-mo]

This is not a user-visible change because this option was not able to be parametrized anyways.

I actually think we should not bake shutdown time to constants, rather, just let the replica take backend config in the init args.
It should be parameterized option because it is useful adjust this parameter properly (not through monkey patching constants) for various testing, release testings, and fault injection testing. Additionally, this will have impact for how long can a replica scale down (relevant to autoscaling). We just need to do little work to make sure it's configurable (and remove experimental now we finalized the mechanism).

[edoakes]

@simon-mo fair point, let me refactor and make it a hidden field in .options for now

[edoakes]

@simon-mo updated

[simon-mo]

LGTM. You might need to fix Java compilation failures.