[Test] gcs_placement_group_manager_test flaky in the master

[rkooo567]

What happened + What you expected to happen

After merging this PR, test_placement_group became flaky. We should investigate and fix it.

cc @xwjiang2010 I will mark it as a release blocker until we find the root cause.

cc @larrylian

Versions / Dependencies

master

Reproduction script

n/a

Issue Severity

No response

[larrylian]

I will look at this issue. It should be only a test issue introduced by #24875 PR. Now it may be related to multi-threaded execution.

[larrylian]

#30577 @rkooo567 I have fix this case, I will see if this use case CI can run successfully tomorrow.

[cadedaniel]

I don't understand if #30577 fixes the root cause in TestRemovingPendingPlacementGroup or not but FYI TestRemovingLeasingPlacementGroup recently failed due to threading issues, if they're related then we should fix the root problem (multithreaded usage on a non-thread-safe class). I'm happy to help with that if it's a release blocker.

The hacky fix for TestRemovingLeasingPlacementGroup was to sleep to avoid a race #29694

[rkooo567]

We merged the PR. Let's see if it becomes unflaky after that

[rkooo567]

I will also add more tests as a follow up. So we shouldn't close this (but just downgrade the priority)

[xwjiang2010]

Just FYI, timeout still happens on release branch: https://buildkite.com/ray-project/oss-ci-build-branch/builds/1306#0184c0a2-9b0c-4ce7-905c-bb016a34d56d and
https://buildkite.com/ray-project/oss-ci-build-branch/builds/1306#0184c0a2-9b0e-4ebf-ba53-1a9146798b86

[rkooo567]

Hmm I guess this is not fixed by that PR... Let me take a look tomorrow.

[xwjiang2010]

Just confirming is this still release blocking or we can drop it?

[rkooo567]

Can you give me one more day for this? I will sort that out by today.

[rkooo567]

(is this the last blocker?)

[larrylian]

Hmm I guess this is not fixed by that PR... Let me take a look tomorrow.

I looked at the few flakey's, maybe the branch code is relatively old and didn't bring my fix PR.

[rkooo567]

@xwjiang2010 this is also a test issue. @cadedaniel it is the same issue as you mentioned. I found if I add until io service is empty, it passes 300 times in a row without an issue. I will fix this separately, but it doesn't need to be a release blocker.