[CI] windows://python/ray/tests:test_runtime_env_working_dir_3 is failing/flaky on master.

[architkulkarni]

• d1eae27 FAILED Buildkite :windows: Build & Test
• 9505cd1 FAILED Buildkite :windows: Build & Test
• 4d74b06 FAILED Buildkite :windows: Build & Test
• 87b7095 FAILED Buildkite :windows: Build & Test
• c4348c1 FAILED Buildkite :windows: Build & Test
• 9f1ea30 FAILED Buildkite :windows: Build & Test

....
Generated from flaky test tracker. Please do not edit the signature in this section.
DataCaseName-windows://python/ray/tests:test_runtime_env_working_dir_3-END
....

[architkulkarni]

Likely due to #28623 and we just need to increase a timeout somewhere.

[architkulkarni]

5/5 failures are in test_pin_runtime_env_uri[ray_client-tmp_working_dir-20]

Example: https://buildkite.com/ray-project/oss-ci-build-branch/builds/204#01837b47-7381-4a73-a193-98de0c9d35c1/8514-8542

[architkulkarni]

___________ test_pin_runtime_env_uri[ray_client-tmp_working_dir-20] ___________
--
  |  
  | start_cluster = (<ray.cluster_utils.Cluster object at 0x00000188AC732DC0>, 'ray://localhost:10004')
  | source = 'C:\\Users\\ContainerAdministrator\\AppData\\Local\\Temp\\tmptp14d912'
  | expiration_s = 20
  | monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x00000188AC7899A0>
  |  
  | @pytest.mark.parametrize("expiration_s", [0, 20])
  | @pytest.mark.parametrize("source", [lazy_fixture("tmp_working_dir")])
  | def test_pin_runtime_env_uri(start_cluster, source, expiration_s, monkeypatch):
  | """Test that temporary GCS URI references are deleted after expiration_s."""
  | monkeypatch.setenv(RAY_RUNTIME_ENV_URI_PIN_EXPIRATION_S_ENV_VAR, str(expiration_s))
  |  
  | cluster, address = start_cluster
  |  
  | start = time.time()
  | ray.init(address, namespace="test", runtime_env={"working_dir": source})
  |  
  | @ray.remote
  | def f():
  | pass
  |  
  | # Wait for runtime env to be set up. This can be accomplished by getting
  | # the result of a task that depends on it.
  | ray.get(f.remote())
  | ray.shutdown()
  |  
  | # Need to re-connect to use internal_kv.
  | ray.init(address=address)
  |  
  | print("Starting Internal KV checks at time ", time.time() - start)
  | if expiration_s > 0:
  | >           assert not check_internal_kv_gced()
  | E           assert not True
  | E            +  where True = check_internal_kv_gced()

[architkulkarni]

Newly flaky after #28589, should be an easy fix.
https://flaky-tests.ray.io/
Example: https://buildkite.com/ray-project/oss-ci-build-branch/builds/261#018388a9-df2d-4b75-a7a7-346815b92f71

def get_local_file_whitelist(cluster, option):
--
  | # On Windows the runtime directory itself is not deleted due to it being in use
  | # therefore whitelist it for the tests.
  | if sys.platform == "win32" and option != "py_modules":
  | runtime_dir = (
  | Path(cluster.list_all_nodes()[0].get_runtime_env_dir_path())
  | / "working_dir_files"
  | )
  | >           return {list(Path(runtime_dir).iterdir())[0].name}
  | E           IndexError: list index out of range

[architkulkarni]

This test is still flaky, due to a new reason:
Ex: https://buildkite.com/ray-project/oss-ci-build-branch/builds/284#01838ca6-974f-41be-94d6-d21fe159416f/7284-7827
wait_for_condition(lambda: check_local_files_gced(cluster, whitelist=whitelist)) timed out.

[rickyyx]

This is timing out on the release branch as well (on mac machine though) - should we cherry pick this? https://buildkite.com/ray-project/oss-ci-build-branch/builds/528#0183d72d-8a29-4fcb-a094-b4e7cddf24c2

@architkulkarni

[architkulkarni]

@rickyyx Can it be fixed by a restart?

I don't think it's a release blocker. If you think it would delay the release less overall if we skip the test, we can cherry pick a PR to skip it on Mac.

We should eventually figure out the root cause though, I made an issue to track the mac flakiness here #29373

[rickyyx]

I see - I think it doesn't show up every single time. I could bear with the flakiness for now.