[Core][Release test] microbenchmark_38.aws failed

[rkooo567]

What happened + What you expected to happen

multi client put calls (Plasma Store) per second 12224.22 +- 138.19
single client put gigabytes per second 19.33 +- 5.13
single client tasks and get batch per second 9.54 +- 0.71
multi client put gigabytes per second 32.88 +- 2.09
single client get object containing 10k refs per second 12.29 +- 0.02
single client wait 1k refs per second 4.09 +- 0.06
single client tasks sync per second 1320.24 +- 34.96
single client tasks async per second 10520.47 +- 403.57
multi client tasks async per second 31509.83 +- 2904.41
1:1 actor calls sync per second 2508.25 +- 43.64
1:1 actor calls async per second 8679.05 +- 167.69
1:1 actor calls concurrent per second 5051.74 +- 204.91
1:n actor calls async per second 10362.16 +- 180.06
n:n actor calls async per second 30507.68 +- 413.78
n:n actor calls with arg async per second 2588.27 +- 99.46
1:1 async-actor calls sync per second 1659.86 +- 34.58
1:1 async-actor calls async per second 3014.39 +- 152.17
Traceback (most recent call last):
  File "run_microbenchmark.py", line 16, in <module>
    results = main() or []
  File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/_private/ray_perf.py", line 258, in main
    results += timeit("1:1 async-actor calls with args async", async_actor, 1000)
  File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/_private/ray_microbenchmark_helpers.py", line 26, in timeit
    fn()
  File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/_private/ray_perf.py", line 256, in async_actor
    ray.get([a.small_value_with_arg.remote(i) for i in range(1000)])
  File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/home/ray/anaconda3/lib/python3.8/site-packages/ray/_private/worker.py", line 2534, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(RecursionError): ray::AsyncActor.small_value_with_arg() (pid=66650, ip=10.0.5.225, repr=<ray._private.ray_perf.AsyncActor object at 0x7fe544f8b6a0>)
  File "/home/ray/anaconda3/lib/python3.8/asyncio/tasks.py", line 918, in run_coroutine_threadsafe
    future = concurrent.futures.Future()
  File "/home/ray/anaconda3/lib/python3.8/concurrent/futures/_base.py", line 318, in __init__
    self._condition = threading.Condition()
  File "/home/ray/anaconda3/lib/python3.8/threading.py", line 224, in __init__
    lock = RLock()
RecursionError: maximum recursion depth exceeded
(AsyncActor pid=66650) sys:1: RuntimeWarning: coroutine 'execute_task.<locals>.deserialize_args' was never awaited
Subprocess return code: 1

https://console.anyscale-staging.com/o/anyscale-internal/jobs/prodjob_f6k46srheek1fh6v8hc24q6992

Versions / Dependencies

master

Reproduction script

n.a

Issue Severity

None

[jjyao]

Might be related: #30498

[jjyao]

Same commit passed: https://buildkite.com/ray-project/release-tests-branch/builds/1624#0187d904-56dc-436d-8cb2-1ef5876eba85

[fishbone]

Could this issue relate to #35015 ?

[jjyao]

Hmm probably not. For this one, I was not able to reproduce it and it's passing in master now.

[scv119]

#30476 also might be related.

[scv119]

https://buildkite.com/ray-project/release-tests-branch/builds/1633#0187e3ac-bfca-451e-9210-0c23e33d8ce3 another error. don't have a clue yet.

[can-anyscale]

It seems like the issue is now a lot more persistent now: https://buildkite.com/ray-project/release-tests-branch/builds/1647#01880bf9-1862-4b17-a764-d1cc1438e596.

I failed to bisect this due to its flakiness level

CC: @scv119

[scv119]

https://console.anyscale-staging.com/o/anyscale-internal/job-runs/job_3x719djkd35fpd8p45sy7b99mi

[can-anyscale]

Ok, after so many bisect, I think this is the culprit: cc3fa33

Could you help verify @vitsai , @scv119

Bisect job: https://buildkite.com/ray-project/release-tests-bisect/builds/152#01882575-5a1e-4c22-8671-24cb54da02dc

[scv119]

thanks @can-anyscale I think probably it's a side effect of the PR make this bug more frequent..

[can-anyscale]

FYI, this test is reliably passing on the release branch, and I think the only relevant pick is this https://buildkite.com/ray-project/oss-ci-build-branch/builds/3954. That PR is reverted on master though cause it breaks other thing, but want to let you know it fixed the tests somehow.

[can-anyscale]

Since this is a release blocker, can you help do a cherry pick as well @rkooo567 , thanks

[rkooo567]

There's already a PR out!

[rkooo567]

#35532