[core][state] Microbench regression with task backend investigation

[rickyyx]

What happened + What you expected to happen

Investigate further if we could resolve the regression or identify the root cause (which function call/callsite causes this)

Versions / Dependencies

master

Reproduction script

NA

Issue Severity

None

[rickyyx]

With the current microbenchmark setup, we will start as many workers as possible, and thus multi threads on a single CoreWorker process (driver/worker) will actually contend for physical CPUs.

With less num_cpus made available to the ray cluster, the regression is much lower <5% due to the added work at submitting work (the owner task is usually CPU bounded)

Also, given the speciality of the microbenchmark (large number of no-op tasks), the owner is actually the bottleneck, which is different from many of the more realistic workloads, e.g. many_tasks and stress_test_many_tasks, many_actors and stress_test_many_actors

Could we close this since the root cause is identified (thread contention among workers) and the impact to more realistic workload is low?

cc @scv119 @rkooo567

[rkooo567]

This sgtm. What do you think @scv119 ?

[rkooo567]

Actually I am curious if we restrict the # of task event we can report per second, it will get better. Maybe we can experiment with 1000 tasks/s max batch after we merge the batch PR?

[rickyyx]

Not a release blocker - mainly needs some experimentation and further validation.

[rickyyx]

This is done as an effort of the following PRs:

• [core][state] Task backend - Profile events capping  #33321
• [core][state] Task Backend - reduce lock contention on debug stats / metric recording on counters.  #32355
• [core][state] Task backend improve performance #32251