[Core] Actors not cleaning up resources correct because force_kill=true.

[cadedaniel]

Discovered in the investigation of #31451 and #33976.

TL;DR things like actor destructors or atexit handlers are not guaranteed to be executed when we destroy actors. This is because we use force_kill=true in gcs_actor_manager.

Ideally, we should send SIGTERM to worker processes so that they clean up any important state. After some time period, if the process has not already died already, we will then send a SIGKILL.

More information in Sang's comment here #33976 (comment)

[clarng]

but if os oom killer kick in these handlers won't get called right? we cannot guarantee os oom killer will never kick in

[cadedaniel]

Does our OOM killer immediately use SIGKILL? Or is there a SIGTERM first?

[clarng]

sigkill
if we wait then it might trigger os oom (which will force kill anyway)

[cadedaniel]

I see. For the process leak case, the raylet will handle this case. For other resource leaks, needs more thought. One low hanging fruit is to improve OOM killer to send SIGTERM then SIGKILL. I believe this should be doable with a bit of design..

[clarng]

trying to clean resources up when the machine is under pressure may not be the best idea

what are we trying to clean up? can that be done async?

[rkooo567]

@clarng this is only for the general shutdown cases. Right now, we always force kill actors even when graceful shutdown is possible (which is bad).

I think OOM killer sending SIGKILL is reasonable, and this just means we cannot properly trigger shutdown handler (that's the point of SIKILL actually) when they are OOM kiled

[rkooo567]

Your case will be handled by #34125

[rkooo567]

#32952 -> this issue has a minimal repro script we should test

[mlguruz]

I believe this is also related:

I kept running into this when using xgboost_ray:

Exception ignored in: <function PlacementGroupResourceManager.__del__ at 0x7f6722c8a550>
File "/home/ubuntu/zstar/lib/python3.8/site-packages/ray/air/execution/resources/placement_group.py", line 211, in __del__
File "/home/ubuntu/zstar/lib/python3.8/site-packages/ray/util/tracing/tracing_helper.py", line 466, in _resume_span
File "/home/ubuntu/zstar/lib/python3.8/site-packages/ray/air/execution/resources/placement_group.py", line 193, in clear
File "/home/ubuntu/zstar/lib/python3.8/site-packages/ray/_private/client_mode_hook.py", line 98, in wrapper
ImportError: sys.meta_path is None, Python is likely shutting down
Exception ignored in: <function PlacementGroupResourceManager.__del__ at 0x7efecd343550>

Is there any way to disable this? Coz its really polluting the logs...

[rkooo567]

Hmm, I believe it is actually a different issue. I think when you execute __del__, any module call is not guaranteed to succeed because modules can be GC'ed before the class is destructed.

cc @krfricke it is something PlacementGroupResourceManager should fix in its layer.