[spark] Automatically shut down ray on spark cluster if user does not execute commands on databricks notebook for a long time

[WeichenXu123]

Signed-off-by: Weichen Xu [email protected]

Why are these changes needed?

Automatically shut down ray on spark cluster if user does not execute commands on databricks notebook for a long time.

Databricks Runtime provides an API:
dbutils.entry_point.getIdleTimeMillisSinceLastNotebookExecution() that returns elapsed milliseconds since last databricks notebook code execution.
This PR code calls this interface to monitor notebook activity and shut down Ray cluster on timeout.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[jjyao]

What about this case:

the notebook cell is

ray.init()
result = long_running_task.remote()

this cell will finish running immediately and the next cell is executed >30 minutes later

ray.get(result)

will the ray cluster gets terminated between these two cells?

[WeichenXu123]

What about this case:

the notebook cell is

ray.init()
result = long_running_task.remote()

this cell will finish running immediately and the next cell is executed >30 minutes later

ray.get(result)

will the ray cluster gets terminated between these two cells?

In this case, the notebook status will be running (blocking on ray.get(result) command), so the getIdleTimeMillisSinceLastNotebookExecution will always return 0 duration remote task execution.

[jjyao]

There are related test failures:

python/ray/tests/spark/test_databricks_hook.py:3:1: F401 'mock' imported but unused
 ```

[WeichenXu123]

There are related test failures:

python/ray/tests/spark/test_databricks_hook.py:3:1: F401 'mock' imported but unused
 ```

Addressed.

[harupy]

@WeichenXu123 Can you add test cases for the following cases?

• DATABRICKS_RAY_ON_SPARK_AUTOSHUTDOWN_MINUTES is not a number
• DATABRICKS_RAY_ON_SPARK_AUTOSHUTDOWN_MINUTES is a negative number

[WeichenXu123]

Filed a follow-up PR #32162