[tune] Disable pytorch-lightning multiprocessing per default

[krfricke]

Signed-off-by: Kai Fricke [email protected]

Why are these changes needed?

Pytorch lightning uses multiprocessing pools per default (e.g. for device lookup), which can lead to hangs (see #28328). This PR sets an environment variable to disable this until #28328 is addressed.

Related issue number

Closes #28197

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[amogkam]

Thanks @krfricke! Seems this is a problem only with PTL 1.7 (Lightning-AI/pytorch-lightning#14292). Should we update the PTL version used in CI to test that these changes work?

[krfricke]

Good idea, I'll add this to this PR

[krfricke]

@amogkam unfortunately we land in a dependency loop here - we can't upgrade to PTL 1.7.X as ray-lightning 0.3.0 is not compatible, but upgrading ray lightning for compatibility requires changes to the library - and the CI won't pass as the trials can hang, as this fix is not merged.
I've run it manually with the fix and it works for me. Adding another pipeline with the specific set of requirements seems like too much overhead to me - let's deploy this fix and focus on ptl 1.7 compatibility for ray lightning - ok?