[Java] make java worker log file prefix (default to java-worker) configurable

[jiafuzha]

Why are these changes needed?

For java worker, it's log file always being prefixed with "java-worker". And in python log_monitor.py, it hardcodes "java-worker*.log" to be polled for new log msg periodically. Some configs, like log_to_driver and RAY_BACKEND_LOG_LEVEL, don't prevent the log monitor from polling and publishing logs to gcs. To save some CPU cycle and network bandwidth, especially if there is large amount of logs produced from JVM, we can have an option. like a JVM system property, to set log file prefix for java worker instead of hard coded to "java-worker".

Related issue number

Closes #33787

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[jiafuzha]

@jovany-wang @kira-lin please help review.

[simon-mo]

This is going to break log monitor, please make the corresponding change there as well:

ray/python/ray/_private/log_monitor.py

Lines 200 to 202 in af759eb

	log_file_paths = glob.glob(f"{self.logs_dir}/worker*[.out|.err]") + glob.glob(
	f"{self.logs_dir}/java-worker*.log"
	)

[jiafuzha]

This is going to break log monitor, please make the corresponding change there as well:

ray/python/ray/_private/log_monitor.py

Lines 200 to 202 in af759eb

	log_file_paths = glob.glob(f"{self.logs_dir}/worker*[.out|.err]") + glob.glob(
	f"{self.logs_dir}/java-worker*.log"
	)

As tested, log monitor worked well with either property "ray.logging.file-prefix" set to "raydp-java-worker" or unset and default to "java-worker".

Could you please be more specific why log_monitor is breaking? From code, 'glob.glob(f"{self.logs_dir}/java-worker*.log"' returns empty list if there is not java-worker* files.

thanks

[jovany-wang]

@zjf2012 So your purpose is to prevent polling java worker log messages to GCS right?

[jiafuzha]

@zjf2012 So your purpose is to prevent polling java worker log messages to GCS right?

exactly. And it's configurable.

[jovany-wang]

Add comment on what this field is.

[jovany-wang]

LGTM.

[jovany-wang]

Hi @simon-mo , I don't think this breaks anything. Maybe also @rkooo567 have a double check.

[jovany-wang]

DO NOT MERGE until 2.4 branch cut.

[jovany-wang]

@zjf2012 do you need this feature in Ray2.4?

[simon-mo]

Ah. This is a hacky way to not to stream log through log monitor it seems. Alternatively an env var in log monitor yo flag this will do.

[jiafuzha]

@zjf2012 do you need this feature in Ray2.4?

yes, we need it in ray 2.4.

By the way, why do we need to wait branch 2.4 before merging it?

[jovany-wang]

@simon-mo If we don't have the need that preventing polling only Java log, it makes more sense to add a flag to logmonitor.

[jovany-wang]

@zjf2012 Because we changed the releasing process. We shouldn't merge unnecessary changes when in the process of releasing duration.

yes, we need it in ray 2.4.

@zhe-thoughts CC

[zhe-thoughts]

I'd like @simon-mo to confirm about his open question first. Thanks for working on this @jovany-wang

[simon-mo]

With the comments added, the change LGTM

[jovany-wang]

Thanks for the contribution! This will be in the next release.