[Tune/Train] Make MLflowLoggerUtil copyable

[Yard1]

Why are these changes needed?

Makes sure mlflow module is not saved as an attribute in MLflowLoggerUtil, which was causing an exception when running deepcopy on the callback.

Related issue number

Closes #23330

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[amogkam]

Hey @Yard1 does this change work for multi-node? I believe the reason we had it stored as an attribute is because on the driver side, we modify mlflow configurations which are saved as module attributes. If on worker nodes, we don't reuse the same mlflow module as on the driver, then would we lose those configurations?

Btw, it seems like this might just be a problem with dictionary deep copying, not with the callback itself. We have a test here for running MLflowLoggerCallback with ray Client, which means it must be serializable with cloudpickle right? https://github.com/ray-project/ray/blob/master/python/ray/tune/tests/test_client.py#L95.

[amogkam]

I think the fix we want to make is just not to include run_config in self._params in the Trainer.

[Yard1]

@amogkam ok I changed the approach, PTAL

[krfricke]

Can we add a test for this?

[Yard1]

@krfricke for every callback or just this?