[BUG] MNMG RandomForest 'print_summary' and 'print_detailed' produce incorrect behavior, in a multi-node environment.

[drobison00]

Describe the bug
After fitting a RandomForestClassifer (cuml.dask.ensemble), its class methods appear to output their information on the worker, rather than the client. This is likely not appropriate.

Steps/Code to reproduce bug

from dask_kubernetes import KubeCluster
from dask.distributed import Client
from cuml.dask.ensemble import RandomForestClassifier

# Worker and scheduler pods are based on 'rapidsai/rapidsai:cuda11.0-runtime-ubuntu18.04-py3.8'
cluster = KubeCluster(pod_template=worker_pod,
                      scheduler_pod_template=sched_pod)

client = Client(cluster)

import dask_cudf
from cuml.dask.cluster import KMeans

higgs_gcp_path = "gs://dvn-cloudml-examples/HIGGS.csv"
columns = ['label'] + [f'col-{i}' for i in range(1, 29)]
workers = client.has_what().keys()

### This block works fine, and produces a cudf Dataframe that works with various other algorithms, including XGboost.
higgs_df = dask_cudf.read_csv(higgs_gcp_path, npartitions=len(workers), headers=None,
                                 names=columns,
                                 storage_options={'token': "/etc/secrets/keyfile.json"})
higgs_df = client.persist(collections=higgs_df, workers=workers)
wait(higgs_df)

y_higgs = higgs_df['label']
X_higgs = higgs_df[higgs_df.columns.difference(['label'])]

rfc = RandomForestClassifier(workers=workers, n_estimators=100)
rfc.fit(X_higgs.astype(np.float32), y_higgs.astype(np.int32))

### PROBLEM OCCURS HERE ###
rfc.print_detailed() # <---- This prints on the worker nodes, and outputs nothing on the client.
rfc.print_summary() # <---- This prints on the worker node, and outputs nothing on the client.

Expected behavior
Each API method should print its results on the client.

Environment details (please complete the following information):

• Environment location: GKE
• Linux Distro/Architecture: Ubuntu 18.04
• GPU Model/Driver: T4 450.xx
• CUDA: 11.0
• Method of cuDF & cuML install: rapids containers rapidsai/rapidsai:cuda11.0-runtime-ubuntu18.04-py3.8

Additional context
RAPIDS containers are augmented with xgboost 1.3.0rc1 and gcsfs

[hcho3]

@drobison00 print_detailed and print_summary should be revised to behave like dump_as_json, which returns a string instead of printing to stdout. Can you try dump_as_json and see if the result is available on the client?

[hcho3]

Oops, never mind. The dump_as_json() isn't yet exposed to the Dask interface. My point still stands: The two functions should return a string instead of directly writing to stdout.

[nmate1988]

Then is there a way to export a MNMG random forest model after the forest has been trained?

[hcho3]

@nmate1988 I submitted #3340 to add dump_as_json() to the MNMG RF model.

[nmate1988]

Thank you very much, I will take a look

[hcho3]

@nmate1988 The latest cuML source code now has get_summary_text() and get_detailed_text() methods, which will export the tree model as human-friendly text. If you want JSON format instead, use dump_as_json() instead. (This method is about to be renamed to get_json() in #3340, which is not yet merged.)