[Data] Ray Data progress bars left partially completed after execution completes on Jupyter notebook

[scottjlee]

What happened + What you expected to happen

When executing a Ray Dataset on Jupyter notebook, the progress bar is left partially complete after the dataset finishes executing:

When we disable ray_tqdm, the progress bars are filled to completion as expected:

So it looks like there is likely a bug in ray_tqdm where we are not updating the progress bar before closing it.

We should also update ProgressBar to use tqdm.auto.tqdm() to initialize the bar when not using ray_tqdm.

https://github.com/ray-project/ray/blob/master/python/ray/data/_internal/progress_bar.py#L68

Versions / Dependencies

ray master

Reproduction script

Run the following in a Jupyter notebook:

import ray
import time

def fn(batch):
    time.sleep(10)
    return batch

ds = ray.data.range(10).repartition(10)
ds.map_batches(fn).take_all()

Issue Severity

None

[jonfritz]

This fix would be awesome! It would be nice to see the progress bar complete - it's much easier than needing to check the completion of a cell in the left corner.