Skip to content

Commit

Permalink
[Data] Simplify and consolidate progress bar outputs (ray-project#47692)
Browse files Browse the repository at this point in the history
## Why are these changes needed?

Currently, the progress bar is pretty verbose because it is very
information dense. This PR:
- Reorganizes progress output to group by relevant concepts and
clarifies labels
- Standardizes global and operator-level progress bar outputs
- Removes the use of all emojis (poor rendering on some platforms /
external logging systems)

Progress bar before this PR:
<img width="1403" alt="Screenshot at Sep 16 13-00-17"
src="https://github.com/user-attachments/assets/4f459b77-06ba-4395-b883-e4c9ac8ca2ef">

Progress bar after this PR:
<img width="1502" alt="Screenshot at Sep 23 13-48-32"
src="https://github.com/user-attachments/assets/0c0f8c94-9439-4fd4-ae1a-2857b3a87b59">

Will follow up with a docs PR once we merge this change, so that I don't
need to continuously modify the docs.

In the future, we should restructure the way progress bars are
grouped/tracked, so that we can tabulate the op-level progress bar
outputs.

## Related issue number

## Checks

- [x] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [x] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [x] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [ ] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Scott Lee <[email protected]>
Signed-off-by: ujjawal-khare <[email protected]>
  • Loading branch information
scottjlee authored and ujjawal-khare committed Oct 15, 2024
1 parent aaa3d8d commit c7ff9c8
Showing 1 changed file with 3 additions and 8 deletions.
11 changes: 3 additions & 8 deletions python/ray/data/_internal/execution/streaming_executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -363,14 +363,9 @@ def _report_current_usage(self) -> None:
pending_usage = self._resource_manager.get_global_pending_usage()
limits = self._resource_manager.get_global_limits()
resources_status = (
"Active & requested resources: "
f"{running_usage.cpu:.4g} of {limits.cpu:.4g} available CPU, "
f"{running_usage.gpu:.4g} of {limits.gpu:.4g} available GPU, "
f"{running_usage.object_store_memory_str()} of "
f"{limits.object_store_memory_str()} available object_store_memory "
"(pending: "
f"{pending_usage.cpu:.4g} CPU, "
f"{pending_usage.gpu:.4g} GPU)"
# TODO(scottjlee): Add dataset name/ID to progress bar output.
"Running Dataset. Active & requested resources: "
f"{running_usage.cpu:.4g}/{limits.cpu:.4g} CPU, "
)
if running_usage.gpu > 0:
resources_status += f"{running_usage.gpu:.4g}/{limits.gpu:.4g} GPU, "
Expand Down

0 comments on commit c7ff9c8

Please sign in to comment.