Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEAT] Enable Progress Bars for PyRunner and RayRunner #1609

Merged
merged 13 commits into from
Nov 17, 2023

Conversation

samster25
Copy link
Member

@samster25 samster25 commented Nov 15, 2023

  • Enables progress bars when in interactive mode (Notebook or IPython)
  • Implementation for Pyrunner, RayRunner[Local] and RayRunner[Remote]

image

@samster25 samster25 changed the title [FEAT] Sammy/progress bar prototype pyrunner [FEAT] progress bar prototype pyrunner Nov 15, 2023
@github-actions github-actions bot added the enhancement New feature or request label Nov 15, 2023
Copy link

codecov bot commented Nov 15, 2023

Codecov Report

Merging #1609 (9ce28b9) into main (56cd1d6) will increase coverage by 2.69%.
Report is 13 commits behind head on main.
The diff coverage is 88.17%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1609      +/-   ##
==========================================
+ Coverage   84.59%   87.29%   +2.69%     
==========================================
  Files          54       52       -2     
  Lines        5201     5115      -86     
==========================================
+ Hits         4400     4465      +65     
+ Misses        801      650     -151     
Files Coverage Δ
daft/execution/execution_step.py 91.66% <100.00%> (-0.43%) ⬇️
daft/execution/physical_plan.py 93.37% <100.00%> (+0.39%) ⬆️
daft/runners/pyrunner.py 96.96% <100.00%> (+0.67%) ⬆️
daft/expressions/expressions.py 91.09% <50.00%> (-0.24%) ⬇️
daft/runners/ray_runner.py 89.75% <80.00%> (-0.05%) ⬇️
daft/runners/progress_bar.py 80.48% <80.48%> (ø)

... and 4 files with indirect coverage changes

@samster25 samster25 force-pushed the sammy/progress-bar-prototype-pyrunner branch from af7ff8c to b52f6ec Compare November 15, 2023 16:48
@samster25 samster25 changed the title [FEAT] progress bar prototype pyrunner [FEAT] progress bar prototype Nov 15, 2023
@samster25 samster25 changed the title [FEAT] progress bar prototype [FEAT] Enable Progress Bars for PyRunner and RayRunner Nov 15, 2023
daft/runners/tqdm.py Outdated Show resolved Hide resolved
daft/runners/pyrunner.py Outdated Show resolved Hide resolved
daft/runners/pyrunner.py Outdated Show resolved Hide resolved
daft/runners/ray_runner.py Outdated Show resolved Hide resolved
self.use_ray_tqdm = use_ray_tqdm
self.show_tasks_bar = show_tasks_bar
self.tqdm_mod = tqdm
self.pbars: dict[int, tqdm] = dict()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initialize as self.pbars = {-1: ...}? We could also document here that -1 represents the progress for "All tasks".

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we actually dont want to initialize the the progress bar until we get a task. Otherwise the total is 0 and it's in a weird state.

@samster25 samster25 merged commit 24c758e into main Nov 17, 2023
37 checks passed
@samster25 samster25 deleted the sammy/progress-bar-prototype-pyrunner branch November 17, 2023 03:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants