Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Better allocation of workers for train and validation sets #260

Open
ndiamant opened this issue May 14, 2020 · 1 comment · Fixed by #327
Open

Better allocation of workers for train and validation sets #260

ndiamant opened this issue May 14, 2020 · 1 comment · Fixed by #327
Labels
enhancement New feature or request

Comments

@ndiamant
Copy link
Contributor

What
Allocate workers based on number of train vs. validation steps, so that the validation workers don't pull way ahead of the train workers.

Why
Better use of cpus for faster training.

How
num train workers = training steps / (training steps + validation steps) or something

Acceptance Criteria
Workers allocated based on number of train vs. validation steps.
Speed comparison would be good, ideally through a test!
@ndiamant ndiamant added the enhancement New feature or request label May 14, 2020
@StevenSong StevenSong mentioned this issue Jun 15, 2020
Closed
StevenSong added a commit that referenced this issue Jun 15, 2020
@StevenSong
#260 #324 #326
9d3a3b6
@StevenSong StevenSong mentioned this issue Jun 15, 2020
Merged
StevenSong added a commit that referenced this issue Jun 18, 2020
@StevenSong
fixes in tensor generator (#327)
9c3e6d7 
* #260 #324 #326

* get stats q

* variable names
@StevenSong
Copy link
Collaborator

hm to be honest I think my fix in #327 made it worse, will reopen

@StevenSong StevenSong reopened this Jun 25, 2020
@mklarqvist mklarqvist mentioned this issue Aug 14, 2020
Closed
lucidtronix pushed a commit that referenced this issue Jan 13, 2023
@StevenSong
fixes in tensor generator (#327)
f747f7c 
* #260 #324 #326

* get stats q

* variable names
lucidtronix pushed a commit that referenced this issue Jan 13, 2023
@StevenSong
fixes in tensor generator (#327)
f6554fe 
* #260 #324 #326

* get stats q

* variable names
lucidtronix pushed a commit that referenced this issue Jan 13, 2023
@StevenSong
fixes in tensor generator (#327)
b87dacb 
* #260 #324 #326

* get stats q

* variable names
lucidtronix pushed a commit that referenced this issue Jan 13, 2023
@StevenSong
fixes in tensor generator (#327)
140f765 
* #260 #324 #326

* get stats q

* variable names
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fixes in tensor generator broadinstitute/ml4h
2 participants
@ndiamant @StevenSong