Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallelize suggest operations during nn_ensemble training #429

Closed
osma opened this issue Jul 21, 2020 · 1 comment · Fixed by #568
Closed

Parallelize suggest operations during nn_ensemble training #429

osma opened this issue Jul 21, 2020 · 1 comment · Fixed by #568
Labels
enhancement
Milestone
0.57

Comments

@osma
Copy link
Member

osma commented Jul 21, 2020

Both the eval and hyperopt commands can now perform suggest operations in parallel on multiple CPUs. When training the nn_ensemble, running the documents through the source projects is the main bottleneck. We could speed it up by implementing parallel suggest operations there as well. There should probably be a new jobs parameter for the train CLI command specifying the number of parallel jobs, just like for eval and hyperopt.
@osma osma added this to the Short term milestone Jul 21, 2020
@osma
Copy link
Member Author

osma commented Feb 4, 2022

There should probably be a new jobs parameter for the train CLI command specifying the number of parallel jobs, just like for eval and hyperopt.

This part was already implemented in PR #512.

osma added a commit that referenced this issue Feb 4, 2022
@osma
Perform suggest operations in parallel using multiprocessing in nn_en…
acd3111 
…semble. Fixes #429
@osma osma mentioned this issue Feb 4, 2022
Merged
@osma osma closed this as completed in #568 Feb 11, 2022
@osma osma modified the milestones: Short term, 0.57 Feb 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
0.57
Development

Successfully merging a pull request may close this issue.

Perform suggest operations in parallel using multiprocessing in nn_ensemble NatLibFi/Annif
1 participant
@osma