Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable more concurrency via the Makefile #5

Open
martin-ueding opened this issue Jul 3, 2018 · 0 comments
Open

Enable more concurrency via the Makefile #5

martin-ueding opened this issue Jul 3, 2018 · 0 comments

Comments

@martin-ueding
Copy link
Contributor

Most calls with pvcall are parallelized. Some other calls use serial = TRUE but then have boot::tsboot with its own parallelism. In general one can say that all the things from the paramval directory are to be run serial, with make -j 1. The rendering of the RMarkdown documents is single-threaded and could be parallelized with make -j $(nproc).

It seems that this mixture is not supported by GNU make, see this Stack Overflow question.

I currently see the following options:

  1. Use a second Makefile with the RMarkdown targets and call that with make -j $(nproc). This will improve throughput, but the dependencies have all to be satisfied before this can be done. Dependency handling inter-Makefile sounds like needless pain.

  2. Not use GNU make but instead see whether Ninja-Build has this feature. Makefile generation should still be retained because system out of our control very likely do not have Ninja-Build installed.

  3. Use a single Makefile and have processes communicate with some sort of semaphore to prevent oversubscription. That is likely going to end in deadlocks rather quickly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant