Test Workflow for slurmworkflow

This repo creates a workflow testing slurmworkflow's capabilities on an HPC.

How To Use

• Create a new GitHub repository by clicking the green Use this template button above.
• Clone this new repo on your computer AND on the HPC (git is required on the HPC)
• On your local computer:
  • Setup renv
    • renv::init(bare = TRUE) - then restart R
    • Install packages: renv::install(c("future.apply", "EpiModel/slurmworkflow"))
    • renv::snapshot()
  • Edit the "0-settings.R" script according to your setup
  • Commit your changes and push to GitHub (including the renv.lock file)
  • Run the script "1-make_workflow.R"

Send the generated workflow directory "workflows/test_slurmworkflow" to the HPC with:

for Windows users:

# bash - local (for WINDOWS users)
set DISPLAY=
scp -r workflows\test_slurmworkflow <user>@<hpc address>:<path_to_project_on_HPC>/workflows/

for Linux / MacOS users

# bash - local (for Linux / MacOS users)
scp -r workflows/test_slurmworkflow <user>@<hpc address>:<path_to_project_on_HPC>/workflows/

• On the HPC
  • cd into your project directory
  • execute the workflow with:

# bash - HPC
./workflows/test_slurmworkflow/start_workflow.sh

The workflow will not work if you source the file (with source <script> or . <script>).

Note:

If you are a Windows user, the script may not be executable. You can solve it with the following command:

# bash - hpc
chmod +x workflows/test_slurmworkflow/start_workflow.sh

Then run:

# bash - HPC
./workflows/test_slurmworkflow/start_workflow.sh

Expected results

If all goes well, you should receive an e-mail a few minutes later with object looking like:

[External] Slurm Job_id=14940582 Name=test_slurmworkflow_step5 Ended, Run time 00:00:02, COMPLETED, ExitCode 0

If the ExitCode is anything other than 0, it means it failed.

On the HPC you can run ls workflows/test_slurmworkflow/log/. The folder should contain:

• 1 "test_slurmworkflow_controler.out" file
• 1 "test_slurmworkflow_step1_xxx_xxx.out" file
• 1 "test_slurmworkflow_step2_xxx_xxx.out" file
• 2 "test_slurmworkflow_step3_xxx_xxx.out" file
• 5 "test_slurmworkflow_step4_xxx_xxx.out" file
• 1 "test_slurmworkflow_step5_xxx_xxx.out" file

If so: SUCCESS! slurmworkflow is working correctly