improve slurm complete/error email to describe where the pipeline output is

[kelly-sovacool]

can we customize the email body so users know which pipeline run failed?

also would be cool to run grep -v completed on the jobby log and include that in the email

[kopardev]

something like this

#SBATCH --mail-type=END,FAIL      # Notifications for job completion and failure
#SBATCH [email protected]

# Custom job body message function
send_custom_email() {
    echo -e "Job Name: $SLURM_JOB_NAME\nJob ID: $SLURM_JOB_ID\nJob Status: $1\nNode List: $SLURM_JOB_NODELIST\nStart Time: $(date)" \
    | mail -s "SLURM Job ${SLURM_JOB_ID} - ${1}" [email protected]
}

# Capture the job exit status
EXIT_STATUS=$?

# Send a custom email based on job success or failure
if [ $EXIT_STATUS -eq 0 ]; then
    send_custom_email "COMPLETED"
else
    send_custom_email "FAILED"
fi

[kelly-sovacool]

solution from stack overflow: https://stackoverflow.com/a/73293621/5787827

[kelly-sovacool]

contact biowulf admins in case they have suggestions