Testing v0.5 on cluster system

[abhi18av]

Hi @TimHHH @LennertVerboven @adippenaar

Here's a quick status update

1. I have completed 90% of the base modules which encapsulates the tools.
2. For the time being, I have kept the core structure and variable names of XBS_main.py codebase to keep some continuity. Though I'd like to change certain things after the first test on the cluster.
3. I'd now like to focus on the some infrastructure based testing on real data to understand the right configuration for the cluster and the behavior of the pipeline

Would it be able for you to please confirm whether the main executor on the cluster is of "PBS/Torque" family or "PBS Pro" family of clusters?

Moreover, it'd be nice to know

• Whether there are other relevant settings such as queue and clusterOptions
• Whether conda environment is enabled by default or not, so that we can create conda envs on the fly when needed
• What is the location where intermediate data should be stored
• What is the location where final data should be stored
• What is the location of various databases on the cluster system

If you could please share this information then we should be able to start testing some workflows (MAP_WF / QUANTTB / CALL_WF) pipeline by early next week.

[TimHHH]

Heya.
I am trying to figure out the PBS family as we speak, haven't found a conclusive command yet.
The relevant settings can be found in the top of the bash scripts;

#PBS -l nodes=1:ppn=8,mem=64g
#PBS -A GHI
#PBS -q scattergather
#PBS -V 

The -A and -q options are essential as is, the other options are suitable for adjusting
temporary files e/o files go here:

#PBS -e /home/magmauser/XBS_pbs_logs/error
#PBS -o /home/magmauser/XBS_pbs_logs/info

Conda environments have to be enabled as such:

source $CONDA_DIR/etc/profile.d/conda.sh
conda activate $TBPROFILER_VENV

The locations of all databases/softwares can be found in the main python script, the main locations are /home/shared_data_ghi/CWGSMTB/References/NC-000962-3-H37Rv and /home/shared_data_ghi/CWGSMTB/Software respectively.
What do you exactly mean by intermediate data? we tend to keep all files in the same output dir, e.g. there is a bam sub-dir for all intermediate and the final bam files.
Let's have a call about the directory structure we use.

[abhi18av]

This is helpful, I'll incorporate this to the configuration file I'm working on https://github.com/abhi18av/xbs-nf/blob/master/conf/pbs.conf and here's the relevant code in Nextflow to generate these directives dynamically.

I'm adding this reference here for fine-tuning we might need to do later on https://www.nextflow.io/docs/edge/config.html#scope-executor

Few more references in case some of the directives are not directly addresses by Nextflow

• https://hpc.nih.gov/apps/nextflow.html
• https://hpc.research.uts.edu.au/getting_started/running/
• https://www.nextflow.io/blog/2021/5_tips_for_hpc_users.html
• https://www.nextflow.io/blog/2021/5-more-tips-for-nextflow-user-on-hpc.html

[abhi18av]

What do you exactly mean by intermediate data?

By intermediate data, I meant files which are generated during data processing and are not used later on.

Nextflow has a concept of work directory (which should be on a shared directory in a cluster) and then a publishDir which is the destination of files you'd actually like to save.

When we connect tomorrow, we can talk more about these and iron out these details.