clashing python libraries in singularity containers and ccbrpipeliner module

[kelly-sovacool]

Activating singularity image /vf/users/guibletwm/charlietest/charlie/.snakemake/singularity/85508a5ad215e06ac7e60750d9a291aa.simg
+ cd /vf/users/guibletwm/charlietest/charlie/ref
+ samtools faidx /vf/users/guibletwm/charlietest/charlie/ref/ref.fa
+ cut -f1-2 /vf/users/guibletwm/charlietest/charlie/ref/ref.fa.fai
+ python /gpfs/gsfs10/users/CCBR_Pipeliner/Pipelines/CHARLIE/.v0.11.0/workflow/scripts/fix_gtfs.py --ingtf /vf/users/guibletwm/charlietest/charlie/ref/ref.gtf --outgtf /vf/users/guibletwm/charlietest/charlie/ref/ref.fixed.gtf
Traceback (most recent call last):
  File "/gpfs/gsfs10/users/CCBR_Pipeliner/Pipelines/CHARLIE/.v0.11.0/workflow/scripts/fix_gtfs.py", line 2, in <module>
    import pandas
  File "/data/CCBR_Pipeliner/Tools/ccbr_tools/v0.1/pandas/__init__.py", line 1
    from __future__ import annotations
    ^
SyntaxError: future feature annotations is not defined

[kelly-sovacool]

able to reproduce this with:

singularity shell -B /lscratch,/data/CCBR_Pipeliner,/data/sovacoolkl,/gpfs/gsfs10/users/CCBR_Pipeliner,/gpfs/gsfs12/users/sovacoolkl docker://nci
ccbr/charlie_star_ucsc_cufflinks:v0.4.0 zsh
python
import pandas
pandas

one workaround:

python -E
import pandas
pandas

we may want to use singularity -C or singularity -E, but will that mess up $SLURM_JOB_ID which is needed for temp dirs in some rules?

[kelly-sovacool]

when using singularity -C, $SLURM_JOB_ID is not available in the containers. However, snakemake calls singularity exec, which does not have the --env argument for setting the $SLURM_JOB_ID in the container. Also, it would likely use the value of $SLURM_JOB_ID from the master job instead of from within the individual rule jobs.

trying python -E inside the rule definition as a workaround.

[kelly-sovacool]

possible solutions:

• 🚫 singularity -C - can't use it, $SLURM_JOB_ID won't be available
• ⚠️ python -E - works but only when we explicitly call python, may not work when using tools that have python dependencies
• ✅ set $PYTHONPATH inside container - WORKS. tested with docker://nciccbr/charlie_star_ ucsc_cufflinks:v0.4.1. 1fc7dc0
• unset $PYTHONPATH at beginning of snakemake rule - almost definitely would work, but would be cumbersome and clutter up snakemake rules.