Cached images duplicated by Nextflow and Apptainer/Singularity when ociAutoPull set to true

[tmoughamer]

Bug report

Cached images duplicated by Nextflow and Apptainer/Singularity when ociAutoPull set to true

Expected behavior and actual behavior

Expection is that on copy of each image file is only pulled and stored by Apptainer

Steps to reproduce the problem

Create a config file with:

profiles {
    singularity {
        singularity.enabled    = true
        singularity.autoMounts = true
        singularity.ociAutoPull = true
    }
    apptainer {
        apptainer.enabled      = true
        apptainer.autoMounts   = true
        apptainer.ociAutoPull  = true
    }
}

Run workflow specifying the above config file:
/nextflow run nf-core/rnaseq -c ./nxf_redirect.config -profile test,singularity --outdir ./rnaseq_3

Program output

During run we get:

NFCORE_RNASEQ:RNASEQ:CUSTOM_DUMPSOFTWAREVERSIONS                                             -
Plus 1 more processes waiting for tasks…
Pulling Singularity image https://depot.galaxyproject.org/singularity/trim-galore:0.6.7--hdfd78af_0 [cache /home/tmoughamer/work/singularity/depot.galaxyproject.org-singularity-trim-galore-0.6.7--hdfd78af_0.img]
Pulling Singularity image https://depot.galaxyproject.org/singularity/fastqc:0.12.1--hdfd78af_0 [cache /home/tmoughamer/work/singularity/depot.galaxyproject.org-singularity-fastqc-0.12.1--hdfd78af_0.img]
Pulling Singularity image https://depot.galaxyproject.org/singularity/ubuntu:20.04 [cache /home/tmoughamer/work/singularity/depot.galaxyproject.org-singularity-ubuntu-20.04.img]
Pulling Singularity image https://depot.galaxyproject.org/singularity/python:3.9--1 [cache /home/tmoughamer/work/singularity/depot.galaxyproject.org-singul
…
WARN: Singularity cache directory has not been defined -- Remote image will be stored in the path: /home/tmoughamer/work/singularity -- Use the environment variable NXF_SINGULARITY_CACHEDIR to specify a different location
…
-[nf-core/rnaseq] Pipeline completed successfully -
Completed at: 23-Aug-2024 12:29:38
Duration    : 15m 10s
CPU hours   : 0.7
Succeeded   : 194

The image files end duplicated and up stored in two locations:

(base) [tmoughamer@W2RSCEWEBPRDN01 ~]$ /bin/ls /scratch/tmoughamer/.apptainer/cache/net/* | cat | wc -l
24
(base) [tmoughamer@W2RSCEWEBPRDN01 ~]$ /bin/ls ./work/singularity/* | cat | wc -l
24
(base) [tmoughamer@W2RSCEWEBPRDN01 ~]$ du -hs /scratch/tmoughamer/.apptainer/cache/net
4.2G /scratch/tmoughamer/.apptainer/cache/net
(base) [tmoughamer@W2RSCEWEBPRDN01 ~]$ du -hs ./work/singularity
4.2G ./work/singularity

Environment

• Nextflow version: 24.04.4
• Java version: "11.0.20.1" 2023-08-24
• Operating system: Red Hat Enterprise Linux release 8.10 (Ootpa)
• Bash version: Red Hat Enterprise Linux release 8.10 (Ootpa)
• container engine: apptainer version 1.3.2-1.el8

Additional context

I've check the file hashes and they are the same. I've done several runs at this including setting the $NXF_SINGULARITY_CACHDIR but in this case I did not. The result is the same. I've also set ociAutoPull to true for both apptainer and singularity profiles in the ~/.nextflow/assets/nf-core/rnaseq/nextflow.config

[pditommaso]

This entirely depends on Singularity and Apptainer engines, which ultimately are two difference piece of softwares.

[tmoughamer]

Granted. Currently I'm not able to run under the apptainer profile likely related to nf-core/tools#2357. I'll try something simpler.