Download pipelines for Tower

[MatthiasZepper]

This PR proposes the addition of a --tower flag to nf-core download to simplify the addition of downloaded pipelines to Seqeralabs Tower SaaS application and addresses #1899.

Instead of downloading a compressed archive of the pipelines source code, a clone of the pipeline's GitHub repository is created in the local cache, modified and subsequently bare-cloned to the output directory to be used with Tower. As before, the tool can detect the required Singularity images and download those as well. Since configuration profiles are likely managed separately with the help of Tower, downloading the configs is omitted.

Enabling those changes required some alterations to the download command:

• The new functionality also uses a cached GitHub repository, as first introduced by @ErikDanielsson for the modules. Thus, most functionality was moved to a generic superclass for both repo types to inherit from.
• Both downloads (classic or Tower) now support downloading multiple revisions with the same invocation (variables and functions were changed from str to dict / list).
• The find_container_images() function has been improved to support the detection of container images declared as variables. Download e.g. the RNA-seq pipeline 3.7 for an exemplary case that was not handled correctly by the current version.
• Refactored the CLI for --singularity-cache in nf-core download from a flag to an argument. The prior options were renamed to amend (container images are only saved in the $NXF_SINGULARITY_CACHEDIR) and copy (a copy of the image is saved with the download). remote was newly introduced and allows to provide a table of contents of a remote cache via an additional argument --singularity-cache-index (#2247).

PR checklist

• This comment contains a description of changes (with reason)
• CHANGELOG.md is updated
• If you've fixed a bug or added code that should be tested, add tests!
• Documentation in docs is updated

[MatthiasZepper]

Will still add tests for the new functionality, but you may already have a go on reviewing the code itself.

[codecov]

Codecov Report

Merging #2247 (259afa6) into dev (ef62904) will increase coverage by 0.03%.
The diff coverage is 67.20%.

@@            Coverage Diff             @@
##              dev    #2247      +/-   ##
==========================================
+ Coverage   72.93%   72.96%   +0.03%     
==========================================
  Files          77       78       +1     
  Lines        8446     8730     +284     
==========================================
+ Hits         6160     6370     +210     
- Misses       2286     2360      +74     

Impacted Files	Coverage Δ
nf_core/modules/lint/main_nf.py	68.19% <0.00%> (ø)
nf_core/utils.py	82.06% <14.28%> (-0.38%)	⬇️
nf_core/download.py	61.91% <60.61%> (+9.19%)	⬆️
nf_core/synced_repo.py	81.11% <81.11%> (ø)
nf_core/__main__.py	57.09% <100.00%> (+0.14%)	⬆️
nf_core/modules/modules_repo.py	75.36% <100.00%> (-8.29%)	⬇️

[MatthiasZepper]

🎉 All tests are now working. Seems that this might be part of version 2.9 ☢️ Plutonium Penguin 🐧 (or whatever it will be called). Yeah!

[mashehu]

Testing it locally. I don't think the following prompt is needed when running it with --tower, because you would always want them with singularity containers, no?
In addition to the pipeline code, this tool can download software containers. ? Download software container images:

[mirpedrol]

I was testing this on Gitpod, and it failed because there isn't a variable $SHELL. We should probably check first if the variable exists.

[MatthiasZepper]

Good catch! I switched it to os.getenv("SHELL", "") now, which has a default value if the variable is not set.

[MatthiasZepper]

Testing it locally. I don't think the following prompt is needed when running it with --tower, because you would always want them with singularity containers, no? In addition to the pipeline code, this tool can download software containers. ? Download software container images:

I am pretty indifferent there. Tower does not require Singularity containers and are bare-clone can also be used to launch a pipeline directly with Nextflow without Tower, so it might be that people do not want to download them?

[mirpedrol]

LGTM