Beta support for configurable dependency resolution & Biocontainers.

[jmchilton]

Consider the included tool seqtk_seq.cwl. It includes the following SoftwareRequirement hint:

hints:
  SoftwareRequirement:
    packages:
    - package: seqtk
      version:
      - r93

I'm not happy that version is a list - but I can live with it for now I guess.

If cwltool is executed with the hidden --beta-conda-dependencies flag, this requirement will be processed by galaxy-lib, Conda will be installed, and seqtk will be installed, and a Conda environment including seqtk will be setup for the job.

virtualenv .venv
. .venv/bin/activate
python setup.py install
pip install galaxy-lib
cwltool --beta-conda-dependencies tests/seqtk_seq.cwl tests/seqtk_seq_job.json

Additional flags are available to configure dependency resolution in a more fine grained way - using Conda however has a number of advantages that make it particularily well suited to CWL. Conda packages are distributed as binaries that work across Mac and Linux and work on relatively old version of Linux (great for HPC). Conda also doesn't require root and supports installation of multiple different versions of a package - again these factors make it great for HPC and non-Docker targets.

The Biocontainers project (previously Biodocker) dovetails nicely with this. Every version of every Bioconda package has a corresponding best-practice (very lightweight, very small) Docker container on quay.io (assembled by @bgruening and colleagues). There are over 1800 such containers currently.

Continuing with the example above, the new --beta-use-biocontainers flag instructs cwltool to fetch the corresponding Biocontainers container from quay.io automatically or build one to use locally (required for instance for tools with multiple software requirements - fat tools).

cwltool --beta-use-biocontainers tests/seqtk_seq.cwl tests/seqtk_seq_job.json

These containers contain the same binaries that the package would use locally (outside of Docker). Therefore this technique allows cross platform reproducibility/remixability across CWL, Galaxy, and CLI - both inside and outside of Docker.

My sincerest hope is that we move away from CWL-specific Dockerfiles. For less effort, a community bioconda package can be made and the result can be used in many more contexts. The Docker image will then be maintained by the community Biocontainer project.

[tetron]

This is awesome! I will review at the soonest opportunity (likely early next week).

"CWL-specific Dockerfiles" I think you mean Docker-specific CWL files?

[tetron]

I definitely want this feature, but I'd like to see some more work before merging. Are you able to continue working on this branch in response to comments? Otherwise can see about working off of this branch.

The intended design of SoftwareRequirement is to avoid picking favorites with the packaging system. The specs field lists URIs for one or more concrete packages to fulfill the abstract requirement. For example:

hints:
  SoftwareRequirement:
    packages:
    - package: seqtk
      version:
      - r93
      specs:
      - https://anaconda.org/bioconda/seqtk
      - https://packages.debian.org/jessie/seqtk

Resolution should work something like this:

1. Determine which resolvers (conda, pip, apt, yum, etc.) are available.
2. Pass the packages list to each resolver
3. Each resolver looks at specs for each package and matches the prefix (eg https://anaconda.org/bioconda/, https://packages.debian.org/)
4. For each resolver that matched one or more packages, generate a fragment of shell script that installs the package.
5. Combine the shell script fragments with the tool execution command to get the job script.

[tetron]

DependenciesConfigruation is misspelled.

Reviewing #212, I'm not sure why a separate PYTHON_RUN_SCRIPT is necessary. Everything that it does can be accomplished generating shell code?

[jmchilton]

@tetron I don't think the specs should strictly be needed - I can look at seqtk and find it Conda, Brew, and Debian. Same is going to hold for most packages in our experience.

Back in #93 I made the same argument passionately - that time about bwa.

do we have a long, long list of IDs in every tool definition?

This remains a false dichotomy and a misstatement of my position. The proposed solution works fine in 95% of cases - it should surprise no one to learn that Homebrew, Conda, MSI's enviornment modules, Debian, and Nix all call bwa... bwa. Small variations will of course pop up - but those can be solved with site or package-manager specific mappings on the backend - small tweaks - there is no need for long lists of IDs in tools.

That comment was +1'd by @pvanheus who has a lot more experience adapting software inventories these platforms - Galaxy and cwl.

The tool author shouldn't need to know every last package manager that this may be deployed against - forcing them to know this a priori is a mistake and is a recipe for non-future facing, brittle tools.

If instead you want to use the spec field as an optional hint when the answer isn't the intuitive, natural one, then I think this idea is acceptable and I'm more than happy to add it to the galaxy-lib stuff myself. The resulting improvement would benefit Galaxy at the margins also. But it is the margins - I assure you.

If you ignore the spec thing however - --beta-dependency-resolvers-configuration can set up cascades of resolvers the way you described. The conda flag is just a shortcut to say - install and enable only the conda resolver. Which is nice because it is the Galaxy communities' best practice and it has reproducibility with the Biocontainers.

[jmchilton]

@tetron If you can accomplish #212 without the Python script that would be awesome sauce. I think that would be a big improvement over my attempt. I didn't really know how to accomplish the sort of safe command handling inside of a shell script.

[jmchilton]

@tetron Rebased with noted spelling fix - thanks for the catch!

[tetron]

If instead you want to use the spec field as an optional hint when the answer isn't the intuitive, natural
one, then I think this idea is acceptable and I'm more than happy to add it to the galaxy-lib stuff myself.
The resulting improvement would benefit Galaxy at the margins also. But it is the margins - I assure you.

Ok, this seems reasonable. In that case, the behavior should be something like, check specs for an explicit package identifier for the resolver, if found, use that; if not, then try the default name from the package field.

Let me see if I'm following: the strategy is to generate shell script which does the package installation prior to running the command. This may run on a separate node (or VM or container environment) from the workflow engine. How does this work with multiple package managers or package versions? Does it generate an if-then-else chain?

[jmchilton]

@tetron So a limitation of the current implementation is that it does the install or generates the shell code to execute runs on the node running cwltool. (This works on fine on clusters that have a certain directory mounted across them and in Galaxy land our remote job execution tool Pulsar knows how to resolve dependencies so for jobs without a shared file system this would happen closer to the job.) I guess cwltool is somewhat explicitly single node so this shouldn't be a problem right? Explicit support for something like Toil would be great - I'd be happy to field PRs to galaxy-lib to enable that or do it myself if I sense enough eagerness for it.

If multiple "package managers" are used (the term is a misnomer since something like Galaxy packages or environment modules are not package mangers) - each may resolve packages individually and produce its own string for injecting the environment into the shell.

[cwl-bot]

Can one of the admins verify this patch?

[jmchilton]

And we are up and running on this branch again at the GCC 2017 Hackathon. Just summarize what I have done and some conversations...

• So I have rebased this branch - there were minor updates on both sides that needed to be adjusted for. The biggest changes were do to modifications to the way default container was handled in this repository. I think what is here is a bit hacky now but it was in my previous attempt as well. The previous variant of this is backed up and can viewed at https://github.com/common-workflow-language/cwltool/compare/master...jmchilton:galaxy_deps_old?expand=1.

• I reran the biocontainers and the conda examples in the test above - they both still work properly on my new devbox.

• Specs have been implemented upstream in Implement Dependency Resolver Mapping galaxyproject/galaxy#3444 - after a big debate we have decided not to expose that concept in Galaxy tools for now - but they should be available for cwltool.

  So in response to @tetron comment:

  In that case, the behavior should be something like, check specs for an explicit package identifier for the resolver, if found, use that; if not, then try the default name from the package field.

  This is in fact how it works now I think - I will follow up with additional tests for that today to demonstrate the functionality.

• Implement Dependency Resolver Mapping galaxyproject/galaxy#3444 also implemented mapping for environment modules. I will try to writeup some documentation about how to leverage that from cwltool. This could be a powerful tool for adapting CWL tools for HPC centers. (Though Conda is even better 😉 ).

• @mr-c requested this be more of a plugin for non-Conda stuff again - I restated that I would definitely field PRs for integrating Debian resolution and we support many non-Conda package approaches - including Homebrew, custom scripts that modify the environment (i.e. "Galaxy packages"), and environment modules.

• galaxy-lib now can build Singularity containers on-demand from these dependency specifications - if I have time I'll try to implement Singularity support in cwltool to utilize this as well.

So my TODO list for the rest of the hackathon is:

• Add example for specs.*
• Add example for environment modules.*
• Redo the type signatures for galaxy-lib - I think they are broken and breaking the build.*
• Check if this branch against toil - see if the concepts work.
• Implement singularity support in cwltool.
• Implement more Debian support in galaxy-lib.

These last two won't actually happen - but they sound fun.

[jxtx]

Tested using --strict --no-container --beta-conda-dependencies on https://github.com/bxlab/vision-workflows/blob/master/chipseq_tf.cwl, works great!

[mr-c]

@jmchilton https://travis-ci.org/common-workflow-language/cwltool/jobs/247570204#L195

[mr-c]

This needs a full type signature. It returns a DependencyManager, but I am unsure of what type config is.

[mr-c]

This needs a full type signature.

[mr-c]

This needs a full type description

[mr-c]

This needs a full type description

[mr-c]

This line is causing multiple tests to fail.

[mr-c]

Can the rest of this be moved to an external file?

[jmchilton]

• I have refactored the dependency handling adapter code out of main.py and into a new software_requirements module.
• I have refined the type signatures for galaxy-lib for the recent changes to cwltool and galaxy-lib.

[jmchilton]

@mr-c This branch's README now contains lots of documentation about SoftwareRequirements including how to resolve them with environment modules, custom package scripts, and Conda. It covers mapping to local infrastructure (for admins) as well mapping to multiple package managers from the tools themselves (for tool developers). It has many example calls that should work without external dependencies covering most of the test data I added and that can be run to try these things out today. It also contains some external links and information about installing the optional dependencies required to run these.

Here are the docs - https://github.com/jmchilton/cwltool/tree/galaxy_deps#leveraging-softwarerequirements-beta.

[jmchilton]

@mr-c Consider this comment documentation that I agree to allow uploading of galaxy-lib stubs to typeshed. Hope this is good enough?

[mr-c]

Hmm.. should we hide them when the deps extra isn't modified? What happens if they are called and galaxy-lib is missing?

[jmchilton]

I have clarified this behavior with #459 and improved it a bit.

[mr-c]

Merging as this is tremendous; we can add additional polish later. Thank you @jmchilton !

[bgruening]

Wuhu, after such a long time. Great to see it finally happen.

[manu-chroma]

@jmchilton are the galaxy typesheds python 3 compatible?

[jmchilton]

The code should be Python 3 compatible. I must confess I'm unsure what it means for the typesheds to be Python 3 compatible.

[manu-chroma]

I just had a looks at the galaxy stub files. They should work with both python2 and python3. :)

I just wanted to confirm if the galaxy codebase is py3 compatible and how the stubs for galaxy project present in typesheds/ folder were generated.

[jmchilton]

@manu-chroma Sounds good! If anything I've added makes the Python 3 stuff more difficult let me know and I can try to update it.