Allow finer grain specification of requirements.

This adapts CWL-style ``SoftwareRequirement`` ``specs`` to solve galaxyproject#1927. Here I'm trying to implement the CWL specification in a way that helps enable the feasibility of Conda packaging in Galaxy. It is a delicate balancing act aimed to upset as many interested parties as I can. To understand the problem - consider the ``blast+`` requirement found in the Galaxy wrappers. It looks something like this: ``` <requirement type="package" version="2.2.31" name="blast+"> ``` Great, that works for Galaxy and Tool Shed packages. It doesn't work for bioconda at all. I think this problem is fairly uncommon - most packages have simple names shared across Debian, Brew, Conda, etc... - but it does happen in some cases that there are inconsistencies. Some people have taken to duplicating the requirement - this is bad and should not be done since they are mutually exclusive and Galaxy will attempt to resolve both. This introduces the following syntax for tools with profile >= 16.10: ``` <requirement type="package" version="2.2.31" name="blast+"> <specification uri="https://anaconda.org/bioconda/blast" /> <specification uri="https://packages.debian.org/sid/ncbi-blast+" version="2.2.31-3" /> </requirement> ``` This allows finer grain resolution of the requirement without sacrificing the abstract name at the top. It allows the name and the version to be adapted by resolvers as needed (hopefully rarely so). This syntax is the future facing one, but obviously this tool would not work on older Galaxy versions. To remedy this - an alternative syntax can be used for tools targetting Galaxy verions pre-16.10: ``` <requirement type="package" version="2.2" specification_uris="https://anaconda.org/bioconda/[email protected],https://packages.debian.org/jessie/[email protected]">blast+</requirement> ``` This syntax sucks - but it does give newer Galaxies the ability to resolve the specifications without breaking the more simple functionality for older Galaxies. For more information on the CWL side of this - checkout the discussion on common-workflow-language/cwltool#214. The CWL specification information is defined at http://www.commonwl.org/v1.0/CommandLineTool.html#SoftwarePackage. Conflicts: lib/galaxy/tools/deps/__init__.py test/functional/tools/samples_tool_conf.xml
jmchilton · Jan 23, 2017 · cd2cf49 · cd2cf49
1 parent 4eade31
commit cd2cf49
Show file tree

Hide file tree

Showing 7 changed files with 171 additions and 24 deletions.
diff --git a/lib/galaxy/tools/deps/__init__.py b/lib/galaxy/tools/deps/__init__.py
@@ -141,6 +141,7 @@ def _requirements_to_dependencies_dict(self, requirements, **kwds):
             # Check requirements all at once
             all_unmet = len(requirement_to_dependency) == 0
             if all_unmet and hasattr(resolver, "resolve_all"):
+                # TODO: Handle specs.
                 dependencies = resolver.resolve_all(resolvable_requirements, **kwds)
                 if dependencies:
                     assert len(dependencies) == len(resolvable_requirements)
@@ -155,7 +156,17 @@ def _requirements_to_dependencies_dict(self, requirements, **kwds):
                 if requirement in requirement_to_dependency:
                     continue
 
-                dependency = resolver.resolve( requirement.name, requirement.version, requirement.type, **kwds )
+                name = requirement.name
+                version = requirement.version
+                specs = requirement.specs
+
+                if hasattr(resolver, "find_specification"):
+                    spec = resolver.find_specification(specs)
+                    if spec is not None:
+                        name = spec.short_name
+                        version = spec.version or version
+
+                dependency = resolver.resolve( name, version, requirement.type, **kwds )
                 if require_exact and not dependency.exact:
                     continue
 

diff --git a/lib/galaxy/tools/deps/requirements.py b/lib/galaxy/tools/deps/requirements.py
@@ -14,29 +14,66 @@ class ToolRequirement( object ):
     run (for example, a program, package, or library).  Requirements can
     optionally assert a specific version.
     """
-    def __init__( self, name=None, type=None, version=None ):
+    def __init__( self, name=None, type=None, version=None, specs=[] ):
         self.name = name
         self.type = type
         self.version = version
+        self.specs = specs
 
     def to_dict( self ):
-        return dict(name=self.name, type=self.type, version=self.version)
+        specs = [s.to_dict() for s in self.specs]
+        return dict(name=self.name, type=self.type, version=self.version, specs=specs)
 
     @staticmethod
     def from_dict( dict ):
         version = dict.get( "version", None )
         name = dict.get("name", None)
         type = dict.get("type", None)
-        return ToolRequirement( name=name, type=type, version=version )
+        specs = [RequirementSpecification.from_dict(s) for s in dict.get("specs", [])]
+        return ToolRequirement( name=name, type=type, version=version, specs=specs )
 
     def __eq__(self, other):
-        return self.name == other.name and self.type == other.type and self.version == other.version
+        return self.name == other.name and self.type == other.type and self.version == other.version and self.specs == other.specs
 
     def __ne__(self, other):
         return not self.__eq__(other)
 
     def __hash__(self):
-        return hash((self.name, self.type, self.version))
+        return hash((self.name, self.type, self.version, frozenset(self.specs)))
+
+
+class RequirementSpecification(object):
+    """Refine a requirement using a URI."""
+
+    def __init__(self, uri, version=None):
+        self.uri = uri
+        self.version = version
+
+    @property
+    def specifies_version(self):
+        return self.version is not None
+
+    @property
+    def short_name(self):
+        return self.uri.split("/")[-1]
+
+    def to_dict(self):
+        return dict(uri=self.uri, version=self.version)
+
+    @staticmethod
+    def from_dict(dict):
+        uri = dict.get["uri"]
+        version = dict.get("version", None)
+        return RequirementSpecification(uri=uri, version=version)
+
+    def __eq__(self, other):
+        return self.uri == other.uri and self.version == other.version
+
+    def __ne__(self, other):
+        return not self.__eq__(other)
+
+    def __hash__(self):
+        return hash((self.uri, self.version))
 
 
 class ToolRequirements(object):
@@ -173,10 +210,29 @@ def parse_requirements_from_xml( xml_root ):
 
     requirements = ToolRequirements()
     for requirement_elem in requirement_elems:
-        name = xml_text( requirement_elem )
+        if "name" in requirement_elem.attrib:
+            name = requirement_elem.get( "name" )
+            spec_elems = requirement_elem.findall("specification")
+            specs = map(specification_from_element, spec_elems)
+        else:
+            name = xml_text( requirement_elem )
+            spec_uris_raw = requirement_elem.attrib.get("specification_uris", "")
+            specs = []
+            for spec_uri in spec_uris_raw.split(","):
+                if not spec_uri:
+                    continue
+                version = None
+                if "@" in spec_uri:
+                    uri, version = spec_uri.split("@", 1)
+                else:
+                    uri = spec_uri
+                uri = uri.strip()
+                if version:
+                    version = version.strip()
+                specs.append(RequirementSpecification(uri, version))
         type = requirement_elem.get( "type", DEFAULT_REQUIREMENT_TYPE )
         version = requirement_elem.get( "version", DEFAULT_REQUIREMENT_VERSION )
-        requirement = ToolRequirement( name=name, type=type, version=version )
+        requirement = ToolRequirement( name=name, type=type, version=version, specs=specs )
         requirements.append( requirement )
 
     container_elems = []
@@ -188,6 +244,12 @@ def parse_requirements_from_xml( xml_root ):
     return requirements, containers
 
 
+def specification_from_element(specification_elem):
+    uri = specification_elem.get("uri", None)
+    version = specification_elem.get("version", None)
+    return RequirementSpecification(uri, version)
+
+
 def container_from_element(container_elem):
     identifier = xml_text(container_elem)
     type = container_elem.get("type", DEFAULT_CONTAINER_TYPE)

diff --git a/lib/galaxy/tools/deps/resolvers/__init__.py b/lib/galaxy/tools/deps/resolvers/__init__.py
@@ -67,6 +67,34 @@ def _to_requirement(self, name, version=None):
         return ToolRequirement(name=name, type="package", version=version)
 
 
+class SpecificationAwareDependencyResolver:
+    """Mix this into a :class:`DependencyResolver` to implement URI specification matching.
+
+    Allows adapting generic requirements to more specific URIs - to tailor name
+    or version to specified resolution system.
+    """
+    __metaclass__ = ABCMeta
+
+    @abstractmethod
+    def find_specification(self, specs):
+        """Find closest matching specification for discovered resolver."""
+
+
+class SpecificationPatternDependencyResolver:
+    """Implement the :class:`SpecificationAwareDependencyResolver` with a regex pattern."""
+
+    @abstractproperty
+    def _specification_pattern(self):
+        """Pattern of URI to match against."""
+
+    def find_specification(self, specs):
+        pattern = self._specification_pattern
+        for spec in specs:
+            if pattern.match(spec.uri):
+                return spec
+        return None
+
+
 class InstallableDependencyResolver:
     """ Mix this into a ``DependencyResolver`` and implement to indicate
     the dependency resolver can attempt to install new dependencies.

diff --git a/lib/galaxy/tools/deps/resolvers/conda.py b/lib/galaxy/tools/deps/resolvers/conda.py
@@ -5,6 +5,7 @@
 
 import logging
 import os
+import re
 
 import galaxy.tools.deps.installable
 
@@ -29,6 +30,7 @@
     InstallableDependencyResolver,
     ListableDependencyResolver,
     NullDependency,
+    SpecificationPatternDependencyResolver,
 )
 
 
@@ -39,9 +41,10 @@
 log = logging.getLogger(__name__)
 
 
-class CondaDependencyResolver(DependencyResolver, ListableDependencyResolver, InstallableDependencyResolver):
+class CondaDependencyResolver(DependencyResolver, ListableDependencyResolver, InstallableDependencyResolver, SpecificationPatternDependencyResolver):
     dict_collection_visible_keys = DependencyResolver.dict_collection_visible_keys + ['conda_prefix', 'versionless', 'ensure_channels', 'auto_install']
     resolver_type = "conda"
+    _specification_pattern = re.compile(r"https\:\/\/anaconda.org\/\w+\/\w+")
 
     def __init__(self, dependency_manager, **kwds):
         self.versionless = _string_as_bool(kwds.get('versionless', 'false'))

diff --git a/lib/galaxy/tools/xsd/galaxy.xsd b/lib/galaxy/tools/xsd/galaxy.xsd
@@ -227,7 +227,7 @@ complete descriptions of the runtime of a tool.
     </xs:sequence>
   </xs:complexType>
 
-  <xs:complexType name="Requirement">
+  <xs:complexType name="Requirement" mixed="true">
     <xs:annotation>
       <xs:documentation xml:lang="en"><![CDATA[
 
@@ -276,20 +276,29 @@ resolver.
 
 ]]></xs:documentation>
     </xs:annotation>
-    <xs:simpleContent>
-      <xs:extension base="xs:string">
-        <xs:attribute name="type" type="RequirementType" use="required">
-          <xs:annotation>
-            <xs:documentation xml:lang="en"> This value defines the which type of the 3rd party module required by this tool. </xs:documentation>
-          </xs:annotation>
-        </xs:attribute>
-        <xs:attribute name="version" type="xs:string">
-          <xs:annotation>
-            <xs:documentation xml:lang="en"> For package type requirements this value defines a specific version of the tool dependency. </xs:documentation>
-          </xs:annotation>
-        </xs:attribute>
-      </xs:extension>
-    </xs:simpleContent>
+    <xs:sequence>
+      <xs:element name="specification" minOccurs="0" maxOccurs="unbounded" type="xs:anyType" />
+    </xs:sequence>
+    <xs:attribute name="type" type="RequirementType" use="required">
+      <xs:annotation>
+        <xs:documentation xml:lang="en"> This value defines the which type of the 3rd party module required by this tool. </xs:documentation>
+      </xs:annotation>
+    </xs:attribute>
+    <xs:attribute name="version" type="xs:string">
+      <xs:annotation>
+        <xs:documentation xml:lang="en"> For package type requirements this value defines a specific version of the tool dependency. </xs:documentation>
+      </xs:annotation>
+    </xs:attribute>
+    <xs:attribute name="name" type="xs:string">
+      <xs:annotation>
+        <xs:documentation xml:lang="en">Name of requirement (if body of ``requirement`` element contains specification URIs).</xs:documentation>
+      </xs:annotation>
+    </xs:attribute>
+    <xs:attribute name="specification_uris" type="xs:string">
+      <xs:annotation>
+        <xs:documentation xml:lang="en">URIs and versions of requirement specification.</xs:documentation>
+      </xs:annotation>
+    </xs:attribute>
   </xs:complexType>
   <xs:complexType name="Container">
     <xs:annotation>

diff --git a/test/functional/tools/requirement_specification_1.xml b/test/functional/tools/requirement_specification_1.xml
@@ -0,0 +1,18 @@
+<tool id="requirement_specification_1" name="requirement_specification_1" version="0.1.0" profile="16.10">
+    <command><![CDATA[
+        blastn -help > $out_file1 ;
+        echo "Moo" >> $out_file1 ;
+    ]]></command>
+    <requirements>
+        <requirement type="package" version="2.2.31" name="blast+">
+            <specification uri="https://anaconda.org/bioconda/blast" />
+            <specification uri="https://packages.debian.org/sid/ncbi-blast+" version="2.2.31-3" />
+        </requirement>
+    </requirements>
+    <inputs>
+        <param name="input1" type="data" optional="true" />
+    </inputs>
+    <outputs>
+        <data name="out_file1" format="txt" />
+    </outputs>
+</tool>
diff --git a/test/functional/tools/requirement_specification_2.xml b/test/functional/tools/requirement_specification_2.xml
@@ -0,0 +1,16 @@
+<tool id="requirement_specification_2" name="requirement_specification_2" version="0.1.0" profile="16.01">
+    <command><![CDATA[
+        blastn -help > $out_file1 ;
+        echo "Moo" >> $out_file1 ;
+    ]]></command>
+    <requirements>
+        <!-- Demonstrate backward-compatible-ish specification_uri syntax. -->
+        <requirement type="package" version="2.2" specification_uris="https://anaconda.org/bioconda/[email protected],https://packages.debian.org/jessie/[email protected]">blast+</requirement>
+    </requirements>
+    <inputs>
+        <param name="input1" type="data" optional="true" />
+    </inputs>
+    <outputs>
+        <data name="out_file1" format="txt" />
+    </outputs>
+</tool>