Naming convention

[ghisvail]

This thread discusses general guidelines for naming in Pydra, including package, task definitions and specs and specification attributes.

For task packages

pydra-<namespace> whereby namespace corresponds to a unique name in lowercase, where the package files will be installed, i.e. under pydra/tasks/namespace. Please verify your namespace availability on PyPI.

For task packages specific to a research group or supporting an application, please use pydra-<group>-<namespace>. For instance, to publish a FSL package for the mylab research group, use pydra-mylab-fsl.

To avoid proliferation of similar task packages, please check for existing packages on PyPI and reach out to maintainers if your desired pieces of functionality are absent. Official task packages are maintained under the Nipype organization.

For task definitions

Task definitions are typically composed of a task class (derived from pydra.engine.ShellCommandTask), input specifications (derived from pydra.specs.ShellSpec) and output specifications (derived from pydra.specs.ShellOutSpec) eventually.

The recommended naming scheme for a foo_bar utility is the following:

• Task class name: FooBar
• Input specifications: FooBarSpec
• Output specifications: FooBarOutSpec

For specification fields

Input and output specifications features one or multiple fields which typically map to a command-line argument. Each field is assigned a name, a type and a set of metadata, such as its corresponding CLI flag name and help string.

The field name must be in snake case (like regular Python class attributes) and should follow the pattern [<qualifiers>]_<suffix>, i.e. a suffix describing the nature of the argument (an image, a point) and one or more qualifiers depending on its use.

We advise AGAINST using the command-line flag name as the field name, unless the former is explicit enough. Command-line arguments are often optimized to save keystrokes and feature abbreviations with varying familiarity to the target audience. Instead, please consider readability and expressiveness by choosing longer and more descriptive field names.

For instance, a CLI argument such as --rndseed, which sets the random seed for an algorithm, could be defined as follows:

class AlgorithmSpec(...):

    random_seed: int = attrs.field(
        metadata={
            "help_string": "random seed for algorithm",
            "argstr": "--rndseed",
        }
    )

[tclose]

The syntactic sugar I was thinking we could add is just to wrap the attrs.field call in a method like

def cmd_arg(default, help_string: str, argstr: str = "", mandatory: bool = False,...):
    """
    Adds an argument to a Pydra shell command task definition

    Parameters
    ------------
    help_string : str
        User-facing description of the argument 
    argstr: str
        Prefix prepended to the argument when printed to the command line
    ....
    """

    return attrs.field(
        default,
        metadata={
            "help_string": help_string,
            "argstr": argstr,
        }
    )

As this would allow us to document all the valid metadata fields that can be passed to the attrs.field. Then the class would look like

from pydra.mark import cmd_arg

@attrs.define
class AlgorithmSpec(...):

    random_seed: int = cmd_arg(help_string="random seed for algorithm", argstr="--rndseed")

[tclose]

Could we also consider adding the InputSpec and OutputSpec as class attributes of the overall shell command, e.g.

from pydra.mark import shell_cmd

@shell_cmd
class Algorithm:

    executable = "algorithm"

    class Inputs:
        random_seed: int = cmd_arg(help_string="random seed for algorithm", argstr="--rndseed")

    class Outputs:
        .... outputs to go here ....

[tclose]

Challenging dynamic shell commands

• FSLmaths
• FSLStats
• ANTSRegistration

[tclose]

I have now implemented the syntax in the post above in PR #655, please check it out and let me know what you think

[tclose]

Looking at the code around input_spec and output_spec, from what I can tell they would be redundant for ShellCommands if the way of defining shell commands in #655 is adopted as Inputs and Outputs could be used their place.

So I have mocked up what it would look like for the function task decorator (minus the code for dynamic function task construction) if we wanted to drop them for a class based approach.

import typing as ty
import attrs
import inspect
from functools import wraps


class FunctionTask:
    func: ty.Callable
    name: str
    Inputs: type
    Outputs: type

    def __init__(self, name: str, **kwargs):
        self.name = name
        self.inputs = type(self).Inputs(**kwargs)

    def __call__(self) -> "FunctionTask":
        return type(self).func(
            **{f.name: getattr(self.inputs, f.name) for f in attrs.fields(self.Inputs)}
        )


def task(
    function: ty.Optional[ty.Callable] = None,
    returns: ty.Optional[ty.Iterable[str]] = None,
) -> ty.Callable | ty.Type[FunctionTask]:
    def decorator(function: ty.Callable) -> ty.Type[FunctionTask]:
        sig = inspect.signature(function)

        inputs_dct = {
            p.name: attrs.field(default=p.default) for p in sig.parameters.values()
        }
        inputs_dct["__annotations__"] = {
            p.name: p.annotation for p in sig.parameters.values()
        }

        outputs_dct = {r: attrs.field() for r in (returns if returns else ["out"])}
        if sig.return_annotation:
            outputs_dct["__annotations__"] = {
                p.name: p.annotation for p in sig.return_annotation
            }

        @wraps(function, updated=())
        @attrs.define(kw_only=True, slots=True, init=False)
        class Task(FunctionTask):
            func = function

            Inputs = attrs.define(type("Inputs", (), inputs_dct))  # type: ty.Any
            Outputs = attrs.define(type("Outputs", (), outputs_dct))  # type: ty.Any

            inputs: Inputs = attrs.field()

        return Task

    if function:
        return decorator(function)
    else:
        return decorator

While it wouldn't make much/any difference to the end user, it would perhaps harmonise the code a little more if all task decorators return a subclass of TaskBase (instead of function decorators returning a wrapped function). I have also thrown in the returns kwarg as a shortcut to avoid having to use the annotate decorator to just define the names of the return values

Perhaps it is a little surprising if a wrapped function returns a class, but then that is kind of what we are doing in effect by essentially getting the user to parameterise the wrapped function, which then returns a function task object, and currently the inputs/outputs of wrapped function aren't easily inspectable.