Support **kwargs

[evanunderscore]

While I can't see a good way of passing arbitrary keyword arguments, users should absolutely be able to pass documented keyword arguments. defopt.run is itself an example of a function that accepts **kwargs for Python 2 compatibility but expects at most argv to be specified.

[evanunderscore]

Perhaps this should only be supported if the function signature is overridden in the docstring as per this example. As a side note, I should update defopt.run to do this.

[evanunderscore]

Everything currently maps to the function signature so it's a bit of work to start adding extra parameters. Can look into this again if someone has a use case for it.

[AbeDillon]

I just left huge comment here, but here is the part that may be relevant to this enhancement:

filter_kwargs is also pretty simple, but it relies on some of the internal workings of defopt, so it makes a better candidate for an enhancement to defopt rather than a stand-alone tool:

import defopt

def filter_kwargs(argv, func, short=None):
    """
    pre-filter the given argument vector such that only valid arguments remain
    return the invalid arguments as a dictionary of pairs.

    Args:
        argv (List[str]):
        func (callable):
        short (Dict[str, str]):

    Returns (Dict[str, str]):
    """
    short = short or {}
    parser = defopt.ArgumentParser(formatter_class=defopt._Formatter)
    parser.set_defaults()
    defopt._populate_parser(func, parser, None, short)
    _, unkonwns = parser.parse_known_args(argv)
    if len(unkonwns) % 2:
        raise ValueError(
            "odd number of remaining args preventing proper key-value pairing: %s", (unknowns,))
    kwargs = {k.lstrip("-").replace("-", "_"): v
              for k, v in zip(unkonwns[::2], unkonwns[1::2])}
    for arg in unkonwns:
        argv.remove(arg)
    return kwargs

In order to attempt casting kwargs to reasonable types, I also use a helper function: yamlize. It converts the Dict[str, str] to a yaml-loadable representation so that the yaml parser can guess the types:

import yaml

def yamlize(d):
    """
    Treat a dict mapping strings to strings as though it was defined in a yaml file so that the
    values can be converted to objects according to the yaml parser.

    Args:
        d (Dict[str, str]): a dict of mapping keys to serialized values

    Returns (Dic[str, Any]): the yaml-interpreted version of d

    Example:
        >>> d = {"a": "123", "b": "hello"}
        >>> yd = yamlize(d)
        >>> assert isinstance(yd['a'], int)
        >>> assert isinstance(yd['b'], str)

    """
    s = "\n".join("%s: %s" % i for i in d.items())
    return yaml.load(s)

[evanunderscore]

yamlize makes me feel a little uneasy, since it puts the responsibility on the user to enter arguments in a particular way in order to get appropriate types, rather than on the developer to specify them in advance.

filter_kwargs looks simple enough, but I'll have to read through #7 a few more times to fully understand your use case. I originally wasn't intending on allowing unrestricted keyword arguments.

[AbeDillon]

I think I agree with you on yamlize. It provides simplicity at the expense of explicitness. It kind-of works in my use-case where any **kwargs are assumed to override fields in a YAML config file, but that's a special enough case that it doesn't need to go into defopt.

The simplest solution might be to pack up the kwargs into a Dict[str, str] like filter_kwargs does and allow the developer to specify a function for converting that to a Dict[str, object]:

def main(**kwargs):
    ...

if __name__ == "__main__":
    defopt.run(main, kwarg_parser=yamlize)

[evanunderscore]

I've been thinking about filter_kwargs, and I think the thing that worries me about it is it starts to stray beyond the capabilities of argparse, which I'd rather avoid if I can. Perhaps a decent compromise would be to make some equivalent of _populate_parser public to allow you to do this in your own code safely?

[anntzer]

I think this is a reasonable feature request, essentially requiring the use of parse_known_args and then "manually" parsing the remainder args into --flag value (or --flag value1 value2 for pairs -- variadic cases would probably be a pain because it becomes impossible to know whether --foo a --bar is {"foo": ["a", "--bar"]} or {"foo": ["a"], "bar": []}). (If there are some subcommands using **kwargs and others not, we could manually check whether there are unknown kwargs passed to a kwargsless subcommand and manually error out in that case.)

The yamlization part seems independent and could be done by annotating **kwargs (or any other arg, in fact) with a pseudo-type yaml_load_string(s: str) -> ... that returns the yaml-loaded value.

I'll leave the implementation to anyone who has a real use for this, though.

[lggruspe]

How about something like this? No filtering needed.

 from argparse import ArgumentParser
                                                                   
 def func(**kwargs: float):                                        
     return kwargs
                                                                   
 parser = ArgumentParser()
 parser.add_argument("--kwargs", nargs="*", default=[])            
 args = parser.parse_args()                                        
 
 def convert_dict(m: list[str]) -> dict[str, float]:
     assert len(m) % 2 == 0
     return dict(zip(m[::2], map(float, m[1::2])))
     
 kwargs = convert_dict(args.kwargs)
 print(kwargs, func(**kwargs))                                

The command would be entered as --kwargs foo 1 bar 2. It still wouldn't work for types like dict[str, list[int]], but it should be possible to make it work for kwargs types with "fixed length."
Ex:

# ex: --kwargs foo 1 2 bar 3 4
def func(**kwargs: tuple[int, int]:
    ...

# this should be generated
def convert_dict(m: list[str]) -> dict[str, tuple[int, int]]:
    assert len(m) % 3 == 0
   ...

[anntzer]

This looks like a different request? What I had in mind was --foo 1 --bar 2 mapping to {"foo": 1, "bar": 2}. In particular, for your case, it would make sense to have two flags both with these semantics (--dict1 foo 1 bar 2 --dict2 quux 3 mapping to {"dict1": {"foo": 1, "bar": 2}, "dict2": {"quux": 3}}), whereas with the original interpretation, the point would be that any unknown flag would end up in kwargs.

I think your request can be more generally thought of as a "custom parser that takes more than one parameter", i.e. something like

class MyDict(dict): pass  # just a marker class
def mydictparser(*args): return dict(zip(args[::2], map(float, args[1::2])))
def main(kws: MyDict): ...
defopt.run(main, parsers={MyDict: mydictparser})

which is not supported now (currently custom parsers all take a single argument) but seems reasonable to have (well, as usual someone needs to write the implementation :-)) -- we could just introspect the signature of the custom parser...

I guess supporting variadic custom parsers is also linked to dataclass support (#82 (comment)), as these basically could behave like variadic parsers (except for the tricky question of whether they should introduce flags or be positional...)

[ashwin153]

I have a use-case where I am embedding the Airflow CLI (which uses argparse) inside my defopt CLI. I would like to be able to pass the command line through to the Airflow CLI. My command is defined as follows. As you can see, positional arguments are passed through. Is there a way to pass through keyword arguments as well? In effect, I would like to instruct defopt to stop parsing arguments after the sub-command.

def airflow(*args: str):
    sys.argv.pop(1)
    from airflow import __main__
    sys.exit(__main__.main())

[anntzer]

The classical way to do this (if I understand your request correctly) would be to use -- to mark everything after as positional args (see end of https://docs.python.org/3/library/argparse.html#arguments-containing).

[ashwin153]

Awesome thank you! In case this is useful to anyone, this is what I ended up using.

def airflow(*args: str):
    sys.argv.pop(1)
    if "--" in sys.argv:
        sys.argv.remove("--")

    from airflow import __main__
    sys.exit(__main__.main())