Type checking operators

[heilerich]

Hello everyone,

I am trying to get type checking to work on my operator, but I can't seem to figure out how to properly define the handler signatures.

Looking at the examples in the documentation and various other resource in this repo have tried

@kopf.on.startup()
def init_memo(*, memo: kopf.Memo, **_: Any) -> None:
    """Initialize memo"""
    pass

as well as

@kopf.on.startup()
def init_memo(memo: kopf.Memo, **_: Any) -> None:
    """Initialize memo"""
    pass

But mypy still complains

error: Argument 1 has incompatible type "Callable[[NamedArg(Memo, 'memo'), KwArg(Any)], None]"; expected "Callable[[NamedArg(OperatorSettings, 'settings'), NamedArg(Index[Any, Any], '*'), NamedArg(int, 'retry'), NamedArg(datetime, 'started'), NamedArg(timedelta, 'runtime'), NamedArg(Union[Logger, LoggerAdapter], 'logger'), NamedArg(Union[Memo, object], 'memo'), DefaultNamedArg(Any, 'param'), KwArg(Any)], Union[Result, None, Coroutine[None, None, Optional[Result]]]]"

Is this the expected behaviour? How is type checking in operators supposed to work? Do I need to explicitly list all the possible arguments (and then silence a ton of warning about unused parameters)?

I am fairly new to python typing so any explanation would be greatly appreciated.

Greetings

[nolar]

No, all arguments are not needed. It seems, the problem is not with the arguments, but specifically with memo: kopf.Memo. The callback type is declared to also accept any kind of object (defined as AnyMemo=Union[Memo, object]), but it does not work as expected when the actual kwarg is more specific than the expected callback type. I'll take a look at how this can be improved.

[heilerich]

There seem to be some more affected functions and parameters, essentially all the Union parameters I could spot, e.g. logger, old and new. The question is does it make sense, i.e. does every callback need to make sure it can deal with any of the declared types at runtime? In that case, I think the current types would need to stay unchanged.

The most convenient situation from a user (or at least my) perspective would be having overloaded function signatures and at runtime, if I declare a callback that expects a kopf.Memo, the library ensures that I will only receive said type.
Maybe this warrants a bug/feature request issue?

[nolar]

There is no need to overcomplicate the Kopf-based operators. It is a bug in the framework. I've made a fix: #747.

If the same problem happens with other keywords, please report them. I'll take a look at the loggers now too. I guess, there will be the same problem with non-None results from the functions.

As a rule of thumb, for every kwarg named whatever, there should be a corresponding type or class named kopf.Whatever (with some exceptions like settings: kopf.OperatorSettings, but this can be revised). If Union[…], Optional[…], third-party modules or stdlib (e.g. logging), or other typing tricks are needed, then it is a bug and should be fixed in the framework.

[heilerich]

Thanks 👍🏻 Greatly appreciated!

[nolar]

I've tried to make all examples type-annotated and strictly type-checked now.

Sadly so, it seems impossible to make Kopf-based operators properly annotated at the moment, and will be impossible at least until Python 3.10 is released in Oct'2021 (and then PEP-612's ParamSpec backported to typing_extensions for Python 3.7-3.9).

Until then, active use of # type: ignore on the lines with decorators seems like the only workaround for strictly typed operators — for indexes, sync daemon stopper flag, and some other edge cases. Though, typing works for the majority of regular cases.

[reegnz]

@nolar any progress? Python 3.10 has been out for a while, but I'm still seeing type checking errors with kopf.

Eg. looking at this code in vscode using pylance:

@kopf.on.create("MyCustomResource")
def create_my_custom_resource(body: kopf.Body, **_: Any) -> None:
    pass

I'm getting the following error:

Argument of type "(body: Body, **_: Any) -> None" cannot be assigned to parameter of type "ChangingFn"
  Type "(body: Body, **_: Any) -> None" cannot be assigned to type "ChangingFn"
    Function accepts too many positional parameters; expected 1 but received 22
      Parameter 1: type "int" cannot be assigned to type "Body"
        "int" is incompatible with "Body"Pylance[reportGeneralTypeIssues](https://github.com/microsoft/pyright/blob/main/docs/configuration.md#reportGeneralTypeIssues)

There's no combination args and types I could use to satisfy the type checker.