Protocol class type representation

[kaos]

Bug Report

To Reproduce

repro.py:

from dataclasses import dataclass
from typing import Generic, Protocol, Type, TypeVar

T = TypeVar("T")

class MyProto(Protocol):
    attr: str

@dataclass
class Impl:
    attr: str

@dataclass
class Demo(Generic[T]):
   proto: T
   impl: Type[T]

wrong_T = Demo(MyProto, Impl)
wrong_T = 1  # to expose inferred type of T (which is `object`)

wrong_arg1 = Demo[MyProto](MyProto, Impl)

Expected Behavior

I expected there be a type representing a Protocol class.

I expected T to be inferred as MyProto (as Type[MyProto] is for types implementing the protocol, not the protocol type itself.) for the wrong_T variable. Since Type[T] should accept any object when T is a protocol, and then if that object does not satisfy the Protocol, that is an error.

Moreover, for wrong_arg1, I expected MyProto to be an acceptable argument for type MyProto, but its type is reported as Type[MyProto], but that does not fly, as if that type is expected, and I pass MyProto, it complains (rightfully so):

t.py:24:34: error: Only concrete class can be given where "Type[MyProto]" is expected [misc]

Actual Behavior

$ python -m mypy repro.py
t.py:22:11: error: Incompatible types in assignment (expression has type "int",
variable has type "Demo[object]")  [assignment]
    wrong_T = 1  # to expose inferred type of T
              ^
t.py:24:28: error: Argument 1 to "Demo" has incompatible type "Type[MyProto]";
expected "MyProto"  [arg-type]
    wrong_arg1 = Demo[MyProto](MyProto, Impl)
                               ^
Found 2 errors in 1 file (checked 1 source file)

Your Environment

• Mypy version used: 0.910
• Mypy command-line flags: none
• Mypy configuration options from mypy.ini (and other config files): none
• Python version used: Python 3.8.5
• Operating system and version: MacOS 11.5.2, Darwin Kernel Version 20.6.0

Related StackOverflow question: https://stackoverflow.com/questions/68822297/protocol-implementation-class-type-check

[erictraut]

I'm having a hard time understanding what you're trying to do. Can you provide more details about what the Demo class is meant to do? Are you trying to verify that an implementation conforms to a protocol? Are you trying to associate a protocol class with an implementation class at runtime?

Based on your code sample, I think you might misunderstand three key type checking concepts: protocols, Type annotations, and TypeVar constraint solving. Let me explain those three concepts and see if this clarifies things for you.

A protocol class is intended to be a "template" that describes the interface or "shape" of a type. Generally, protocols are type checking constructs, not runtime constructs. You can't instantiate a protocol class, for example. It looks like you're trying to use protocol classes at runtime, which is generally not what they were designed for.

When you specify a class name within a type annotation (whether it's a protocol class or a non-protocol class), a type checker assumes that you are referring to an instance of that class (or a subtype thereof). If you want to refer to the instantiable class itself, you need to use Type[T]. Using the types in your example, MyProto refers to an instance of a class that conforms to the MyProto protocol, and Type[MyProto] refers to an instantiable class that conforms to the protocol. When a class name is used outside of a type annotation, it refers to the class itself, not an instance of the class. For example, in the expression Demo(MyProto, Impl), the two argument subexpressions refer to classes.

When a function is called and there are type variables within that function's signature, the type checker attempts to "solve" those type variables using the types of the provided arguments. In your example, the Demo class has an implied constructor method that looks like this:

def __init__(self, proto: T impl: Type[T]) -> None: ...

You're calling this __init__ method with the expression Demo(MyProto, Impl). The constraint solver needs to find a type for T that is compatible with both Type[MyProto] and Impl. Mypy uses a "join" operation to find the common type for these two types, and the result of that join operation is object. That means the result of Demo(MyProto, Impl) is typed as Demo[object].

I hope that helps. If you provide more details about what you're trying to do, I might be able to provide more suggestions.

[kaos]

Thank you very much for that detailed response. I'll try and explain what I try to do in more detail (and also, given your emphasis on that the Protocol class is not intended for runtime use, I'm abusing that here, maybe need to reconsider.. well, here goes)

Say I have a registry of classes. When I register a class, I do so by providing an interface as key to what the class supports when registering it. Many different classes can be registered for each interface key. At runtime, the registry is queried using such interface, giving a list of classes back that implements said interface. This is my attempt to use Protocol classes as that interface spec.

The registration is static and hardcoded, thus I'd like to be able to ensure that the registered classes in fact support the interface they are registered under, and thought this would be suitable for a type checker. However, the interface is re-used at runtime, to look up registered classes.

From my draft PR pantsbuild/pants#12577 this example show case the client side of this scenario:

from typing import Protocol, runtime_checkable
from pants.engine.unions import UnionRule

@runtime_checkable
class Vehicle(Protocol):
    """Union base for vehicle types."""
    def model(self) -> str: ...

class Truck(Vehicle, Protocol):
    """Another union base, derived from the first one."""
    wheel_count(self) -> int: ...

class Volvo:
    """Union member."""
    def model(self) -> str: return "XC90"

class Peterbilt:
    """Another union member, for trucks."""
    def model(self) -> str: return "359"
    def wheel_count(self) -> int: return 6

def rules() ->
    return [
        UnionRule(Vehicle, Volvo),
        UnionRule(Truck, Peterbilt),
    ]

The UnionRule is very close to my Demo class above.. and it works, apart from the static type check issue I'm faced with.

So, what I'm trying to get at is, to achieve the same static type checkability for Demo(MyProto, Impl) that I get with a: MyProto = Impl, for any protocol class T.

Does that make sense?

[kaos]

Note that if I change my Demo.proto type to instead be Type, it works, except the type checker will not verify if my Type[T] impl class supports the protocol. (which is the current state of the pants PR, where I instead rely on the runtime_checkable attribute to verify the protocol implementation, at runtime.)

[erictraut]

I don't think this approach will work.

Here is an alternative solution to consider. Perhaps not as elegant as what you were hoping to achieve here. It involves defining a separate register function for each protocol that is registerable.

def register_class(interface: type, cls: type):
    # If all protocol classes are runtime checkable,
    # you can enforce this contract at runtime using
    # the following.
    if not issubclass(cls, interface):
        raise ValueError()
    ...

def register_vehicle(cls: Type[Vehicle]):
    register_class(Vehicle, cls)

def register_truck(cls: Type[Truck]):
    register_class(Truck, cls)

register_vehicle(Volvo) # Works
register_vehicle(Peterbilt) # Works

register_truck(Volvo)  # Type violation
register_truck(Peterbilt) # Works

Another approach that might suit your needs is to use abstract base classes.

[kaos]

Thank you. Much appreciated. I'll consider your suggestions, see where I end up.

[kaos]

Rereading your initial feedback regarding mypy's join, wouldn't it be feasible to teach it to consider Protocol classes specially. That is, that it could accept T and T' to be that union if T is a protocol class and T' is a (any) class. If T' does not fulfill the protocol T, then that could result in an error further down..

I get that protocol classes are not meant for runtime use, however there is a runtime checkable decorator for them, which gives them some credible use at runtime any how.. just feel like it shouldn't be impossible to make this work.. (even though perhaps not worth it)

Many thanks for your valuable input nonetheless. Abstract base classes may very well prove to work out well too.

[erictraut]

That would be a breaking change. It would also create a really odd special case in the constraint solver, and it's not at all clear how it would compose. For example, what happens if the union contains T and T' and some other type? Solving for TypeVars is already very complex and full of tricky edge cases when unions are involved. Of course, you're welcome to propose changes to the current typing standards by posting to the typing-sig, but I think that a special case like the one you've proposed is unlikely to get much support.

It sounds like you're looking for a form of generic meta-programming. The current Python type system isn't designed for this.

[kaos]

I'll close this, given that there is a suitable alternative in abstract base classes to use when the interface is used in a runtime setting. Thank you @erictraut for valuable information and feedback. It's been a very enlightening experience wrgt Protocol classes and their uses. Really powerful concept, when used right ;)