ValueError in a finalizer on PyPy

[whitequark]

When running code that works fine on CPython under PyPy, I'm getting a ValueError raised here:

    def __registerFinalizer(self, handle, finalizer):
        if handle in self.__finalizer_dict:
            finalizer.detach()
            raise ValueError
        self.__finalizer_dict[handle] = finalizer

Is this just a remnant on some debug code that happens to work by chance on CPython, or is there an actual reason to raise it? I'm very confused.

I'm using python-libusb1 3.1.0.

[vpelletier]

This is not a remnant, but a way to cleanly fail instead of having the finalizer randomly break (ex: on interpreter shutdown). __finalizer_dict exists both to enforce the finalization order between instances (basically: the USB context object must be kept alive for any object obtained from it can be finalized) and to allow the "owner" instance to cascade closure requests to its dependents (closing a device handle closes all transfers related to that device).

So if two finalizers fall into the same entry in __finalizer_dict, it is a bug, and this raise is here to alert about this before this becomes the usual nightmare of garbage-collection-related heisenbugs, which themselves could turn into segfaults when libusb calls happen in the wrong order.

Could you post more of the traceback which leads to this failure ? Especially, I wonder which class' __registerFinalizer this is about, and what the caller is so I can figure out what the handle is computed from and hence guess why it may be reused.

Also, do you have a suggestion to clarify the intent of those raises, so they do not look like they may be leftover debugging code ?

EDIT: finalization order happens because references to the required objects are held by the finalizer itself, not by __finalizer_dict.

[whitequark]

Especially, I wonder which class' __registerFinalizer this is about, and what the caller is so I can figure out what the handle is computed from and hence guess why it may be reused.

That was USBDeviceHandle. Probably I got some other exception and it's crashed in the finalizer instead of whichever other way it was going to...

[vpelletier]

Is there a way for me to try to replicate this issue ? I unfortunately have very little time these days (which I would otherwise spend trying to get the new python-libusb1 version out, master has several improvements which were long overdue) but maybe I can figure something out.

At least, I did a quick try of the python-functionfs usbcat example (running device.py with CPython3 and host.py with pypy3 7.3.14), using kernel's dummy_hcd, sent a few packets back and forth and did not get any obvious failure.

In case you would like to reproduce, it goes like this:

Terminal 1, in python-functionfs:

$ sudo modprobe dummy_hcd
$ sudo modprobe libcomposite
$ virtualenv vpy3
$ vpy3/bin/pip install .
$ sudo vpy3/bin/python examples/usbcat/device.py

Terminal 2, in python-libusb1:

$ virtualenv -p pypy3 vpypy3
$ vpypy3/bin/pip install .
$ sudo vpypy3/bin/python .../examples/usbcat/host.py

But this may be too simple of an example to have a chance of triggering the issue.

[whitequark]

Is there a way for me to try to replicate this issue ?

Unfortunately the way I discovered it is by running an userspace RGMII PHY driver with Glasgow on PyPy3.9.