CuQuantum exception

[renatomello]

I am using the cuquantum backend and the following error keeps being raised.

Exception ignored in: <function CuQuantumBackend.__del__ at 0x7ff976e4c160>
Traceback (most recent call last):
  File /path/to/lib/python3.9/site-packages/qibojit/backends/gpu.py", line 509, in __del__
TypeError: 'NoneType' object is not callable

Though I could not trace any reason for it or how an minimal code could raise it.

[alecandido]

First, I noticed that cuquantum installation is not managed by Poetry at all (not even as an extra). The required package should be cuquantum-python (not yet sure for what would you need the bare cuquantum).

The only function there seems to be:

qibojit/src/qibojit/backends/gpu.py

Line 509 in 0050f99

self.cusv.destroy(self.handle)

(the line could be different according to the version, but the __del__ function has only two lines, and it's unchanged for 10 months)

Could you provide more details to reproduce the issue?
I tried to manually call custatevec.destroy() from the interpreter, and it's definitely not None.
Which version of cuquantum are you using? How did you install it?

[scarrazza]

Any progress/debug on this issue?

[andrea-pasquale]

Any progress/debug on this issue?

Not yet, I still need to have a look to see if I can reproduce the issue.

[leofang]

Q: Is this issue only seen during the Python interpreter shutdown?

[renatomello]

Q: Is this issue only seen during the Python interpreter shutdown?

IIRC yes, but it's been almost half a year since I last saw it so I'm not 100% sure.

[alecandido]

Exception ignored in: <function CuQuantumBackend.__del__

The part of the traceback reported is telling that is happening on del, and I believe there is no explicit removal of the backend, so it should not be deallocated before the completion of the process (i.e. when Python interpreter exits).

Qibo itself does not delete explicitly the backend, and the reference is lost only if the backend is replaced.

[andrea-pasquale]

I was also never able to reproduce the issue.
Just to give some context that line was introduced because according to cuquantum examples after each execution the handle is manually destroyed
https://github.com/NVIDIA/cuQuantum/blob/2ac5645a56063d4d83b1c82f833ccdaa75b3736f/python/samples/custatevec/gate_application.py#L59
More practically by running qibo CI tests locally on cuquantum you can verify that without that line the GPU will be full very soon making tests fail. Perhaps we could fix from our side qibo tests to delete the backend properly after each test.