[BUG] When using rmm_cupy_allocator with Cupy, the memory resource is constructed twice.

[harrism]

Describe the bug
I added print statements to the ctor and dtor in the C++ cuda_memory_resource and then set Cupy to use RMM's allocator. The constructor print statement is called twice:

Steps/Code to reproduce bug
Add print statements to the constructor and destructor (and optionally allocate/deallocate functions) of cuda_memory_resource. Then run this code:

>>> import rmm
>>> from rmm.allocators.cupy import rmm_cupy_allocator
>>> import cupy as cp
>>> cp.cuda.set_allocator(rmm_cupy_allocator)
>>> array = cp.ndarray(1000)
cuda_memory_resource constructor
do_allocate
cuda_memory_resource constructor
>>> del array
do_deallocate

Notice cuda_memory_resource constructor prints twice, showing that the C++ object is constructed twice.

Expected behavior
cuda_memory_resource constructor should only print once (I think).

Environment details (please complete the following information):

• Environment location: [Docker / dev container]
• Method of RMM install: [dev container]

[harrism]

I can't see immediately why this would happen. I believe this just goes to DeviceBuffer, which goes to c++ device_buffer, which calls get_current_device_resource() in its ctor. This function calls initial_resource() if set_current_device_resource() was never called. And initial_resource constructs the cuda_memory_resource() the first time it is called (stored in a function local static). That shouldn't be called twice and even if it were the static value would be returned.

[shwina]

This ultimately comes down to Python creating a cuda_memory_resource (first), and then calling set_per_device_resource which itself constructs a cuda_memory_resource()(second).

I think it's harmless as long as the initial_resource() is a cuda_memory_resource and as long as cuda_memory_resource objects are cheap to create.

[wence-]

This is a consequence of a confluence of factors:

1. DeviceBuffer on the Python side doesn't accept an mr parameter and so uses get_current_device_resource to obtain an mr.
2. get_current_device_resource in Python cannot call get_current_device_resource from C++ (because that hands out raw pointers which can't be lifetime managed)
3. We don't pass an mr into the allocation of the C++ device_buffer that is wrapped inside DeviceBuffer so that then calls the C++ get_current_device_resource.

This is only harmless because get_current_device_resource (without a set_current_device_resource first) returns a cuda_memory_resource, and all cuda_memory_resources are interchangeable. If it returned a more complicated default resource, then we would be out of luck (since we'd end up allocating a device buffer with resource-A and attempting to free it with resource-B). This is another example of #1492 raising its head.

#1514 is a narrow fix for this issue, but we need to go through all the Cython interfaces and make sure we are explicit about tracking memory resources.