Running DeepLabStream with SLEAP (deprecating to an old SLEAP version?)

[gracegliu]

Hello! I am attempting to run DeepLabStream/A-Real on top of SLEAP to produce real-time closed-loop optogenetic control. I have been able to use SLEAP versions 1.3.2 and 1.3.3 without incident on this machine.

DeepLabStream does not initialize at all when installed on top of the latest version of SLEAP (1.3.3) so I installed it in an environment with the version of SLEAP that was current when it was built in 2021 (SLEAP 1.1.5). However, while it does now initialize and run, it has very low FPS and does not use the GPU at all and I get a stream of Tensorflow errors as shown below, though the GPU does seem to be detected.

(dlstream) C:\Users\AhmariLab>python -c "import sleap; sleap.system_summary()"
2024-06-25 11:56:25.331959: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2024-06-25 11:56:25.332090: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2024-06-25 11:56:27.587071: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library nvcuda.dll
2024-06-25 11:56:27.610412: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2024-06-25 11:56:27.611781: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2024-06-25 11:56:27.613054: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cublas64_11.dll'; dlerror: cublas64_11.dll not found
2024-06-25 11:56:27.614347: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cublasLt64_11.dll'; dlerror: cublasLt64_11.dll not found
2024-06-25 11:56:27.615581: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cufft64_10.dll'; dlerror: cufft64_10.dll not found
2024-06-25 11:56:27.616826: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'curand64_10.dll'; dlerror: curand64_10.dll not found
2024-06-25 11:56:27.618049: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cusolver64_11.dll'; dlerror: cusolver64_11.dll not found
2024-06-25 11:56:27.619420: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cusparse64_11.dll'; dlerror: cusparse64_11.dll not found
2024-06-25 11:56:27.620756: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found
2024-06-25 11:56:27.620819: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1766] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.

I assume this is because the deprecated version of SLEAP uses a version of tensorflow that no longer works with my GPU drivers.

Has anyone else run into this issue or has any experience using SLEAP with a real-time/closed-loop manipulation software?

Is there a set of cuda files I can install so SLEAP 1.1.5 can work in this environment? I would also need it to be compatible with Python version 3.6.* because of DLS's version compatibility.

Thanks so much!

GPU: NVIDIA GeForce RTX 2080 Ti
Driver version: 551.68

[talmo]

Hi @gracegliu,

Apologies for the delay! We're a bit behind on support responses at the moment.

This indeed looks like an issue with detecting the GPU. This is a toughie since those versions are all quite old, but you can try to install the CUDA dependencies according to the environment specs on v1.1.5:

conda install cudatoolkit=10.1 cudnn=7.6

If that doesn't work, you can try installing those versions at the system level (not recommended since it might break other stuff on your computer).

It might also not be too bad to update DLS if you want to work with A-Real -- it shouldn't be too much work to get it working with newer versions of SLEAP (the API is pretty much the same, so you just have to deal with dependencies).

As an alternative, you can try out Bonsai-Rx as a solution for realtime inference.

Cheers,

Talmo

[gracegliu]

Hi Talmo,

Thanks for the reply, and sorry for the delay on my end as well--this does in fact work, and it works fine just installing it within the environment, as long as I'm careful about the correct versions of cuda/cudnn/tensorflow that's correct for the Python version.

I'm curious about Bonsai-RX though--it looks like it can do real-time inference, but it does not give behavioral estimation (ie, if grooming is occuring or not) -- is that correct?

Thanks,
Grace

[talmo]

That's correct, you'll need to do something with the tracking to determine whether grooming is happening. You can try bringing this up in the Bonsai Community channels to see if folks have any suggestions (perhaps even supporting something like A-Real?).

[gracegliu]

Hi Talmo,

Thanks for those links! I work closely with the Yttri lab and I believe they are interested in writing a Bonsai module for ASOiD/A-Real (the offline and real time versions of their behavior estimator). I think the potential issue is the need to convert the existing Python code to the Bonsai programming environment? I am not as informed on that aspect of the project.

Best,
Grace

Will you be at the JAX machine learning short course next month in person? I would love to continue this conversation.