CUDA_ERROR_OUT_OF_MEMORY after Anaconda reinstallation

[BeckyB24]

Hello,
I was wondering if anyone solved a similar issue. I had to reinstall Anaconda to fix an issue with another program. This required that I reinstall SLEAP. However, now anytime I try and run predictions on a video with a model that previously worked I get the error CUDA_ERROR_OUT_OF_MEMORY. I have tried installing SLEAP via the various recommended methods and went through the recommended steps to troubleshoot common issues (i.e. restarting, updating drivers etc.) Are there are any recommended steps I could try to get these previously working projects up and running again?

Thanks!

[talmo]

Hi @BeckyB24,

Do you know what version of SLEAP you were on before reinstalling?

Are you getting the higher memory usage on v1.3.3 or v1.4.1a1?

Finally, do you mind sharing the training_job_config.json from the model you're trying to use?

Thanks!

Talmo

[BeckyB24]

Thanks for such a quick response! I was mainly focusing on installing v1.3.3. With v1.4.1a1, I tried creating a new model as my existing one didn't appear compatible. Even with creating new models, v1.3.3 appears to get further in the prediction process before I get the CUDA out of memory error. Below I have included both the centroid and centered instance training config files that I had previously used in my older version for multi-animal label predictions.
Thanks so much!
training_config.json
training_config.json