You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
That and the other memory controls - but for my own issue I'm happy going for the blunt force approach to get torch to free up some of it's excessive allocations.
Bumping slightly as this is posing more of a problem. I'm probably going to resort to using https://crates.io/crates/nvml-wrapper in the short term to detect issues before they happen and otherwise try to find some time to look into how empty_cache would be implemented
In the tch and torch-sys crates there doesn't appear to be a version of https://pytorch.org/docs/stable/generated/torch.cuda.empty_cache.html#torch-cuda-empty-cache or
torch._C.cuda_emptyCache
which it called. I'll have a deeper look into this and PRing it but any guidance would be appreciated as this is a fairly important feature when sharing GPUs with other jobs.The text was updated successfully, but these errors were encountered: