Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

训练的时候出现问题:RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation #30

Closed
oneofforgettable opened this issue Jul 23, 2024 · 1 comment
Labels
question Further information is requested

Comments

@oneofforgettable
Copy link

Question

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 8, 1092, 1092]], which is output 0 of ReluBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

补充信息

Traceback (most recent call last):
File "main.py", line 222, in
train()
File "main.py", line 195, in train
train_one_epoch_acc(
File "/home/xx/DETR/Relation-DETR/util/engine.py", line 58, in train_one_epoch_acc
accelerator.backward(losses)
File "/home/xx/.conda/envs/rdetr/lib/python3.8/site-packages/accelerate/accelerator.py", line 2151, in backward
loss.backward(**kwargs)
File "/home/xx/.conda/envs/rdetr/lib/python3.8/site-packages/torch/_tensor.py", line 363, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/home/xx/.conda/envs/rdetr/lib/python3.8/site-packages/torch/autograd/init.py", line 173, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass

@oneofforgettable oneofforgettable added the question Further information is requested label Jul 23, 2024
@xiuqhou
Copy link
Owner

xiuqhou commented Jul 23, 2024

Hi,这个问题应该是Relation-DETR的bug,今天下午有其他研究者提出了相同的问题,链接在这里,我们下午修复了这个bug。您手头的代码可能不是最新版,请用最新版再试试看呢!

因为这不是本仓库Salience-DETR的问题,所以我先把issue关掉了哈,如果有问题可以在Relation-DETR的仓库开个issue~

@xiuqhou xiuqhou closed this as completed Jul 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants