Code Hangs Indefinitely When Evaluating Neuron Models on GPU

[bamsumit]

This is a known PyTorch issue that many of our users have experienced. Here are the steps to resolve it.

How to identify?

The tell tell sign of this issue is the code hangs indefinitely in baton.wait() command of torch/utils/cpp_extension.py while doing just in time compilation of a cuda code.

Why it happens?

This seems to happen if the python thread gets killed for some reason during cpp/cuda code compilation and PyTorch is unable to clear the lock file on cpp extension.

How to fix?

Follow the fix here: zhou13/neurvps#1 (comment)

[mibaa-ai]

at the last stage, it throws an error when installing packages.

      copying pytorch3d\datasets\shapenet\shapenet_synset_dict_v2.json -> build\lib.win-amd64-cpython-311\pytorch3d\datasets\shapenet
      running build_ext
      C:\Users\HOME-PC\ComfyUI_windows_portable\python_miniconda_env\Lib\site-packages\torch\utils\cpp_extension.py:381: UserWarning: Error checking compiler version for cl: [WinError 2] Не удается найти указанный файл
        warnings.warn(f'Error checking compiler version for {compiler}: {error}')
      error: [WinError 2] Не удается найти указанный файл
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pytorch3d
  Running setup.py clean for pytorch3d
  Building wheel for torch-scatter (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [33 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build\lib.win-amd64-cpython-311
      creating build\lib.win-amd64-cpython-311\torch_scatter
      copying torch_scatter\placeholder.py -> build\lib.win-amd64-cpython-311\torch_scatter
      copying torch_scatter\scatter.py -> build\lib.win-amd64-cpython-311\torch_scatter
      copying torch_scatter\segment_coo.py -> build\lib.win-amd64-cpython-311\torch_scatter
      copying torch_scatter\segment_csr.py -> build\lib.win-amd64-cpython-311\torch_scatter
      copying torch_scatter\testing.py -> build\lib.win-amd64-cpython-311\torch_scatter
      copying torch_scatter\utils.py -> build\lib.win-amd64-cpython-311\torch_scatter
      copying torch_scatter\__init__.py -> build\lib.win-amd64-cpython-311\torch_scatter
      creating build\lib.win-amd64-cpython-311\torch_scatter\composite
      copying torch_scatter\composite\logsumexp.py -> build\lib.win-amd64-cpython-311\torch_scatter\composite
      copying torch_scatter\composite\softmax.py -> build\lib.win-amd64-cpython-311\torch_scatter\composite
      copying torch_scatter\composite\std.py -> build\lib.win-amd64-cpython-311\torch_scatter\composite
      copying torch_scatter\composite\__init__.py -> build\lib.win-amd64-cpython-311\torch_scatter\composite
      running egg_info
      creating torch_scatter.egg-info
      writing torch_scatter.egg-info\PKG-INFO
      writing dependency_links to torch_scatter.egg-info\dependency_links.txt
      writing requirements to torch_scatter.egg-info\requires.txt
      writing top-level names to torch_scatter.egg-info\top_level.txt
      writing manifest file 'torch_scatter.egg-info\SOURCES.txt'
      reading manifest file 'torch_scatter.egg-info\SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      adding license file 'LICENSE'
      writing manifest file 'torch_scatter.egg-info\SOURCES.txt'
      running build_ext
      C:\Users\HOME-PC\ComfyUI_windows_portable\python_miniconda_env\Lib\site-packages\torch\utils\cpp_extension.py:381: UserWarning: Error checking compiler version for cl: [WinError 2] Не удается найти указанный файл
        warnings.warn(f'Error checking compiler version for {compiler}: {error}')
      error: [WinError 2] Не удается найти указанный файл
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for torch-scatter
  Running setup.py clean for torch-scatter
  Building wheel for kiui (setup.py) ... done
  Created wheel for kiui: filename=kiui-0.2.4-py3-none-any.whl size=1190200 sha256=946bcc4e6c3c2c17e4eaf91591c53a0d6378b46b8a3268f1cfa67322217441c5
  Stored in directory: C:\Users\HOME-PC\AppData\Local\Temp\pip-ephem-wheel-cache-7fqmgog4\wheels\20\3c\17\e791a15a84c4c4e2281c95331ff68b27a12f93ce8aa4c0aaf9
Successfully built nvdiffrast kiui
Failed to build pointnet2-ops simple-knn diff-gaussian-rasterization pytorch3d torch-scatter
ERROR: Could not build wheels for pointnet2-ops, simple-knn, diff-gaussian-rasterization, pytorch3d, torch-scatter, which is required to install pyproject.toml-based projects
Install Finished. Press any key to continue...