DepthMap fails - Cannot get available memory information for CUDA gpu device 0 #594

RafaelLinux · 2019-08-14T20:38:35Z

I just download and execute Meshroom in OpenSUSE 15.1 with the propietary Nvidia drivers. However, process always fails in "DepthMap" module. Please, could you help me?

This is the log:

`Program called with the following parameters:

downscale = 2
exportIntermediateResults = 0
imagesFolder = "/tmp/MeshroomCache/PrepareDenseScene/b7f2988d7aeb01eaef9ba71381ea55dbcd34d57d"
input = "/tmp/MeshroomCache/StructureFromMotion/b37bea3c5876c5aec6ddb824486ecb76c7ca9ad9/sfm.abc"
maxViewAngle = 70
minViewAngle = 2
nbGPUs = 0
output = "/tmp/MeshroomCache/DepthMap/dff0c8d68b092f5650ef612ce38629be6708b520"
rangeSize = 3
rangeStart = 0
refineGammaC = 15.5
refineGammaP = 8
refineMaxTCams = 6
refineNDepthsToRefine = 31
refineNSamplesHalf = 150
refineNiters = 100
refineSigma = 15
refineUseTcOrRcPixSize = 0
refineWSH = 3
sgmGammaC = 5.5
sgmGammaP = 8
sgmMaxTCams = 10
sgmWSH = 4
verboseLevel = "info"

[22:31:22.795596][warning] Cannot get available memory information for CUDA gpu device 0.
[22:31:22.708762][warning] CUDA-Enabled GPU.
Device information:
- id: 0
- name: GeForce GTX 1050 Ti
- compute capability: 6.1
- total device memory: 4038 MB
- device memory available: 0 MB
- per-block shared memory: 49152
- warp size: 32
- max threads per block: 1024
- max threads per SM(X): 2048
- max block sizes: {1024,1024,64}
- max grid sizes: {2147483647,65535,65535}
- max 2D array texture: {131072,65536}
- max 3D array texture: {16384,16384,16384}
- max 2D linear texture: {131072,65000,2097120}
- max 2D layered texture: {32768,32768,2048}
- number of SM(x)s: 6
- registers per SM(x): 65536
- registers per block: 65536
- concurrent kernels: yes
- mapping host memory: yes
- unified addressing: yes
- texture alignment: 512 byte
- pitch alignment: 32 byte

[22:31:22.796074][info] Supported CUDA-Enabled GPU detected.
[22:31:22.813640][info] Found 1 image dimension(s):
[22:31:22.813684][info] - [1024x1536]
[22:31:22.936193][info] Overall maximum dimension: [1024x1536]
[22:31:22.936298][info] Create depth maps.

CUDAError: unknown error
file: /opt/AliceVision_git/src/aliceVision/depthMap/cuda/planeSweeping/plane_sweeping_cuda.cu
function: ps_listCUDADevices
line: 205

[22:31:22.936615][info] # GPU devices: 1, # CPU threads: 4
[22:31:22.936649][info] Plane sweeping parameters:
- scale: 2
- step: 2
[22:31:22.936690][info] PlaneSweepingCuda:
- _nImgsInGPUAtTime: 13
- scales: 2
- subPixel: Yes
- varianceWSH: 4
terminate called after throwing an instance of 'std::runtime_error'
what(): Device alloc 2D array failed: unknown error
`

fabiencastan · 2019-08-15T09:19:36Z

The problem comes from here: "Cannot get available memory information for CUDA gpu device 0."

Do you have another software using the GPU at the same time?
Do you have this problem only with the release 2019.2 or did you already have the same problem with previous releases?

RafaelLinux · 2019-08-15T12:57:18Z

Well, I tried again just after restarting OpenSUSE, to avoid any game or graphical program use CUDA. In fact, CUDA doesn't seem to be installed. Latest version of Nvidia drivers doesn't show any CUDA related package. However, I can use the GPU in Blender without problems. I did the test again, and same result.
Is first time I try Meshroom, so I don't know if previous versions worked in OpenSUSE.

RafaelLinux · 2019-08-16T10:03:04Z

If it help to solve this problem, I tested same version on another PC with OpenSUSE but Intel GPU. This time, clearly Meshroom detect that is not a CUDA capabled GPU:

`Program called with the following parameters:

downscale = 2
exportIntermediateResults = 0
imagesFolder = "/home/PALACIO/1400/SpiderOak Hive/3D/Meshroom/MeshroomCache/PrepareDenseScene/b7f2988d7aeb01eaef9ba71381ea55dbcd34d57d"
input = "/home/PALACIO/1400/SpiderOak Hive/3D/Meshroom/MeshroomCache/StructureFromMotion/b37bea3c5876c5aec6ddb824486ecb76c7ca9ad9/sfm.abc"
maxViewAngle = 70
minViewAngle = 2
nbGPUs = 0
output = "/home/PALACIO/1400/SpiderOak Hive/3D/Meshroom/MeshroomCache/DepthMap/dff0c8d68b092f5650ef612ce38629be6708b520"
rangeSize = 3
rangeStart = 0
refineGammaC = 15.5
refineGammaP = 8
refineMaxTCams = 6
refineNDepthsToRefine = 31
refineNSamplesHalf = 150
refineNiters = 100
refineSigma = 15
refineUseTcOrRcPixSize = 0
refineWSH = 3
sgmGammaC = 5.5
sgmGammaP = 8
sgmMaxTCams = 10
sgmWSH = 4
verboseLevel = "info"

[11:50:18.603172][warning] Could not determine number of CUDA cards in this system
[11:50:18.602765][warning] No CUDA-Enabled GPU.
[11:50:18.604628][error] cudaGetDeviceCount failed: CUDA driver version is insufficient for CUDA runtime version
[11:50:18.604647][info] Can't find CUDA-Enabled GPU.
[11:50:18.604655][error] This program needs a CUDA-Enabled GPU (with at least compute capablility 2.0).
`

natowi · 2019-08-16T11:01:02Z

You have to make sure the NVIDIA driver with CUDA is correctly installed on OpenSUSE.

RafaelLinux · 2019-08-16T11:38:37Z

As I wrote, there is not (un)official CUDA packages for OpenSUSE 15.1 from Nvidia, don't know the reasons, but they are not available yet.

natowi · 2019-08-16T11:46:22Z

You could try installing the nVidia CUDA Toolkit

RafaelLinux · 2019-08-16T13:55:16Z

I didn't know that way. I'll try tonight.
Thank you for your suggestion, natowi

RafaelLinux · 2019-08-17T01:25:32Z

No way. After installing the OpenSUSE version of Nvidia CUDA toolkit and reboot the system, nothing changed. Same result. Process stop in same point with same error. :(

natowi · 2019-08-17T08:46:35Z

Did you test your cuda installation? https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#verify-installation

meshuser786 · 2019-08-17T10:46:10Z

Got the same Error in Meshroom 2019.2.0.

but Meshroom 2018.1.0 work's fine on the same system
Ubnuntu Gtx1060

RafaelLinux · 2019-08-17T21:59:09Z

Did you test your cuda installation? https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#verify-installation

Well, the test proposed in that page are fucosed to a user compilation of CUDA. In the case of a installed CUDA toolkit, executables are not located in the path related in Nvidia page. Anyway, I got to locate where the executables are located in my system, and the result are passed:

/usr/local/cuda-10.1/extras/demo_suite/deviceQuery   
Detected 1 CUDA Capable device(s)

Device 0: "GeForce GTX 1050 Ti"
  CUDA Driver Version / Runtime Version          10.1 / 10.1
  CUDA Capability Major/Minor version number:    6.1
  Total amount of global memory:                 4039 MBytes (4234936320 bytes)
  ( 6) Multiprocessors, (128) CUDA Cores/MP:     768 CUDA Cores
  GPU Max Clock rate:                            1481 MHz (1.48 GHz)
  Memory Clock rate:                             3504 Mhz
  Memory Bus Width:                              128-bit
  L2 Cache Size:                                 1048576 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 2 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 10.1, CUDA Runtime Version = 10.1, NumDevs = 1, Device0 = GeForce GTX 1050 Ti
Result = PASS

Even bandwidthTest is ok:

/usr/local/cuda-10.1/extras/demo_suite/bandwidthTest
[CUDA Bandwidth Test] - Starting...
Running on...

 Device 0: GeForce GTX 1050 Ti
 Quick Mode

 Host to Device Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)        Bandwidth(MB/s)
   33554432                     6199.9

 Device to Host Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)        Bandwidth(MB/s)
   33554432                     6443.1

 Device to Device Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)        Bandwidth(MB/s)
   33554432                     72991.1

Result = PASS

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

IvNvLyS · 2019-08-19T18:43:02Z

Same here.
Linux Mint 19.2
Propietary Nvidia drivers
Cuda 10.1
`
[20:39:09.275438][warning] Cannot get available memory information for CUDA gpu device 0.
[20:39:09.049583][warning] CUDA-Enabled GPU.
Device information:
- id: 0
- name: GeForce GTX 1050
- compute capability: 6.1
- total device memory: 4040 MB
- device memory available: 0 MB
- per-block shared memory: 49152
- warp size: 32
- max threads per block: 1024
- max threads per SM(X): 2048

`

Meshroom-2018.1.0 works ok.

fabiencastan · 2019-08-23T14:27:41Z

I have updated the release binaires.
The binaries are now done again with cuda-7.0 (instead of 8.0) as in the previous releases and I have also changed some libraries versions.

This fixes the problem on the 2 computers on which I managed to reproduce the problem.

Let me know if it works better for you now.

meshuser786 · 2019-08-24T05:59:17Z

Thanks for me it works now.
Linux 5.0.0-25-generic #26~18.04.1-Ubuntu
Cuda compilation tools, release 9.1, V9.1.85
gtx 1060

RafaelLinux · 2019-08-24T17:37:02Z

The update solves the bug. Now I can at least try it, but I hope in the future Meshroom doesn't require CUDA ;)

Thank you

Baasje85 · 2019-08-24T18:54:12Z

The binaries are now done again with cuda-7.0 (instead of 8.0) as in the previous releases and I have also changed some libraries versions.

This fixes the problem on the 2 computers on which I managed to reproduce the problem.

@fabiencastan Did you also change anything in AliceVision to resolve the bug? Hence if I would compile it with a version later than 7.0, would the error still be present?

miegl · 2019-08-25T10:09:46Z

Confirming new binaries work fine, but it would be nice to fix compiling with a never cuda version.

fabiencastan · 2019-08-25T10:24:35Z

@Baasje85: I used an older commit in cctag. The CMakeList.txt in AliceVision now uses an explicit commit for CCTag.

@miegl:
Yes, we need to continue the investigation to understand why cuda-8.0 creates compatibility problems.
Any inputs are welcome.

nmz787 · 2019-09-01T19:49:29Z

I have CUDA 8.0 installed (didn't compile myself)... does this mean I need to downgrade to 7.0 ?? I was following another issue on this topic where suggestions said to UPGRADE drivers

norru · 2019-09-03T10:41:10Z

Uh? A new set of binaries have been released with the same version number as the old binaries? This is bound to be hell for someone in the future. (I have been bitten by this type of problem quite a few times in the past. It's not fun to troubleshoot.)

Here's the sha256 checksum for future reference (I know I'll need it).

Broken version  25165efdef790923de1aac2bb775d8ddea215a85704556c0f743fd3c19397ca8
Correct version bdd741692161c234d27b3a8ce7109071c99bbe6c30b04d8108675a19f160a93a

mokizzz · 2020-12-22T06:51:16Z

If it help to solve this problem, I tested same version on another PC with OpenSUSE but Intel GPU. This time, clearly Meshroom detect that is not a CUDA capabled GPU:

`Program called with the following parameters:

downscale = 2

exportIntermediateResults = 0

imagesFolder = "/home/PALACIO/1400/SpiderOak Hive/3D/Meshroom/MeshroomCache/PrepareDenseScene/b7f2988d7aeb01eaef9ba71381ea55dbcd34d57d"

input = "/home/PALACIO/1400/SpiderOak Hive/3D/Meshroom/MeshroomCache/StructureFromMotion/b37bea3c5876c5aec6ddb824486ecb76c7ca9ad9/sfm.abc"

maxViewAngle = 70

minViewAngle = 2

nbGPUs = 0

output = "/home/PALACIO/1400/SpiderOak Hive/3D/Meshroom/MeshroomCache/DepthMap/dff0c8d68b092f5650ef612ce38629be6708b520"

rangeSize = 3

rangeStart = 0

refineGammaC = 15.5

refineGammaP = 8

refineMaxTCams = 6

refineNDepthsToRefine = 31

refineNSamplesHalf = 150

refineNiters = 100

refineSigma = 15

refineUseTcOrRcPixSize = 0

refineWSH = 3

sgmGammaC = 5.5

sgmGammaP = 8

sgmMaxTCams = 10

sgmWSH = 4

verboseLevel = "info"

[11:50:18.603172][warning] Could not determine number of CUDA cards in this system
[11:50:18.602765][warning] No CUDA-Enabled GPU.
[11:50:18.604628][error] cudaGetDeviceCount failed: CUDA driver version is insufficient for CUDA runtime version
[11:50:18.604647][info] Can't find CUDA-Enabled GPU.
[11:50:18.604655][error] This program needs a CUDA-Enabled GPU (with at least compute capablility 2.0).
`

I solved this problem by updating Nvidia driver from 418 to 450 and CUDA from 10.1 to 11.0.
Ubuntu 18.04 Server, Meshroom 2020.1.0 Linux.

natowi added the linux label Aug 14, 2019

natowi added the CUDA label Aug 17, 2019

natowi mentioned this issue Aug 20, 2019

Bug: Cannot get available memory information for CUDA gpu device 0. alicevision/AliceVision#621

Closed

natowi changed the title ~~Error trying to process "RockScan" available demo~~ DepthMap fails - Cannot get available memory information for CUDA gpu device 0 Aug 20, 2019

fabiencastan added the prio:blocker label Aug 23, 2019

fabiencastan closed this as completed Aug 25, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DepthMap fails - Cannot get available memory information for CUDA gpu device 0 #594

DepthMap fails - Cannot get available memory information for CUDA gpu device 0 #594

RafaelLinux commented Aug 14, 2019

fabiencastan commented Aug 15, 2019

RafaelLinux commented Aug 15, 2019

RafaelLinux commented Aug 16, 2019

natowi commented Aug 16, 2019

RafaelLinux commented Aug 16, 2019

natowi commented Aug 16, 2019 •

edited

Loading

RafaelLinux commented Aug 16, 2019

RafaelLinux commented Aug 17, 2019

natowi commented Aug 17, 2019

meshuser786 commented Aug 17, 2019

RafaelLinux commented Aug 17, 2019

IvNvLyS commented Aug 19, 2019 •

edited

Loading

fabiencastan commented Aug 23, 2019 •

edited

Loading

meshuser786 commented Aug 24, 2019

RafaelLinux commented Aug 24, 2019

Baasje85 commented Aug 24, 2019

miegl commented Aug 25, 2019

fabiencastan commented Aug 25, 2019

nmz787 commented Sep 1, 2019

norru commented Sep 3, 2019 •

edited

Loading

mokizzz commented Dec 22, 2020

DepthMap fails - Cannot get available memory information for CUDA gpu device 0 #594

DepthMap fails - Cannot get available memory information for CUDA gpu device 0 #594

Comments

RafaelLinux commented Aug 14, 2019

fabiencastan commented Aug 15, 2019

RafaelLinux commented Aug 15, 2019

RafaelLinux commented Aug 16, 2019

natowi commented Aug 16, 2019

RafaelLinux commented Aug 16, 2019

natowi commented Aug 16, 2019 • edited Loading

RafaelLinux commented Aug 16, 2019

RafaelLinux commented Aug 17, 2019

natowi commented Aug 17, 2019

meshuser786 commented Aug 17, 2019

RafaelLinux commented Aug 17, 2019

IvNvLyS commented Aug 19, 2019 • edited Loading

fabiencastan commented Aug 23, 2019 • edited Loading

meshuser786 commented Aug 24, 2019

RafaelLinux commented Aug 24, 2019

Baasje85 commented Aug 24, 2019

miegl commented Aug 25, 2019

fabiencastan commented Aug 25, 2019

nmz787 commented Sep 1, 2019

norru commented Sep 3, 2019 • edited Loading

mokizzz commented Dec 22, 2020

natowi commented Aug 16, 2019 •

edited

Loading

IvNvLyS commented Aug 19, 2019 •

edited

Loading

fabiencastan commented Aug 23, 2019 •

edited

Loading

norru commented Sep 3, 2019 •

edited

Loading