CUDA 12.0, linking problem

[kylophone]

New machine with NVIDIA L4. I'm using cuda 12.0 from the public Ubuntu package. Meson is not finding -lcuda, the other machine I've been using has cuda 11.0. @gedoensmax wondering if this sounds familiar to you in any way? If not I can spend some time trying to figure it out.

FAILED: test/test_picture 
c++  -o test/test_picture test/test_picture.p/test.c.o test/test_picture.p/test_picture.c.o test/test_picture.p/.._src_picture.c.o test/test_picture.p/.._src_mem.c.o test/test_picture.p/.._src_ref.c.o test/test_picture.p/.._src_thread_pool.c.o -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -pthread -Wl,--start-group -lcuda /usr/local/cuda/lib64/libcudart.so -Wl,--end-group -L/usr/local/cuda/lib64
/usr/bin/ld: cannot find -lcuda: No such file or directory

$ dpkg -l | grep cuda

ii  cuda                                       12.0.1-1                                amd64        CUDA meta-package
ii  cuda-12-0                                  12.0.1-1                                amd64        CUDA 12.0 meta-package
ii  cuda-cccl-12-0                             12.0.140-1                              amd64        CUDA CCCL
ii  cuda-command-line-tools-12-0               12.0.1-1                                amd64        CUDA command-line tools
ii  cuda-compiler-12-0                         12.0.1-1                                amd64        CUDA compiler
ii  cuda-cudart-12-0                           12.0.146-1                              amd64        CUDA Runtime native Libraries
ii  cuda-cudart-dev-12-0                       12.0.146-1                              amd64        CUDA Runtime native dev links, headers
ii  cuda-cuobjdump-12-0                        12.0.140-1                              amd64        CUDA cuobjdump
ii  cuda-cupti-12-0                            12.0.146-1                              amd64        CUDA profiling tools runtime libs.
ii  cuda-cupti-dev-12-0                        12.0.146-1                              amd64        CUDA profiling tools interface.
ii  cuda-cuxxfilt-12-0                         12.0.140-1                              amd64        CUDA cuxxfilt
ii  cuda-demo-suite-12-0                       12.0.140-1                              amd64        Demo suite for CUDA
ii  cuda-documentation-12-0                    12.0.140-1                              amd64        CUDA documentation
ii  cuda-driver-dev-12-0                       12.0.146-1                              amd64        CUDA Driver native dev stub library
ii  cuda-drivers                               525.85.12-1                             amd64        CUDA Driver meta-package, branch-agnostic
ii  cuda-drivers-525                           525.85.12-1                             amd64        CUDA Driver meta-package, branch-specific
ii  cuda-gdb-12-0                              12.0.140-1                              amd64        CUDA-GDB
ii  cuda-libraries-12-0                        12.0.1-1                                amd64        CUDA Libraries 12.0 meta-package
ii  cuda-libraries-dev-12-0                    12.0.1-1                                amd64        CUDA Libraries 12.0 development meta-package
ii  cuda-nsight-12-0                           12.0.140-1                              amd64        CUDA nsight
ii  cuda-nsight-compute-12-0                   12.0.1-1                                amd64        NVIDIA Nsight Compute
ii  cuda-nsight-systems-12-0                   12.0.1-1                                amd64        NVIDIA Nsight Systems
ii  cuda-nvcc-12-0                             12.0.140-1                              amd64        CUDA nvcc
ii  cuda-nvdisasm-12-0                         12.0.140-1                              amd64        CUDA disassembler
ii  cuda-nvml-dev-12-0                         12.0.140-1                              amd64        NVML native dev links, headers
ii  cuda-nvprof-12-0                           12.0.146-1                              amd64        CUDA Profiler tools
ii  cuda-nvprune-12-0                          12.0.140-1                              amd64        CUDA nvprune
ii  cuda-nvrtc-12-0                            12.0.140-1                              amd64        NVRTC native runtime libraries
ii  cuda-nvrtc-dev-12-0                        12.0.140-1                              amd64        NVRTC native dev links, headers
ii  cuda-nvtx-12-0                             12.0.140-1                              amd64        NVIDIA Tools Extension
ii  cuda-nvvp-12-0                             12.0.146-1                              amd64        CUDA Profiler tools
ii  cuda-opencl-12-0                           12.0.140-1                              amd64        CUDA OpenCL native Libraries
ii  cuda-opencl-dev-12-0                       12.0.140-1                              amd64        CUDA OpenCL native dev links, headers
ii  cuda-profiler-api-12-0                     12.0.140-1                              amd64        CUDA Profiler API
ii  cuda-repo-ubuntu2204-12-0-local            12.0.1-525.85.12-1                      amd64        cuda repository configuration files
ii  cuda-runtime-12-0                          12.0.1-1                                amd64        CUDA Runtime 12.0 meta-package
ii  cuda-sanitizer-12-0                        12.0.140-1                              amd64        CUDA Sanitizer
ii  cuda-toolkit-12-0                          12.0.1-1                                amd64        CUDA Toolkit 12.0 meta-package
ii  cuda-toolkit-12-0-config-common            12.0.146-1                              all          Common config package for CUDA Toolkit 12.0.
ii  cuda-toolkit-12-config-common              12.0.146-1                              all          Common config package for CUDA Toolkit 12.
ii  cuda-toolkit-config-common                 12.0.146-1                              all          Common config package for CUDA Toolkit.
ii  cuda-tools-12-0                            12.0.1-1                                amd64        CUDA Tools meta-package
ii  cuda-visual-tools-12-0                     12.0.1-1                                amd64        CUDA visual tools

$ find /usr/local/cuda/ | grep libcuda

/usr/local/cuda/targets/x86_64-linux/lib/cmake/libcudacxx
/usr/local/cuda/targets/x86_64-linux/lib/cmake/libcudacxx/libcudacxx-header-search.cmake
/usr/local/cuda/targets/x86_64-linux/lib/cmake/libcudacxx/libcudacxx-config.cmake
/usr/local/cuda/targets/x86_64-linux/lib/cmake/libcudacxx/libcudacxx-config-version.cmake
/usr/local/cuda/targets/x86_64-linux/lib/libcudart.so
/usr/local/cuda/targets/x86_64-linux/lib/libcudart_static.a
/usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.12.0.146
/usr/local/cuda/targets/x86_64-linux/lib/libcudadevrt.a
/usr/local/cuda/targets/x86_64-linux/lib/stubs/libcuda.so
/usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.12
/usr/local/cuda/extras/Debugger/include/libcudacore.h
/usr/local/cuda/extras/Debugger/lib64/libcudacore.a

[gedoensmax]

Hey could you do me favor and paste the output of ldconfig -p | grep cuda ? Also please check if nvcc is on the PATH.

[gedoensmax]

Oh also the library you are looking for would be /usr/local/cuda/targets/x86_64-linux/lib/stubs/libcuda.so which is only a stub. The actual library is installed with the driver so also check if nvidia-smi is working as expected.

[kylophone]

$ nvidia-smi

Mon Mar 27 11:24:02 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA L4           On   | 00000000:C1:00.0 Off |                    0 |
| N/A   28C    P8    16W /  75W |      4MiB / 23034MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2272      G   /usr/lib/xorg/Xorg                  4MiB |
+-----------------------------------------------------------------------------+

$ ldconfig -p | grep cuda

        libpcsamplingutil.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libpcsamplingutil.so
        libnvrtc.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvrtc.so.12
        libnvrtc.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvrtc.so
        libnvrtc-builtins.so.12.0 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvrtc-builtins.so.12.0
        libnvrtc-builtins.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvrtc-builtins.so
        libnvperf_target.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvperf_target.so
        libnvperf_host.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvperf_host.so
        libnvjpeg.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvjpeg.so.12
        libnvjpeg.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvjpeg.so
        libnvblas.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvblas.so.12
        libnvblas.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvblas.so
        libnvToolsExt.so.1 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvToolsExt.so.1
        libnvToolsExt.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvToolsExt.so
        libnvJitLink.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvJitLink.so.12
        libnvJitLink.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnvJitLink.so
        libnpps.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnpps.so.12
        libnpps.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnpps.so
        libnppitc.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppitc.so.12
        libnppitc.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppitc.so
        libnppisu.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppisu.so.12
        libnppisu.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppisu.so
        libnppist.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppist.so.12
        libnppist.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppist.so
        libnppim.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppim.so.12
        libnppim.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppim.so
        libnppig.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppig.so.12
        libnppig.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppig.so
        libnppif.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppif.so.12
        libnppif.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppif.so
        libnppidei.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppidei.so.12
        libnppidei.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppidei.so
        libnppicc.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppicc.so.12
        libnppicc.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppicc.so
        libnppial.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppial.so.12
        libnppial.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppial.so
        libnppc.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppc.so.12
        libnppc.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libnppc.so
        libicudata.so.70 (libc6,x86-64) => /lib/x86_64-linux-gnu/libicudata.so.70
        libicudata.so.70 (ELF) => /lib/i386-linux-gnu/libicudata.so.70
        libicudata.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libicudata.so
        libcusparse.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcusparse.so.12
        libcusparse.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcusparse.so
        libcusolverMg.so.11 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcusolverMg.so.11
        libcusolverMg.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcusolverMg.so
        libcusolver.so.11 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcusolver.so.11
        libcusolver.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcusolver.so
        libcurand.so.10 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcurand.so.10
        libcurand.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcurand.so
        libcupti.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcupti.so.12
        libcupti.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcupti.so
        libcuinj64.so.12.0 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcuinj64.so.12.0
        libcuinj64.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcuinj64.so
        libcufile_rdma.so.1 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcufile_rdma.so.1
        libcufile_rdma.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcufile_rdma.so
        libcufile.so.0 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcufile.so.0
        libcufile.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcufile.so
        libcufftw.so.11 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcufftw.so.11
        libcufftw.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcufftw.so
        libcufft.so.11 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcufft.so.11
        libcufft.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcufft.so
        libcudart.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.12
        libcudart.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcudart.so
        libcudadebugger.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcudadebugger.so.1
        libcuda.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcuda.so.1
        libcuda.so.1 (libc6) => /lib/i386-linux-gnu/libcuda.so.1
        libcuda.so (libc6) => /lib/i386-linux-gnu/libcuda.so
        libcublasLt.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcublasLt.so.12
        libcublasLt.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcublasLt.so
        libcublas.so.12 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcublas.so.12
        libcublas.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcublas.so
        libcheckpoint.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcheckpoint.so
        libaccinj64.so.12.0 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libaccinj64.so.12.0
        libaccinj64.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libaccinj64.so
        libOpenCL.so.1 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libOpenCL.so.1
        libOpenCL.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libOpenCL.so

[gedoensmax]

This is interesting /lib/x86_64-linux-gnu/ should be automatically linked due to ldconfig even though it is not explicitly used with -L. Below does the job for me. So not really sure what is happening on your system.

FROM nvidia/cuda:12.1.0-devel-ubuntu20.04
ENV PATH=/usr/local/cuda/bin:$PATH
RUN DEBIAN_FRONTEND=noninteractive apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y libopenjp2-7-dev \
    ninja-build cmake git python3 python3-pip nasm xxd pkg-config curl

RUN python3 -m pip install meson cpython pandas matplotlib

RUN mkdir /code
WORKDIR /code
COPY build.sh /code/build.sh

build.sh:

#!/bin/bash
git clone https://github.com/gedoensmax/vmaf.git
cd vmaf
git checkout psnr
meson libvmaf/build libvmaf -Denable_cuda=true -Denable_avx512=true --buildtype release 
ninja -vC libvmaf/build 
ninja -vC libvmaf/build  install

=> docker build -t vmaf . && docker run -it --rm --gpus all vmaf bash ./build.sh

EDIT: Actually that might be something i should submit as a documentation PR 🤔

[vibhoothi]

Hi @kylophone,

How did you manage to fix this problem,

I am facing exactly same problem on test/test_picture,

$ ninja 
[1/25] Linking target test/test_picture
FAILED: test/test_picture 
c++  -o test/test_picture test/test_picture.p/test.c.o test/test_picture.p/test_picture.c.o test/test_picture.p/.._src_picture.c.o test/test_picture.p/.._src_mem.c.o test/test_picture.p/.._src_ref.c.o test/test_picture.p/.._src_thread_pool.c.o -Wl,--as-needed -Wl,--no-undefined -Wl,-O1 -pthread -Wl,--start-group -lcuda /usr/local/cuda/lib64/libcudart.so -Wl,--end-group -L/usr/local/cuda/lib64
/usr/bin/ld: cannot find -lcuda: No such file or directory

Edit: FWIW CUDA version is 12.1

Edit2: The docker method was only method which worked for me, prolly due to some libs linkage problem in my pc or meson

[kylophone]

Yes, I think the cuda bit of the meson probably needs work. On one system, I had to manually add the following line to get build working. I think there is a lot of room for improvement here, let me know if you have any suggestions.

cuda_dependency += declare_dependency(link_args : ['-L/usr/local/cuda/lib64/stubs/'])