Trouble using cuda device on RHEL 8

[ururk]

This is the error I'm getting:

Detected language 'en' with probability 1.000000
Traceback (most recent call last):
  File "/users/username/transcribey/faster_whisp.py", line 189, in <module>
    main()
  File "/users/username/transcribey/faster_whisp.py", line 108, in main
    for segment in segments:
  File "/users/username/env/lib64/python3.9/site-packages/faster_whisper/transcribe.py", line 324, in generate_segments
    encoder_output = self.encode(segment)
  File "/users/username/env/lib64/python3.9/site-packages/faster_whisper/transcribe.py", line 483, in encode
    return self.model.encode(features, to_cpu=to_cpu)
RuntimeError: cuDNN failed with status CUDNN_STATUS_NOT_INITIALIZED

Output from /usr/local/cuda/extras/demo_suite/deviceQuery:

Detected 1 CUDA Capable device(s)

Device 0: "NVIDIA RTX A2000 12GB"
  CUDA Driver Version / Runtime Version          11.4 / 11.4
  CUDA Capability Major/Minor version number:    8.6
  Total amount of global memory:                 12025 MBytes (12609519616 bytes)
  (26) Multiprocessors, (128) CUDA Cores/MP:     3328 CUDA Cores
  GPU Max Clock rate:                            1200 MHz (1.20 GHz)
  Memory Clock rate:                             6001 Mhz
  Memory Bus Width:                              192-bit
  L2 Cache Size:                                 3145728 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  1536
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 1 / 0

I'm a little unsure why this is happening. I installed CUDA 11.5 and cuDNN 8.6 on RHEL 8.

CPU works.

I run it by calling:

model = WhisperModel("medium", device="cuda", compute_type="int8", cpu_threads=0)

[guillaumekln]

How did you install cuDNN? Make sure you installed cuDNN 8.6 for CUDA 11 (I think there is also cuDNN 8.6 for CUDA 10.2 which will not work here).

This error could also happen when you are running out of memory. Is there another running process that could use too much RAM or VRAM?

When in doubt, you could also try running your code in the official RHEL 8 Docker image from NVIDIA (e.g. nvidia/cuda:11.5.2-cudnn8-runtime-ubi8). If it works here it means there is something wrong in your CUDA/cuDNN installation.

[ururk]

Thanks! That was it - I had installed cuDNN for Cuda 12. Installing the correct version fixed the issue.