Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] raft:cuda_error when calling cuml.metrics.pairwide_distances #4289

Closed
mayankanand007 opened this issue Oct 15, 2021 · 0 comments · Fixed by rapidsai/raft#365
Closed

[BUG] raft:cuda_error when calling cuml.metrics.pairwide_distances #4289

mayankanand007 opened this issue Oct 15, 2021 · 0 comments · Fixed by rapidsai/raft#365
Labels
bug Something isn't working

Comments

@mayankanand007
Copy link
Contributor

mayankanand007 commented Oct 15, 2021 •
edited
Loading

Describe the bug
The program crashes while running pairwise_distances on a CSR matrix with the error: terminate called after throwing an instance of raft::cuda_error
Stack trace:

terminate called after throwing an instance of 'raft::cuda_error'
  what():  CUDA error encountered at: file=/workspace/.conda-bld/cuml_1632479467059/work/python/_external_repositories/raft/cpp/include/raft/handle.hpp line=268: call='cudaEventDestroy(event_)', Reason=cudaErrorIllegalAddress:an illegal memory access was encountered
Obtained 27 stack frames
#0 in /raid/mayanka/miniconda3/envs/rapids-21.10/lib/python3.8/site-packages/cuml/raft/common/handle.cpython-38-x86_64-linux-gnu.so(_ZN4raft9exception18collect_call_stackEv+0x3b) [0x7fe7c81fde1b]
#1 in /raid/mayanka/miniconda3/envs/rapids-21.10/lib/python3.8/site-packages/cuml/raft/common/handle.cpython-38-x86_64-linux-gnu.so(_ZN4raft10cuda_errorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE+0x5a) [0x7fe7c82061ba]
#2 in /raid/mayanka/miniconda3/envs/rapids-21.10/lib/python3.8/site-packages/cuml/raft/common/handle.cpython-38-x86_64-linux-gnu.so(_ZN4raft8handle_t17destroy_resourcesEv+0x3fa) [0x7fe7c8207b4a]
#3 in /raid/mayanka/miniconda3/envs/rapids-21.10/lib/python3.8/site-packages/cuml/raft/common/handle.cpython-38-x86_64-linux-gnu.so(+0x2a60d) [0x7fe7c81f760d]
#4 in /raid/mayanka/miniconda3/envs/rapids-21.10/lib/python3.8/site-packages/cuml/metrics/pairwise_distances.cpython-38-x86_64-linux-gnu.so(+0x30fe8) [0x7fe7b737efe8]
#5 in /raid/mayanka/miniconda3/envs/rapids-21.10/bin/python(PyObject_Call+0x24d) [0x55732466711d]
#6 in /raid/mayanka/miniconda3/envs/rapids-21.10/bin/python(_PyEval_EvalFrameDefault+0x2113) [0x557324710b13]
#7 in /raid/mayanka/miniconda3/envs/rapids-21.10/bin/python(_PyEval_EvalCodeWithName+0x2c3) [0x5573246f0ee3]
#8 in /raid/mayanka/miniconda3/envs/rapids-21.10/bin/python(_PyFunction_FastCallDict+0x1b2) [0x55732460eaac]
#9 in /raid/mayanka/miniconda3/envs/rapids-21.10/lib/python3.8/site-packages/cuml/metrics/pairwise_distances.cpython-38-x86_64-linux-gnu.so(+0x25276) [0x7fe7b7373276]
#10 in /raid/mayanka/miniconda3/envs/rapids-21.10/lib/python3.8/site-packages/cuml/metrics/pairwise_distances.cpython-38-x86_64-linux-gnu.so(+0x285f1) [0x7fe7b73765f1]
#11 in /raid/mayanka/miniconda3/envs/rapids-21.10/bin/python(PyObject_Call+0x24d) [0x55732466711d]
#12 in /raid/mayanka/miniconda3/envs/rapids-21.10/bin/python(_PyEval_EvalFrameDefault+0x2113) [0x557324710b13]
#13 in /raid/mayanka/miniconda3/envs/rapids-21.10/bin/python(_PyEval_EvalCodeWithName+0x2c3) [0x5573246f0ee3]
#14 in /raid/mayanka/miniconda3/envs/rapids-21.10/bin/python(_PyFunction_Vectorcall+0x378) [0x5573246f22c8]
#15 in /raid/mayanka/miniconda3/envs/rapids-21.10/bin/python(_PyEval_EvalFrameDefault+0x178e) [0x55732471018e]
#16 in /raid/mayanka/miniconda3/envs/rapids-21.10/bin/python(_PyEval_EvalCodeWithName+0x2c3) [0x5573246f0ee3]
#17 in /raid/mayanka/miniconda3/envs/rapids-21.10/bin/python(PyEval_EvalCodeEx+0x39) [0x5573246f1f49]
#18 in /raid/mayanka/miniconda3/envs/rapids-21.10/bin/python(PyEval_EvalCode+0x1b) [0x557324796bbb]
#19 in /raid/mayanka/miniconda3/envs/rapids-21.10/bin/python(+0x254c53) [0x557324796c53]
#20 in /raid/mayanka/miniconda3/envs/rapids-21.10/bin/python(+0x26e353) [0x5573247b0353]
#21 in /raid/mayanka/miniconda3/envs/rapids-21.10/bin/python(+0x273402) [0x5573247b5402]
#22 in /raid/mayanka/miniconda3/envs/rapids-21.10/bin/python(PyRun_SimpleFileExFlags+0x1b2) [0x5573247b55e2]
#23 in /raid/mayanka/miniconda3/envs/rapids-21.10/bin/python(Py_RunMain+0x36d) [0x5573247b5b5d]
#24 in /raid/mayanka/miniconda3/envs/rapids-21.10/bin/python(Py_BytesMain+0x39) [0x5573247b5d19]
#25 in /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7fe90666ab97]
#26 in /raid/mayanka/miniconda3/envs/rapids-21.10/bin/python(+0x1e7459) [0x557324729459]

Aborted (core dumped)

Steps/Code to reproduce bug

from cuml.metrics import pairwise_distances
import cupy

new_matrix = cupy.sparse.random(53569, 1632, density=0.01, format='csr')
similarities = pairwise_distances(new_matrix, metric='cosine')

Expected behavior
Should not crash

Environment details (please complete the following information):

  • Linux Distro/Architecture: Ubuntu 18.04.3 LTS, Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz
  • GPU Model/Driver: [V100 and driver 460.32.03]
  • CUDA: [11.2]
  • Method of cuDF & cuML install: [conda]
conda create -n rapids-21.10 -c rapidsai-nightly -c nvidia -c conda-forge rapids=21.10 python=3.8 cudatoolkit=11.2 jupyterlab --force
@mayankanand007 mayankanand007 added bug Something isn't working ? - Needs Triage Need team to review and classify labels Oct 15, 2021
@cjnolet cjnolet removed the ? - Needs Triage Need team to review and classify label Oct 18, 2021
@cjnolet cjnolet mentioned this issue Oct 27, 2021
Merged
cjnolet added a commit to cjnolet/cuml that referenced this issue Oct 27, 2021
@cjnolet
Closes rapidsai#4289
d2de1a7
rapids-bot bot pushed a commit to rapidsai/raft that referenced this issue Oct 27, 2021
@cjnolet
Fixing overflow in expanded distances (#365)
091cfdf 
Closes rapidsai/cuml#4289

Authors:
  - Corey J. Nolet (https://github.com/cjnolet)

Approvers:
  - Dante Gama Dessavre (https://github.com/dantegd)

URL: #365
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[REVIEW] Fixing overflow in expanded distances cjnolet/raft
2 participants
@cjnolet @mayankanand007