CUDA out of memory due to huge memory allocation request #18767
Assignees
Labels
bug 🐞
Something isn't working
Comments
|
Thank you for the analysis. This is most likely an index/shape issue in the new onnx path and will need triage. It may be incomplete support for a negative index on an op that then gets multiplied through to what you are seeing. Let me raise it to the appropriate folks. |
|
I'm able to repro the error on On CPU compilation failure may/not be related to the GPU runtime failures |
|
When trying to get a min repro for the above issue, I found that there is another issue just before, with onnx.Reshape Turns out that onnx.Reshape with reproducer torch_onnx mlir : module {
func.func @tf2onnx(%arg0: !torch.vtensor<[?,?],si32>) -> !torch.vtensor<[?,?],si32> attributes {torch.onnx_meta.ir_version = 8 : si64, torch.onnx_meta.opset_version = 21 : si64, torch.onnx_meta.opset_versions = {ai.onnx.ml = 2 : si64}, torch.onnx_meta.producer_name = "tf2onnx", torch.onnx_meta.producer_version = "1.16.1 15c810"} {
%0 = torch.operator "onnx.Constant"() {torch.onnx.value = dense_resource<const_starts__778> : tensor<1xsi64>} : () -> !torch.vtensor<[1],si64>
%1 = torch.operator "onnx.Constant"() {torch.onnx.value = dense_resource<const_fold_opt__2625> : tensor<1xsi32>} : () -> !torch.vtensor<[1],si32>
%2 = torch.operator "onnx.Constant"() {torch.onnx.value = dense_resource<const_ends__1662> : tensor<1xsi64>} : () -> !torch.vtensor<[1],si64>
%3 = torch.operator "onnx.Constant"() {torch.onnx.value = dense_resource<const_axes__1378> : tensor<1xsi64>} : () -> !torch.vtensor<[1],si64>
%none = torch.constant.none
%4 = torch.operator "onnx.Shape"(%arg0) : (!torch.vtensor<[?,?],si32>) -> !torch.vtensor<[2],si64>
%5 = torch.operator "onnx.Cast"(%4) {torch.onnx.to = 6 : si64} : (!torch.vtensor<[2],si64>) -> !torch.vtensor<[2],si32>
%6 = torch.operator "onnx.Slice"(%5, %2, %0, %3) : (!torch.vtensor<[2],si32>, !torch.vtensor<[1],si64>, !torch.vtensor<[1],si64>, !torch.vtensor<[1],si64>) -> !torch.vtensor<[1],si32>
%7 = torch.operator "onnx.Concat"(%1, %6) {torch.onnx.axis = 0 : si64} : (!torch.vtensor<[1],si32>, !torch.vtensor<[1],si32>) -> !torch.vtensor<[2],si32>
%8 = torch.operator "onnx.Cast"(%7) {torch.onnx.to = 7 : si64} : (!torch.vtensor<[2],si32>) -> !torch.vtensor<[2],si64>
%9 = torch.operator "onnx.Reshape"(%arg0, %8) : (!torch.vtensor<[?,?],si32>, !torch.vtensor<[2],si64>) -> !torch.vtensor<[?,?],si32>
return %9 : !torch.vtensor<[?,?],si32>
}
}
{-#
dialect_resources: {
builtin: {
const_starts__778: "0x080000000200000000000000",
const_fold_opt__2625: "0x08000000FFFFFFFF",
const_ends__1662: "0x080000000100000000000000",
const_axes__1378: "0x080000000000000000000000"
}
}
#-}
with the command below the expected value of commands: iree-compile --iree-hal-target-backends=rocm --iree-hip-target=gfx90a --dump-compilation-phases-to=./model-phases-rocm/ model.torch.onnx.mlir -o model.vmfb > output.mlir 2>&1
iree-run-module --trace_execution=true --print_statistics=true --device=hip://<UUID> --module=model.vmfb --function=tf2onnx --input="1x4xsi32=1" > trace.txt 2>&1
Runtime Error: iree/runtime/src/iree/hal/buffer.c:591: OUT_OF_RANGE; attempted to access an address outside of the valid buffer range (offset=0, adjusted_length=18446744073709551600, end=18446744073709551599, buffer byte_length=16); invalid subspan of an existing buffer (source_offset=0, length=18446744073709551600); while invoking native function hal.buffer_view.create; while calling import;
[ 1] bytecode module.tf2onnx$async:1930 /home/pbarwari/NOD/SHARK-TestSuite/alt_e2eshark/test-run/mygpt4_trunc_Reshape_0/model.torch_onnx.mlir:14:5
[ 0] bytecode module.tf2onnx:62 /home/pbarwari/NOD/SHARK-TestSuite/alt_e2eshark/test-run/mygpt4_trunc_Reshape_0/model.torch_onnx.mlir:2:3; invoking function 'tf2onnx'From the onnx graph this is the computation we are looking at |
|
Can you please post the IR dump after all. For this model. |
will be lower to @PhaneeshB Since the input data size for torch.aten.reshape is ‘?x?’, we need to figure out a way to covert ‘?’ to a positive value/real shape size(materialize -1 in the IR). And since aten.reshape is decompose to aten.view op, we need to add/debug this sematic either in ConvertAtenViewOp/ConvertAtenViewOpStrict/ConvertAtenViewOpToReshape in TorchToLinalg/DataMovement.cpp. Here is the previous work on view op: Based on https://pytorch.org/docs/stable/generated/torch.reshape.html, a single dimension may be -1, in which case it’s inferred from the remaining dimensions and the number of elements in input. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened?
I compiled GPT and tried to run it using
iree-run-module. It errored with the following message:Before the error message there is the statistics dump:
And right before it, is the command that caused the error:
If I understand the statistics correctly, this is not caused by my GPU not having enough memory, as the allocation is for ~4GB, the peak GPU usage so far was ~466MB and my GPU has 32GB of memory. So I went digging.
I inserted a print statement at iree/runtime/src/iree/hal/drivers/cuda/memory_pools.c:234, inside the function
iree_hal_cuda_memory_pools_alloca(...)right before the call tocuMemAllocFromPoolAsync(...)(actually right before the wrapperIREE_CURESULT_TO_STATUS) to see how much memory the runtime tried to allocate (the allocation size was a 64-bit unsigned integer for my machine so i used the %zu format identifier).The amount printed was actually 18446744073709548544 bytes which explains the error. I traced the call chain back to the file
iree/runtime/src/iree/modules/hal/module.cand specifically to the shim definition at lines 1096-1122. In that function, the argument to the allocation command is cast to the HAL device size. So, I added 2 print statements to see what the value of the argument was originally and what the value of the allocation size is, after the cast.The size before the cast was -3072 (though compiler errors, I determined the right format identifier for the
args->i7variable). For int32_t, 4294964224 (which is the supposed allocation size, see trace) overflows to -3072 so I suspect an overflow is the cause of all this.Thank you in advance.
Steps to reproduce your issue
Download the GPT onnx model
Either directly download it from https://drive.google.com/file/d/1w-TgnDylg43YUOQtffjo1SeRTCF_h7lE/view?usp=sharing
Or:
Export the pretrained huggingface TF model from
openai-community/openai-gpt(view gist)Convert it to ONNX using the tf2onnx package and the command
python -m tf2onnx.convert --saved-model ./gpt-tf/ --output model.onnx --opset 17**(
pip install transformers tensorflow tf-keras tf2onnx)Import it with the command
iree-import-onnx model.onnx -o model.mlir(**pip install iree-compiler[onnx])Compile it with the command
iree-compile --iree-hal-target-backends=cuda --iree-cuda-target=sm_70 --dump-compilation-phases-to=./model-phases/ --iree-vm-target-index-bits=64 --iree-stream-resource-index-bits=64 model.mlir -o model.vmfb > output.mlir 2>&1Run the compiled module with the command
iree-run-module --trace_execution=true --print_statistics=true --device=cuda --module=model.vmfb --function=tf2onnx --input="1x4xsi32=1" --input="1x4xsi32=1" --input="1x4xsi32=1" > trace.txt 2>&1View the trace file
trace.txtWhat component(s) does this issue relate to?
Runtime
Version information
Commit hash of iree:
1f3382d7305d7b2920fe7cb6072b07ca81945f28Versions of pip packages used:
Additional context
Build environment and commands:
My system has a 80-thread Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz and a Tesla V100-PCIE-32GB GPU.
Im using the following docker file:
To build the compiler and runtime Im using the following commands:
python -m pip install -r runtime/bindings/python/iree/runtime/build_requirements.txtcmake --build ../iree-build/ -j 30The text was updated successfully, but these errors were encountered: