-
Notifications
You must be signed in to change notification settings - Fork 231
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] Report GPU OOM on recent passed CI premerges. #9914
Comments
This issue is invalid. In test case When GPU use default 1532m in IT:
and:
// 2147483712 > 2G, 2147483712 is the trunk size of the reading Parquet file. When I set GPU memory as 10G: |
@abellina help check the last comment. |
@jlowe Do we need to set GPU memory > 2G in premerge? We expect CPU error is:
I think we also do not expect this. |
No. We expect the GPU load to fail, whether that's due to OOM or size overflow. We do expect the CPU BrotliCodec error, since we did not configure brotli support in Spark. The parquet_testing_test will programmatically detect all files and try each one. The brotli file is problematic in different ways between the CPU and GPU which is why it just checks for any exception rather than the same type of exception. If desired we could extend the file detection logic to skip a list of known files we don't really want to test like this one. |
Describe the bug
It's separated from 9829
In recent CI premerge, it reports GPU OutOfMemory error:
Although the CI is passed.
Note: it's not related to non-UTC time zone PR: #9719, because it's not merged yet when I create this issue.
It's not related to non-UTC time zone PR: #9773, because this is only adding a non-UTC xfail mark. The error is reported when testing UTC time zone.
I think it's not related to non-UTC related PRs, it's a problem already existing.
Steps/Code to reproduce bug
refer to a passed CI 8591.
On the premerge #8591, click Blue Ocean and then click Premerge CI 2, then download the log.
The text was updated successfully, but these errors were encountered: