-
Notifications
You must be signed in to change notification settings - Fork 42
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[UT] regression in test_subprocess.py with the PTDB 0.5.3 #800
Comments
Continue to fail with agama 821.32. |
Blocked by new driver's bug. Already file a JIRA. |
This should not be a driver's bug. Because driver team cannot reproduce it with oneapi 2024.0. I confirmed that in the same environment(should be 821.30), 2024.0 works, but 2024.1 fails. So this may be a compiler regression. Will file a new JIRA to compiler team. |
Introducing a new approach for skipping tests. Skip lists are located in the directory `scripts/skiplist`. Currently there are two skip lists: * `scripts/skiplist/default` - default, tests to skip in the main workflow and by `test-triton.sh` script. * `scripts/skiplist/conda` - tests to skip in the conda workflow (currently it requires more tests to skip). In future, it is possible to add more skip lists. To specify a custom skip list set `TRITON_TEST_SKIPLIST_DIR` before executing `test-triton.sh`, for example, conda workflow sets this to use a custom skip list: ``` TRITON_TEST_SKIPLIST_DIR=scripts/skiplist/conda ``` Every skip list can contain 0 or more `.txt` files, each file corresponds to a "test suite". The whole list of existing test suites can be obtained from `test-triton.sh` by searching for different values of `TRITON_TEST_SUITE`, for example: ``` TRITON_DISABLE_LINE_INFO=0 TRITON_TEST_SUITE=line_info \ pytest --verbose --device xpu language/test_line_info.py ``` Currently there are 7 "tests suites": * `language` * `subprocess` * `runtime` * `line_info` * `interpreter` * `operators` * `regression` For example, if you want to skip a test in a "subprocess" test suite, add a line with the fully qualified test name to `scripts/skiplist/default/subprocess.txt`: ``` # This is a comment. Please use comments to specify why next tests are skipped, for example # #800 test/unit/language/test_subprocess.py::test_print[print-int32] ``` If a `.txt` file for a test suite does not exist, or empty then no tests will be skipped. To get a full list of the available tests for a test suite, use `pytest` with `--collect-only`, for example: ``` $ pytest --collect-only language/test_subprocess.py -q | sort 33 tests collected in 0.01s test/unit/language/test_subprocess.py::test_assert[assert] test/unit/language/test_subprocess.py::test_assert[device_assert] ... ``` This PR contains only several tests to skip (see `scripts/skiplist/default/subprocess.txt`), see #800. The corresponding Python code to skip tests programmatically has been removed from `test_subprocess.py`. If this works, we can gradually populate the default skip list by removing code to skip from Python files. Note that an error will be generated if you try to skip a test that does not exists. This is intendent behavior (we use `--select-fail-on-missing` for [pytest-select](https://pypi.org/project/pytest-select/)) and required to calculate a pass rate.
This issue needs to be rechecked after June Rolling Driver release. |
Continue to fail with agama 821.35 & 881.19 6/12/2024 |
Continue to fail with agama 821. 6/17/2024 |
Continue to fail with agama 821. |
@AshburnLee could you please retest with the Agama 914? |
Continue to fail with agama 914. |
1 similar comment
Continue to fail with agama 914. |
Closes #800 I accidentally discovered this effect while I was working on #1082. It seems that accessing the tensor elements via `repr` builtin python function stimulates the execution of the kernel and obtaining output from it. Although the behavior is different from other backends, it allows us to at least somehow test this feature right now. --------- Signed-off-by: Anatoly Myachev <[email protected]>
Reminder: don't forget to remove: intel-xpu-backend-for-triton/python/test/unit/language/print_helper.py Lines 119 to 122 in fea510c
|
With PTDB 0.5.3 and Agama 950 this issue is still reproducible without the
With
|
@etiotto and @whitneywhtsang what are the next steps to resolve the issue? |
There is a CMPLRLLVM ticket opened for this issue, and we should continue to follow up there to have it fixed. |
Closes intel#800 Signed-off-by: Kirill Suvorov <[email protected]>
12 tests cases are failing:
The text was updated successfully, but these errors were encountered: