{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":840437666,"defaultBranch":"main","name":"iree-test-suites","ownerLogin":"iree-org","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2024-08-09T17:50:03.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/107954215?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1723584659.0","currentOid":""},"activityList":{"items":[{"before":"a02aca9aac18ab35d5998f91f5ef13e84d0d0f37","after":"7b8bdf790064f8a1eb437c499df442f1b4c3e5df","ref":"refs/heads/main","pushedAt":"2024-09-19T20:25:51.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ScottTodd","name":"Scott Todd","path":"/ScottTodd","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4010439?s=80&v=4"},"commit":{"message":"Begin testing models from the ONNX Model Zoo. (#23)\n\nProgress on https://github.com/iree-org/iree-test-suites/issues/6.\r\n\r\nA sample test report HTML file is available here:\r\nhttps://scotttodd.github.io/iree-test-suites/onnx_models/report_2024_09_17.html\r\n\r\nThese new tests\r\n\r\n* Download models from https://github.com/onnx/models\r\n* Extract metadata from the models to determine which functions to call\r\nwith random data\r\n* Run the models through [ONNX Runtime](https://onnxruntime.ai/) as a\r\nreference implementation\r\n* Import the models using `iree-import-onnx` (until we have a better\r\nAPI: https://github.com/iree-org/iree/issues/18289)\r\n* Compile the models using `iree-compile` (currently just for `llvm-cpu`\r\nbut this could be parameterized later)\r\n* Run the models using `iree-run-module`, checking outputs using\r\n`--expected_output` and the reference data\r\n\r\nTests are written in Python using a set of pytest helper functions. As\r\nthe tests run, they can log details about what commands they are\r\nrunning. When run locally, the `artifacts/` directory will contain all\r\nthe relevant files. More can be done in follow-up PRs to improve the\r\nergonomics there (like generating flagfiles).\r\n\r\nEach test case can use XFAIL like\r\n`@pytest.mark.xfail(raises=IreeRunException)`. As we test across\r\nmultiple backends or want to configure the test suite from another repo\r\n(e.g. [iree-org/iree](https://github.com/iree-org/iree)), we can explore\r\nmore expressive marks.\r\n\r\nNote that unlike the ONNX _operator_ tests, these tests use\r\n`onnxruntime` and `iree-import-onnx` at test time. The operator tests\r\nhandle that as an infrequently ran offline step. We could do something\r\nsimilar here, but the test inputs and outputs can be rather large for\r\nreal models and that gets into Git LFS or cloud storage territory.\r\n\r\nIf this test authoring model works well enough, we can do something\r\nsimilar for other ML frameworks like TFLite\r\n(https://github.com/iree-org/iree-test-suites/issues/5).","shortMessageHtmlLink":"Begin testing models from the ONNX Model Zoo. (#23)"}},{"before":"f37eae2013c4d643e7f9eba1e694389f51bbad46","after":"a02aca9aac18ab35d5998f91f5ef13e84d0d0f37","ref":"refs/heads/main","pushedAt":"2024-09-16T16:07:53.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ScottTodd","name":"Scott Todd","path":"/ScottTodd","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4010439?s=80&v=4"},"commit":{"message":"Move flaky onnx op test to skip_run_tests. (#28)\n\nFollow-up to https://github.com/iree-org/iree-test-suites/pull/27.","shortMessageHtmlLink":"Move flaky onnx op test to skip_run_tests. (#28)"}},{"before":"03f10e99d5f80696107038f3e8da8525aa31d50a","after":"f37eae2013c4d643e7f9eba1e694389f51bbad46","ref":"refs/heads/main","pushedAt":"2024-09-16T15:48:36.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ScottTodd","name":"Scott Todd","path":"/ScottTodd","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4010439?s=80&v=4"},"commit":{"message":"Update ONNX op test statuses after recent IREE changes. (#27)\n\nSee recent workflow history:\r\nhttps://github.com/iree-org/iree-test-suites/actions/workflows/test_onnx_ops.yml.","shortMessageHtmlLink":"Update ONNX op test statuses after recent IREE changes. (#27)"}},{"before":"cc9b0267634a3c5905b6179983e55e838b5275cc","after":"03f10e99d5f80696107038f3e8da8525aa31d50a","ref":"refs/heads/main","pushedAt":"2024-09-04T21:25:18.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ScottTodd","name":"Scott Todd","path":"/ScottTodd","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4010439?s=80&v=4"},"commit":{"message":"Rename matmul/ to linalg_ops/, rebase IREE, add test workflow. (#21)\n\nProgress on https://github.com/iree-org/iree-test-suites/issues/2.\r\n\r\n* Renamed top level `matmul/` folder to `linalg_ops/`. We can put\r\n`convolution` (linalg) and `attention` (linalg_ext) tests under here\r\ntoo. Maybe other named ops or generics too.\r\n* Rebased on the latest IREE code, including flag renames (though we\r\nshould drop usage of some flags and prefer to test the main user-facing\r\noptions)\r\n* Added a workflow that goes through the instructions in\r\n`linalg_ops/README.md` and configured both workflows to run on `push`\r\nevents in addition to the current triggers.","shortMessageHtmlLink":"Rename matmul/ to linalg_ops/, rebase IREE, add test workflow. (#21)"}},{"before":"9e921d0ea271a85f772eee22965585461c9b14c2","after":"cc9b0267634a3c5905b6179983e55e838b5275cc","ref":"refs/heads/main","pushedAt":"2024-09-04T15:49:36.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ScottTodd","name":"Scott Todd","path":"/ScottTodd","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4010439?s=80&v=4"},"commit":{"message":"Update ONNX op XFAILs after latest nightly IREE release. (#20)\n\nThese tests are newly passing with the torch-mlir update from\r\nhttps://github.com/iree-org/iree/pull/18417.\r\n\r\nWe could also add an action to update this file automatically, or we\r\ncould relax the `strict` xfails.","shortMessageHtmlLink":"Update ONNX op XFAILs after latest nightly IREE release. (#20)"}},{"before":"9f2ca6dde62a628adc8ae0bbdf3afe1f0a883b9b","after":"9e921d0ea271a85f772eee22965585461c9b14c2","ref":"refs/heads/main","pushedAt":"2024-08-29T17:46:34.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ScottTodd","name":"Scott Todd","path":"/ScottTodd","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4010439?s=80&v=4"},"commit":{"message":"Log input files and test case paths when ONNX tests fail. (#19)\n\nFixes https://github.com/nod-ai/SHARK-TestSuite/issues/319\r\n\r\n* Sample compile error logs:\r\nhttps://gist.github.com/ScottTodd/1fbb40a72b226464b9ec2a1396880914\r\n* Sample run error logs:\r\nhttps://gist.github.com/ScottTodd/111320271f012f42f984fdd5cfc65186","shortMessageHtmlLink":"Log input files and test case paths when ONNX tests fail. (#19)"}},{"before":"452106c372d92adbd61be71dbe0ccaa9fa03c46a","after":"9f2ca6dde62a628adc8ae0bbdf3afe1f0a883b9b","ref":"refs/heads/main","pushedAt":"2024-08-26T15:35:09.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ScottTodd","name":"Scott Todd","path":"/ScottTodd","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4010439?s=80&v=4"},"commit":{"message":"Update ONNX op XFAILs after latest nightly IREE releases. (#18)\n\nThese tests are newly passing with the torch-mlir update from\r\nhttps://github.com/iree-org/iree/pull/18345.","shortMessageHtmlLink":"Update ONNX op XFAILs after latest nightly IREE releases. (#18)"}},{"before":"3d381545f83cb39ae1d8420aff552ef9c981ede9","after":"452106c372d92adbd61be71dbe0ccaa9fa03c46a","ref":"refs/heads/main","pushedAt":"2024-08-23T18:06:24.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ScottTodd","name":"Scott Todd","path":"/ScottTodd","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4010439?s=80&v=4"},"commit":{"message":"Drop matmul \"compilation_info\" generator code and test cases. (#15)\n\nProgress on https://github.com/iree-org/iree-test-suites/issues/2\r\n\r\n(This is not yet running on CI anywhere, still refactoring the test\r\nsuite before passing it back into developer hands)\r\n\r\nThis was leaking compiler implementation details into test cases. The\r\ncompiler should be selecting a pipeline on its own based on the target\r\ndevice(s). If a developer wants to try a different pipeline, they can\r\nmake a code change, introduce a dedicated flag (global/hacky), or use a\r\ntransform dialect script.\r\n\r\nI think this drops some of the rocm test suites, which were only added\r\nusing explicit compilation_info settings. Leaving adding that back to a\r\nfollow-up / further refactoring. Editing the 2000 line `CMakeLists.txt`\r\nfile (generated from a 700 line `BUILD.bazel` file) is error-prone\r\nespecially with how this file has grown over the years. _If_ we stick to\r\nCMake/CTest, it probably makes sense to start the file over from\r\nscratch.","shortMessageHtmlLink":"Drop matmul \"compilation_info\" generator code and test cases. (#15)"}},{"before":"a2cfb01c1ecf7bbff3447c0eaf4f4d66882a5df8","after":"3d381545f83cb39ae1d8420aff552ef9c981ede9","ref":"refs/heads/main","pushedAt":"2024-08-21T14:59:54.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ScottTodd","name":"Scott Todd","path":"/ScottTodd","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4010439?s=80&v=4"},"commit":{"message":"Make matmul generator acc_type argument required/explicit. (#16)\n\nThe comment on `infer_acc_type` was outdated and making the argument\r\nrequired will simplify the generator.","shortMessageHtmlLink":"Make matmul generator acc_type argument required/explicit. (#16)"}},{"before":"ec400693f8988cc467bb3a77021e96c4fcfdaf9f","after":"a2cfb01c1ecf7bbff3447c0eaf4f4d66882a5df8","ref":"refs/heads/main","pushedAt":"2024-08-20T17:46:16.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ScottTodd","name":"Scott Todd","path":"/ScottTodd","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4010439?s=80&v=4"},"commit":{"message":"Begin porting matmul test suite from iree-org/iree. (#13)\n\nProgress on https://github.com/iree-org/iree-test-suites/issues/2.\r\n\r\nThis begins to port the tests from\r\nhttps://github.com/iree-org/iree/tree/main/tests/e2e/matmul. The changes\r\nhere aim to be minimal, but a lot of cleanup is needed to make this\r\nsustainable.\r\n\r\n* A new CMake project named `iree-test-suites-matmul` is added. We could\r\nshare this with other test suites that need CMake, or let each test\r\nsuite create its own. I don't want to overgeneralize yet while so much\r\nis still up in the air.\r\n* The CMake project depends on the IREE runtime libraries and header\r\nfiles to build a `iree-e2e-matmul-test` binary. This binary handles\r\npreparing inputs, running a compiled program, and comparing outputs\r\nagainst a reference implementation. That may be split into multiple\r\nlibraries or parts of it may move into Python or another test harness.\r\nTBD.\r\n* The CMake project then defines some test cases using\r\n`iree_generated_e2e_runner_test`. This may be split into an offline\r\ngeneration step and a test runner other than ctest. TBD.\r\n\r\nTODO:\r\n\r\n- [x] Fix namespace/symbol collisions in `test_utils.h` (fork further)\r\nand include it correctly\r\n- [x] Fix namespace/symbol collisions in\r\n`iree_e2e_generated_runner_test.cmake` (fork further)\r\n- [ ] Add a GitHub Actions workflow running the tests\r\n- [x] Structure the CMake project such that it can be used from\r\niree-org/iree in place of the current tests\r\n- [ ] Fork or remove dependencies on the upstream CMake helpers used by\r\n`matmul/iree_e2e_generated_runner_test.cmake`?\r\n- [x] Allow pointing at a local IREE, like\r\nhttps://github.com/ScottTodd/iree/tree/infra-e2e-test-suites which\r\ndeletes the in-tree tests and supporting infra","shortMessageHtmlLink":"Begin porting matmul test suite from iree-org/iree. (#13)"}},{"before":"323c2df477544d789c56e4dfda9b047580b77cbf","after":"ec400693f8988cc467bb3a77021e96c4fcfdaf9f","ref":"refs/heads/main","pushedAt":"2024-08-16T15:46:44.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ScottTodd","name":"Scott Todd","path":"/ScottTodd","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4010439?s=80&v=4"},"commit":{"message":"Update ONNX op xfails after latest IREE nightly. (#11)","shortMessageHtmlLink":"Update ONNX op xfails after latest IREE nightly. (#11)"}},{"before":"d944fed69ecacbfeb495d86b1b7b40b903ab668f","after":"323c2df477544d789c56e4dfda9b047580b77cbf","ref":"refs/heads/main","pushedAt":"2024-08-15T20:34:29.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ScottTodd","name":"Scott Todd","path":"/ScottTodd","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4010439?s=80&v=4"},"commit":{"message":"Rename onnx-ops folder to onnx_ops. (#10)\n\nThis is consistent with file names and style in (most) other iree-org\r\nrepositories.","shortMessageHtmlLink":"Rename onnx-ops folder to onnx_ops. (#10)"}},{"before":"9b75f5150ffd64194944b5ec78a2b88ab4682ef6","after":"d944fed69ecacbfeb495d86b1b7b40b903ab668f","ref":"refs/heads/main","pushedAt":"2024-08-14T21:45:13.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ScottTodd","name":"Scott Todd","path":"/ScottTodd","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4010439?s=80&v=4"},"commit":{"message":"Add back `if: failure()` to test job.\n\nWe could always upload (assuming the job got far enough to produce the files), but the main thing is to run if the tests failed.","shortMessageHtmlLink":"Add back if: failure() to test job."}},{"before":"62eab0f509d9b868f7916078315db109c5be5658","after":"9b75f5150ffd64194944b5ec78a2b88ab4682ef6","ref":"refs/heads/main","pushedAt":"2024-08-14T21:28:04.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ScottTodd","name":"Scott Todd","path":"/ScottTodd","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4010439?s=80&v=4"},"commit":{"message":"Fix badge for ONNX Ops test suite status in README.\n\nThe workflow is not running on `push` events, so just allow any event (`workflow_dispatch` or `schedule`).","shortMessageHtmlLink":"Fix badge for ONNX Ops test suite status in README."}},{"before":"63137ec793f98890f72b131eea88e087e718c3c9","after":"62eab0f509d9b868f7916078315db109c5be5658","ref":"refs/heads/main","pushedAt":"2024-08-14T21:21:34.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"ScottTodd","name":"Scott Todd","path":"/ScottTodd","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4010439?s=80&v=4"},"commit":{"message":"Use Exceptions in onnx importer to avoid importing broken test cases. (#9)\n\nA few of these cases were allowing errors to slip through as broken test\r\ncases. Rather than rely on checking for `None` or watching the logs,\r\nthis uses Exceptions as error handling.\r\n\r\nAs a result of fixing these oversights, some broken test cases are\r\ndeleted. I'm not sure how `update_config_xfails.py` will handle the list\r\nof tests changing.\r\n\r\nLogs from running the import script:\r\nhttps://gist.github.com/ScottTodd/bd327bc7e95485b12a9f9ebdcddf0bc2","shortMessageHtmlLink":"Use Exceptions in onnx importer to avoid importing broken test cases. ("}},{"before":"e985dfe565a1d72b89867cb26659e373be98e302","after":"63137ec793f98890f72b131eea88e087e718c3c9","ref":"refs/heads/main","pushedAt":"2024-08-14T17:45:12.000Z","pushType":"pr_merge","commitsCount":4,"pusher":{"login":"ScottTodd","name":"Scott Todd","path":"/ScottTodd","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4010439?s=80&v=4"},"commit":{"message":"Port ONNX op test suite. (#8)\n\nFixes https://github.com/iree-org/iree-test-suites/issues/1.\r\n\r\nThis is a port of the generated ONNX operator (\"node\") tests from\r\nhttps://github.com/nod-ai/SHARK-TestSuite/tree/main/iree_tests. Major\r\nchanges / simplifications:\r\n\r\n* Focused the README on the ONNX operator tests, avoiding generalizing\r\nto other frameworks\r\n* Switched to loading upstream tests from sitepackages instead of a git\r\nsubmodule\r\n* Added `requirements-dev.txt` and `requirements-iree.txt` files so\r\nrunning the tests doesn't require ONNX\r\n* Revised function and variable names and added pytype annotations in\r\nthe import script\r\n* Added TODOs for error handling and logging\r\n* Stopped generating `.npy` files, favoring `.bin` files\r\n* Limited to just `run_module_io_flags.txt`, no `test_cases.json`\r\n* Limited to only local files, no remote or Git LFS files\r\n* Inlined/simplified Exception subclasses and other helper functions","shortMessageHtmlLink":"Port ONNX op test suite. (#8)"}},{"before":"f4dbefe051cf682914fe0c4b47571500bb754921","after":null,"ref":"refs/heads/onnx-ops","pushedAt":"2024-08-13T21:30:59.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"ScottTodd","name":"Scott Todd","path":"/ScottTodd","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4010439?s=80&v=4"}},{"before":null,"after":"f4dbefe051cf682914fe0c4b47571500bb754921","ref":"refs/heads/onnx-ops","pushedAt":"2024-08-13T21:30:10.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ScottTodd","name":"Scott Todd","path":"/ScottTodd","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4010439?s=80&v=4"},"commit":{"message":"Add workflow testing onnx ops.","shortMessageHtmlLink":"Add workflow testing onnx ops."}},{"before":"3014569926f06d1e08fb12c0d2be635e58e5cdf2","after":"e985dfe565a1d72b89867cb26659e373be98e302","ref":"refs/heads/main","pushedAt":"2024-08-09T18:21:20.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"ScottTodd","name":"Scott Todd","path":"/ScottTodd","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4010439?s=80&v=4"},"commit":{"message":"Initial commit of general repository files.","shortMessageHtmlLink":"Initial commit of general repository files."}},{"before":"2fe750570d7f289190bdb9321aec01a2f693f244","after":"3014569926f06d1e08fb12c0d2be635e58e5cdf2","ref":"refs/heads/main","pushedAt":"2024-08-09T18:20:50.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"ScottTodd","name":"Scott Todd","path":"/ScottTodd","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4010439?s=80&v=4"},"commit":{"message":"Initial commit of general repository files.","shortMessageHtmlLink":"Initial commit of general repository files."}},{"before":null,"after":"2fe750570d7f289190bdb9321aec01a2f693f244","ref":"refs/heads/main","pushedAt":"2024-08-09T17:50:03.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"ScottTodd","name":"Scott Todd","path":"/ScottTodd","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/4010439?s=80&v=4"},"commit":{"message":"Initial commit","shortMessageHtmlLink":"Initial commit"}}],"hasNextPage":false,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"Y3Vyc29yOnYyOpK7MjAyNC0wOS0xOVQyMDoyNTo1MS4wMDAwMDBazwAAAAS7Sl7q","startCursor":"Y3Vyc29yOnYyOpK7MjAyNC0wOS0xOVQyMDoyNTo1MS4wMDAwMDBazwAAAAS7Sl7q","endCursor":"Y3Vyc29yOnYyOpK7MjAyNC0wOC0wOVQxNzo1MDowMy4wMDAwMDBazwAAAASWgz4E"}},"title":"Activity · iree-org/iree-test-suites"}