[DONOTMERGE] add CI tests for torch.compile'ing the transforms.v2 kernels

[pmeier]

I don't intend to merge this. Rather, this should serve as base to get a feeling how far away we are from achieving our goal in #8056.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/8127

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 8 New Failures, 1 Unrelated Failure

As of commit 90ab254 with merge base 6e18cea ():

NEW FAILURES - The following jobs have failed:

• Tests / compile (false, eager, false) / dynamic=false,backend=eager,fullgraph=false (gh)
  test/test_transforms_v2.py::TestJPEG::test_kernel_video
• Tests / compile (false, eager, true) / dynamic=false,backend=eager,fullgraph=true (gh)
  test/test_transforms_v2.py::TestJPEG::test_kernel_video
• Tests / compile (false, inductor, false) / dynamic=false,backend=inductor,fullgraph=false (gh)
  test/test_transforms_v2.py::TestJPEG::test_kernel_video
• Tests / compile (false, inductor, true) / dynamic=false,backend=inductor,fullgraph=true (gh)
  test/test_transforms_v2.py::TestJPEG::test_kernel_video
• Tests / compile (true, eager, false) / dynamic=true,backend=eager,fullgraph=false (gh)
  test/test_transforms_v2.py::TestJPEG::test_kernel_video
• Tests / compile (true, eager, true) / dynamic=true,backend=eager,fullgraph=true (gh)
  test/test_transforms_v2.py::TestJPEG::test_kernel_video
• Tests / compile (true, inductor, false) / dynamic=true,backend=inductor,fullgraph=false (gh)
  test/test_transforms_v2.py::TestJPEG::test_kernel_video
• Tests / compile (true, inductor, true) / dynamic=true,backend=inductor,fullgraph=true (gh)
  test/test_transforms_v2.py::TestJPEG::test_kernel_video

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Build M1 Wheels / pytorch/vision / upload / wheel-py3_8-cpu (gh) (trunk failure)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[pmeier]

With torch-2.2.0.dev20231121+cpu and torchvision @ 893b4ab:

• backend='eager', dynamic=True: 259 failed
• backend='inductor', dynamic=True: 307 failed
• backend='eager', dynamic=False: 98 failed
• backend='inductor', dynamic=False: 110 failed

[pmeier]

Some relevant issues on core that will resolve a lot of the issues:

• math.radians with integer input creates a graph break for dynamic shapes pytorch#114218
• boolean masking creates a graph break due to aten.nonzero pytorch#114220 avoided by use torch.where over boolean masking #8171
• inductor with dynamic shapes fails on float tensor creation from a tuple of ints pytorch#114231
• Support builtin round in torch.compile with dynamic shapes pytorch#114310
• add support for linalg lstsq in dynamo pytorch#114353
• conv2d fails in inductor with dynamic shapes when using groups pytorch#114354
• inductor silently ignores clone followed by a reshape pytorch#114464
• dynamo supports Tensor.tolist but not Tensor.item pytorch#114483
• [dynamo] Added dyn shapes support for math trigo ops: sin(h), cos(h), tan(h) ... pytorch#114866

[pmeier]

Failures on pad_mask with fullgraph are a test error and fixed in #8132.

[pmeier]

In c5c72ab and 29ea48a, I've added atol=1, rtol=0 for uint8 / bilinear resize. With this, the most lenient setting, i.e. static shapes, eager backend, and graphbreaks being allowed, the tests are now passing: https://github.com/pytorch/vision/actions/runs/6971806789/job/18972632003?pr=8127 🎉

[pmeier]

With torch-2.3.0.dev20231218+cpu and torchvision @ 6c2e0ae:

dynamic	backend	fullgraph	failing tests
False	eager	False	0
False	eager	True	63
False	inductor	False	8
False	inductor	True	71
True	eager	False	72
True	eager	True	192
True	inductor	False	86
True	inductor	True	206

[pmeier]

I've factored out e6a54bf into #8171.

[pmeier]

With torch-2.3.0.dev20231222+cpu and torchvision @ 26fb5ef:

dynamic	backend	fullgraph	failing tests	diff to previous
False	eager	False	0	0
False	eager	True	23	-40
False	inductor	False	8	0
False	inductor	True	31	-40
True	eager	False	72	0
True	eager	True	95	-97
True	inductor	False	80	-6
True	inductor	True	103	-103

[NicolasHug]

Some great progress with torch-2.3.0.dev20240117+cpu and torchvision @ 1de7a74 where 72 failing tests were resolved from all of the dynamic=True jobs.

[vfdev-5]

With torch-2.4.0.dev20240401+cpu and torchvision 5181a85 we have the following failures:

• test.test_transforms_v2.TestPerspective: image/bbox - due to torch._dynamo.exc.Unsupported: dynamic shape operator: aten.linalg_lstsq.default - can't be fixed now
• test.test_transforms_v2.TestSanitizeBoundingBoxes.test_kernel - due to torch._dynamo.exc.Unsupported: dynamic shape operator: aten.nonzero.default - probably, can't be fixed neither
• test.test_transforms_v2.TestJPEG: image/video - due to torch._dynamo.exc.TorchRuntimeError: Failed running call_function image.encode_jpeg(*(FakeTensor(..., size=(3, 17, 11), dtype=torch.uint8), 5), **{}):