Tracking CI status

[apostasie]

With the current effort on testing, we should start looking into our run duration, platform coverage, and hard data on flakyness / disabled tests.

This issue here is to record that information over time.

This is the first sample (of #3535), ran on 10/19/2024

Target	Prep Time	Non-flaky	Flaky
windows	238 s	256 tests, 159 skipped in 81.173s	166 tests, 66 skipped in 263.612s
rootful-1.6	277 s	562 tests, 276 skipped in 120.279s	2 runs, 737 tests, 81 skipped, 2 failures in 1024.881s
rootful-1.7	292 s	562 tests, 276 skipped in 141.374s	735 tests, 76 skipped in 1134.421s
rootful-2	273 s	562 tests, 276 skipped in 124.186s	735 tests, 80 skipped in 1041.623s
rootful-2-arm	458 s	567 tests, 275 skipped in 185.060s	735 tests, 81 skipped in 1308.355s
rootless-1.6	283 s	502 tests, 305 skipped in 161.850s	727 tests, 143 skipped in 893.197s
rootless-1.7	272 s	550 tests, 274 skipped in 130.962s	727 tests, 115 skipped in 1020.648s
rootless-1.7-slirp	288 s	550 tests, 274 skipped in 151.411s	727 tests, 115 skipped in 1012.942s
rootless-2	274 s	550 tests, 274 skipped in 135.481s	2 runs, 729 tests, 119 skipped, 2 failures in 937.661s
ipv6	272 s	341 tests, 328 skipped in 70.056s	na
kube	153 s	33.571 s	na
docker	25 s	467 tests, 302 skipped in 126.707s ipv6: 331 tests, 325 skipped in 15.012s	CURRENTLY BUST

Here is the list of the worst offenders (taken from the arm run):

github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer 2m38.51s
github.com/containerd/nerdctl/v2/cmd/nerdctl/ipfs TestIPFSCompNoBuild/with-stargz-no-byAddr 2m21.76s
github.com/containerd/nerdctl/v2/cmd/nerdctl/login TestLoginAgainstVariants 2m7.84s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer/SRC_PATH_specifies_a_file 1m24.52s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer/SRC_PATH_specifies_a_directory 1m3.08s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer/SRC_PATH_specifies_a_directory/DEST_PATH_exists_and_is_a_directory 42.04s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestRunWithLogBinary 27.11s
github.com/containerd/nerdctl/v2/cmd/nerdctl/compose TestComposeImages 26.48s
github.com/containerd/nerdctl/v2/cmd/nerdctl/ipfs TestIPFSCompNoBuild/with-no-stargz-byAddr 25.61s
github.com/containerd/nerdctl/v2/cmd/nerdctl/ipfs TestIPFSCompNoBuild/with-no-stargz-no-byAddr 25.56s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestRunPort 24.71s
github.com/containerd/nerdctl/v2/cmd/nerdctl/compose TestComposeRestart 24.51s
github.com/containerd/nerdctl/v2/cmd/nerdctl/compose TestComposeStart 22.37s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestRestartIPCContainer 21.37s
github.com/containerd/nerdctl/v2/cmd/nerdctl/login TestLoginAgainstVariants/Login_against_`tls:_true_port:_443_auth:_token` 21.36s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer/SRC_PATH_specifies_a_file/DEST_PATH_exists_and_is_a_directory 21.23s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer/SRC_PATH_specifies_a_file/DEST_PATH_is_in_a_volume 21.08s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer/SRC_PATH_specifies_a_directory/DEST_PATH_exists_and_is_a_directory/SRC_PATH_does_end_with_`/.` 21.04s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer/SRC_PATH_specifies_a_directory/DEST_PATH_exists_and_is_a_directory/SRC_PATH_does_not_end_with_`/.` 21s
github.com/containerd/nerdctl/v2/cmd/nerdctl/compose TestComposePauseAndUnpause 20.92s
github.com/containerd/nerdctl/v2/cmd/nerdctl/compose TestComposeDownRemoveUsedNetwork 20.78s

Current take-aways:

1. arm is significantly slower - either the machine we are using is not big enough, or it is emulated, or github offering is not really good...
2. build time overall is consistent - at 4 minutes though, we should be able to cut that to 1 minute or less (implies rethinking the way we build the image and possibly push the "dependencies" part on a registry instead of rebuilding them every time)
3. windows testing is really weak, with 100 safe tests, and 100 flaky tests
4. we now have about 300 safe tests (not counting ipv6 and kube), and about 650 "flaky" tests - so, about 1/3 are considered safe - when the container tests suite will be done, the balance should shift towards 50%
5. for the worst offenders:

• TestCopy are very slow - current in-progress rewrite will parallelize and reduce this
• Compose tests are definitely the number 1 contributor (including some IPFS tests) - we should look into this and figure out why compose is so slow

6. current flakyness status

• 2 failed runs out of the 8 main target platforms (with 4 test failures) - these are both TestIPFSCompNoBuild - the issue being likely content digest not found #3513 (it might be a snapshotter issue though, and not a nerdctl one)
• 4 "rewritten" tests are explicitly flagged as flaky (3 on windows, 1 on IPFS)
• 9 tests are currently disabled (marked NerdctlNeedsFixing), although some of them might have been fixed recently - they should be reviewed

Tagging @djdongjin @AkihiroSuda

Will post other samples on this ticket in the future to track progress.

[apostasie]

Ran on 11/02/2024 from https://github.com/containerd/nerdctl/actions/runs/11645055099/job/32427652231?pr=3636

Target	Prep Time	Non-flaky	Flaky
rootful-1.6	206 s	577 tests, 271 skipped in 133.410s	738 tests, 89 skipped in 989.100s
rootful-1.7	205 s	577 tests, 271 skipped in 216.106s	BUST - DEADLOCK?
rootful-2	199 s	577 tests, 271 skipped in 127.292s	738 tests, 88 skipped in 1000.347s
rootful-2-arm	163 s	582 tests, 270 skipped in 127.236s	2 runs, 739 tests, 89 skipped, 1 failure in 1121.278s
rootless-1.6	207 s	509 tests, 302 skipped in 128.673s	730 tests, 149 skipped in 1120.853s
rootless-1.7	192 s	565 tests, 269 skipped in 137.297s	730 tests, 123 skipped in 1007.554s
rootless-1.7-slirp	202 s	565 tests, 269 skipped in 154.632s	730 tests, 123 skipped in 1135.860s
rootless-2	199 s	565 tests, 269 skipped in 156.960s	730 tests, 127 skipped in 1135.354s
windows	286 s	268 tests, 156 skipped in 130.698s	170 tests, 75 skipped in 229.758s
ipv6	201 s	344 tests, 329 skipped in 72.671s	na
kube	150 s	36 s	na
docker	18 s	508 tests, 304 skipped in 126.412s ipv6: 343 tests, 329 skipped in 17.319s	472 tests, 121 skipped in 709.353s

Here is the list of the worst offenders (taken from the arm run):

github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer 2m40.68s
github.com/containerd/nerdctl/v2/cmd/nerdctl/login TestLoginAgainstVariants 1m44.79s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer/SRC_PATH_specifies_a_file 1m25.17s
github.com/containerd/nerdctl/v2/cmd/nerdctl/ipfs TestIPFSCompNoBuild/with-stargz-no-byAddr 1m24.49s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer/SRC_PATH_specifies_a_directory 1m3.8s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer/SRC_PATH_specifies_a_directory/DEST_PATH_exists_and_is_a_directory 42.47s
github.com/containerd/nerdctl/v2/cmd/nerdctl/compose TestComposeRestart 34.86s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestRunPort 27.68s
github.com/containerd/nerdctl/v2/cmd/nerdctl/login TestLoginAgainstVariants/Login_against_`tls:_false_port:_443_auth:_token` 27.1s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestRestartIPCContainer 25.05s
github.com/containerd/nerdctl/v2/cmd/nerdctl/compose TestComposeImages 23.23s
github.com/containerd/nerdctl/v2/cmd/nerdctl/ipfs TestIPFSCompNoBuild/with-no-stargz-byAddr 22.91s
github.com/containerd/nerdctl/v2/cmd/nerdctl/ipfs TestIPFSCompNoBuild/with-no-stargz-no-byAddr 22.88s
github.com/containerd/nerdctl/v2/cmd/nerdctl/compose TestComposeStart 22.8s
github.com/containerd/nerdctl/v2/cmd/nerdctl/login TestLoginAgainstVariants/Login_against_`tls:_false_port:_80_auth:_token` 22.21s
github.com/containerd/nerdctl/v2/cmd/nerdctl/login TestLoginAgainstVariants/Login_against_`tls:_true_port:_0_auth:_token` 22.19s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer/SRC_PATH_specifies_a_file/DEST_PATH_is_in_a_volume 21.33s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer/SRC_PATH_specifies_a_directory/DEST_PATH_exists_and_is_a_directory/SRC_PATH_does_not_end_with_`/.` 21.3s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer/SRC_PATH_specifies_a_file/DEST_PATH_exists_and_is_a_directory 21.23s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestCopyToContainer/SRC_PATH_specifies_a_directory/DEST_PATH_exists_and_is_a_directory/SRC_PATH_does_end_with_`/.` 21.17s
github.com/containerd/nerdctl/v2/cmd/nerdctl/compose TestComposePauseAndUnpause 21.09s
github.com/containerd/nerdctl/v2/cmd/nerdctl/compose TestComposeDownRemoveUsedNetwork 21.07s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestSyslogFormat 20.48s
github.com/containerd/nerdctl/v2/cmd/nerdctl/container TestSyslogFacilities 20.35s

Main take-aways:

• arm time significantly down now that we use a bigger instance
• prep time is down - because the dependencies stage is now built separately
• docker and windows see higher increase in non-flaky than other targets, as previously disabled tests have been enabled

[apostasie]

I wish we could automate data collection and have some graphs generated automatically.