Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strengthen mutable_set_v5 expect test against timing differences #415

Merged
merged 3 commits into from
Nov 22, 2023

Conversation

jmid
Copy link
Collaborator

@jmid jmid commented Nov 22, 2023

The internal STM test mutable_set_v5.ml is currently run with --verbose which includes timing (0.0s) and subsequently diffed as an expect test. This creates the possibility of test failures due to a timing difference if it is run on a particular slow (CI) machine.

I just had this happen to me on the CI of an unrelated branch which happened to take 0.1s instead:
https://github.com/ocaml-multicore/multicoretests/actions/runs/6946194440/job/18897107446

STM test exception during next_state sequential failed with Random_next_state_failure as expected
STM test exception during next_state parallel failed with Random_next_state_failure as expected
diff --git a/_build/default/test/mutable_set_v5.expected b/_build/default/test/mutable_set_v5.output
old mode 100755
new mode 100644
index 84ab110..ef19a64
--- a/_build/default/test/mutable_set_v5.expected
File "test/mutable_set_v5.expected", line 1, characters 0-0:
/usr/bin/git --no-pager diff --no-index --color=always -u _build/default/test/mutable_set_v5.expected _build/default/test/mutable_set_v5.output
Command exited with code 1.
File "test/mutable_set_v5.expected", line 1, characters 0-0:
/usr/bin/git --no-pager diff --no-index --color=always -u _build/default/test/mutable_set_v5.expected _build/default/test/mutable_set_v5.output
Command exited with code 1.
+++ b/_build/default/test/mutable_set_v5.output
@@ -1,6 +1,6 @@
 
random seed: 229109553
 generated error fail pass / total     time test name
-
[ ]    0    0    0    0 /  100     0.0s STM sequential tests
[ ]    0    0    0    0 /  100     0.0s STM sequential tests (generating)
[✗]    2    0    1    1 /  100     0.0s STM sequential tests
+
[ ]    0    0    0    0 /  100     0.0s STM sequential tests
[ ]    0    0    0    0 /  100     0.0s STM sequential tests (generating)
[✗]    2    0    1    1 /  100     0.1s STM sequential tests
 
 --- Failure --------------------------------------------------------------------
 
Error: Process completed with exit code 1.

This little PR therefore removes the --verbose option for this expect test.

@jmid
Copy link
Collaborator Author

jmid commented Nov 22, 2023

CI summary for 790cc71:

Out of 63 workflows, 28 succeeded, 34 were canceled, 1 failed with a genuine failure

@jmid
Copy link
Collaborator Author

jmid commented Nov 22, 2023

CI summary for 73d172c:

Out of 63 workflows, 58 succeeded, 2 were cancelled, 3 failed with 1 genuine error, 1 false alarm, 1 unknown (CI flaky)

@jmid jmid merged commit a9ff236 into main Nov 22, 2023
48 of 53 checks passed
@jmid jmid deleted the strengthen-mutable-set-test branch November 22, 2023 16:43
@jmid
Copy link
Collaborator Author

jmid commented Nov 23, 2023

CI summary for merge to main:

  • Cygwin trunk / part1 failed with what looks like a network issue: InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebExc
  • freebsd-amd64-5.1 timed out during 12444.9s Lin Out_channel test with Domain Long Out_channel shrinking runs cause timeout #378

Out of 38 workflows, 2 failed with 1 borderline testsuite/ocaml5 issue, and 1 CI network issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant