How do I know if 2 threads are running concurrently?

[krmahadevan]

There are some tests within the TestNG codebase that try to test if test methods were started concurrently by TestNG.

Sample failure: https://github.com/testng-team/testng/actions/runs/8391184326/job/22980937095?pr=3100

Failing test details from gradle build scan: https://scans.gradle.com/s/m4kgqbuhsc4xo/tests/task/:testng-core:test/details/test.factory.issue326.IssueTest/testToCheckParallelExecutionOfInstancesWithOneThreadPerInstance?top-execution=1

Error details:

java.lang.AssertionError: |  
-- | --
  | Expecting actual: |  
  | 1096L |  
  | to be less than or equal to: |  
  | 1000L

The failing test: https://github.com/testng-team/testng/blob/master/testng-core/src/test/java/test/factory/issue326/IssueTest.java#L16

The piece of code that is causing the flakiness

    // Ensure that the difference in start times for the methods in both instances is <= 1000 ms.
    long diff = computeDiffInStartTimeFor(listener.getResults(), SampleTestClass.FREDDY);
    assertThat(diff).isLessThanOrEqualTo(1000);
    diff = computeDiffInStartTimeFor(listener.getResults(), SampleTestClass.BARNEY);
    assertThat(diff).isLessThanOrEqualTo(1000);

The test is trying to basically ascertain if 2 test methods started at the same time. The logic that's included in the test works most of the time, but the moment the JVM gets stressed out, it becomes flaky 🤦

TZ="Pacific/Chatham" ./gradlew build -Duser.country=RU -Duser.language=ru \
-Ptestng.test.extra.jvmargs="-Duser.country=RU -Duser.language=ru -XX:+UnlockDiagnosticVMOptions -XX:+StressGCM -XX:+StressLCM -XX:+StressIGVN -XX:+StressCCP" \
-PtestDisableCaching="JIT randomization should not be cached"

@juherr @vlsi - Any suggestions on how to modify the test and ensure it gets to pass all the time? Should I bump up the expected tolerance value from 1,000 ms to say 5,000 ms (i.e., the test would then check if the 2 concurrent test methods had a start time diff of max 5 seconds). I am not able to think of any other way. I think there maybe more such flaky tests which are going to keep failing randomly whenever the JVM gets run with the above flags.

[juherr]

The test is a performance test.
If you increase the expected max delay, you assume we expect less performance.

Instead you can find where the time is wasting and try to improve it.

Both options are valid and have to make this choice everytime a performance test is failing.