build: ensure order of tests using the @DataProvider annotation

[bitcoder]

Review test code, namely of the datadriven tests, to ensure the order of tests and avoid flakiness on the build.

See:

• @Factory with dataProvider changes order of iterations testng-team/testng#799
• ensure order of tests whenever using a dataProvider testng-team/testng#3109

[itkhanz]

just curious as to how the order of tests is effecting the Xray extension?

regardless of the order of execution, the last data/example {5, true} will always fail. Also in the test method body, we are just attaching the same attachment as attribute for all the examples.

xray-testng-extensions/src/test/java/app/getxray/xray/testng/tests/DataDrivenExamples.java

Line 39 in 828fe56

result.setAttribute("attachments", attachments);

Perhaps I wasn't able to spot whats the linkage of this issue with build/release? Could you please elaborate a little whats the issue here so maybe i can also take a detailed look to investigat the problem

[bitcoder]

• Example of failed build: https://github.com/Xray-App/xray-testng-extensions/actions/runs/8598517318/job/23559579756#step:5:1270
• Example of successful build: https://github.com/Xray-App/xray-testng-extensions/actions/runs/8647824008

The problem is with this datadriven scenario.
TestNG doesn't ensure the order of the created tests... so sometimes the build passes, sometimes it fails because this test fails.

[itkhanz]

Thank you for sharing the detailed issue. So if I understood correctly, the build failed because of the random order of iterations in which the TestNG executes the tests from DataProvider.

We have hardcoded expected results in our test, and we expect the actual results iterations to be in ascending order (as in Data Provider). We should be able to solve this problem by implementing custom comparator for our use iterations object which sorts the iterations based on number parameter's integer value before performing assertions.

xray-testng-extensions/src/test/java/app/getxray/xray/testng/tests/XrayJsonReportTests.java

Line 214 in 828fe56

JSONArray iterations = (JSONArray)actualTest.getJSONArray("iterations");

Here is how the solution may look like:

JSONArray iterations = (JSONArray) actualTest.getJSONArray("iterations");

// Sort the iterations based on the "number" parameter value
iterations.sort(new Comparator<Object>() {
    @Override
    public int compare(Object o1, Object o2) {
        JSONObject iteration1 = (JSONObject) o1;
        JSONObject iteration2 = (JSONObject) o2;

        int number1 = iteration1.getJSONObject("parameters")
                           .getJSONArray("parameters")
                           .getJSONObject(0)
                           .getInt("value");

        int number2 = iteration2.getJSONObject("parameters")
                           .getJSONArray("parameters")
                           .getJSONObject(0)
                           .getInt("value");

        return Integer.compare(number1, number2);
    }
});

Now that the iterations is sorted, the next set of assert statements should be passed our iterations is now in order.

I look forward to your feedback if you could please try this solution and let me know if it works.

[bitcoder]

Thanks @itkhanz ; I applied a workaround based on that code snippet with some adjustments.

[bitcoder]

released in 0.2.0