Cannot view reports for draft test plans

[mcking65]

Issues #649 and #719 include requirements that are intended to enable users to access reports from test plans that are in draft review. These reports are intentionally not available from the reports page because we do not want them widely visible, however we still need to be able to review them before advancing a plan to candidate review.

After a report from a test plan that is in draft review is marked final, it should be available from either the report status dialog or the test plan versions page. Those report links are not provided as specified in issues #649 and #719.

In issue #649, the description of the Report Status column is:

• If the report is complete, a link to the report with text "Report Completed on DATE"
• If report is missing, the string "Missing", and if user is admin, button "Add to Test Queue"
• If report is not complete and not missing and not in test queue, the string "None", and if user is admin, "Add to Test Queue" button
• If report is in the test queue, the string "X% complete by Y testers with Z conflicts"
• Note: Percent complete = NUMBER_COMPLETED_TESTS_BY_ALL_TESTERS / (NUMBER_ASSIGNED_TESTERS * NUMBER_TESTS_IN_PLAN) * 100

Currently, if a test plan is in draft review and a report is complete, the report status column is showing the value that it should show if the test plan run for that report were still in the test queue and not yet marked final. For example, it shows:

100% complete by 2 testers with 0 conflicts

Here is a screen shot of the report status dialog for meter that I get after finalizing all required reports. This is the dialog when there are 0 runs of the meter test plan in the test queue.

Similarly, as specified in #719, the test plan versions page for meter does not include a link to view reports under the heading with text:

V22.08.05 Draft on Sep 22, 2023

[mcking65]

Re-opening; I can't find the links to reports for test plans in the draft review phase.

Looking at prod, the toggle button test plan V24.03.12 is in draft review and some of the required reports are complete. Here is the data management page.

Here is the report status dialog. Note that JAWS+Chrome and NVDA+Chrome have complete reports, but the status does not include a link to the reports.

[alflennik]

Hi @mcking65 thanks for putting this on my radar, I'll assign to Carmen so she can prioritize it.

[mcking65]

I don't know if this is related, but the status calculation in the report status dialog is also incorrect. For example, at this time, the test queue shows that the Toggle Button V24.03.12 run for JAWS/Chrome has:

1. IsaDC 2 of 8 tests complete
2. hadirangin 2 of 8 tests complete

Yet, the report status dialog shows the status for JAWS/Chrome as:

100% complete by 2 testers with 0 conflicts

It should be:

25% complete by 2 testers with 0 conflicts

Here is the report status dialog screenshot:

Here is the test queue screenshot:

[alflennik]

Hi there @mcking65, I think I can weigh in regarding this comment:

Looking at prod, the toggle button test plan V24.03.12 is in draft review and some of the required reports are complete.

I can verify that the view generated reports link is not showing up for V24.03.12 of Toggle Button, so I ran a query on production to verify that there are final reports for that test plan.

I found that there are 12 reports for Toggle Button but only three reports for Toggle Button V24.03.12. There are reports for JAWS / Chrome, NVDA / Chrome and VoiceOver / Safari. None of those three reports are final, and all three are currently found in the test queue.

Therefore I'm currently concluding that this particular feature is still working. But please let me know if you have any suspicions or evidence otherwise and I'd be more than happy to investigate.

Also I should mention that I ran some tests locally to make sure the button is behaving itself. I completed a report and it did show up as intended in the test plan versions page.

In case you or anyone else wants to replicate those findings about the data currently on production, the query is below and can be run on the API playground on https://aria-at.w3.org/api/graphql

query {
  testPlanReports {
    testPlanVersion {
      title
      versionString
    }
    at {
      name
    }
    browser {
      name
    }
    isFinal
  }
}

[mcking65]

@howard-e

In staging, the calculation of percent complete is now correct in the report status dialog.

@alflennik, in the report status dialog, after a report for a draft test plan is marked final, the status column of the table in the dialog still does not include the link specified in #649. In that issue, it says that the report status column should contain:

If the report is complete, a link to the report with text "Report Completed on DATE"

For example, in staging, I marked the JAWS/Chrome report for Command Button Test Plan V23.12.13 final. In the report status dialog, The status for that report is still:

100% complete by IsaDC with 0 conflicts

But it should be:

Report Complete on March 23, 2024

The other required access point for the report on the Test Plan Version History page is working.

[mcking65]

@ccanash

Per my prior comment, I think this issue is not yet resolved because links to reports on draft test plans are still not available from the report status dialog.

However, the open problems in this issue do not need to block deployment of the patches on Monday march 25. We can keep this issue open and target full resolution in the next deployment.

[mcking65]

Yay!! Tested in staging, and it is working!!! Thank you!