[wdspec] /webdriver/tests/bidi/browsing_context/reload/ is flaky in Chromium

[OrKoN]

See https://github.com/web-platform-tests/wpt/pull/41041/checks?check_run_id=15094630093

It appears that reload tests are flaky in Chromium.

• reload.py
• frame.py

[OrKoN]

cc @thiagowfx

[whimboo]

I've noticed the same today and did some quick checks. The problem seems to happen only when more than one test method in that module gets run. Running each test on its own doesn't produce this failure. It seems that some unexpected content might be sent by chromedriver?

This can be reproduced locally as well, but cannot tell more details because I'm not able to enable any debug/trace logs for chromedriver.

[thiagowfx]

@whimboo Do you observe the failure in Geckodriver?

[whimboo]

We do not have reload implemented yet. So all these tests are failing with an unknown operation error. That means I cannot test it at the moment. Are you able to reproduce it locally when running the following command with a Canary build of Chrome?

./wpt run chrome webdriver/tests/bidi/browsing_context/reload/reload.py --channel=nightly

[whimboo]

The problem here is most likely present because the initial navigation is not awaited for. As such the reload is started while the current page is still loading, which then can cause side-effects.

https://github.com/web-platform-tests/wpt/blame/c6e26d40406b88796028c912d3051a869a557647/webdriver/tests/bidi/browsing_context/reload/reload.py#L12-L13

I'm going to fix that on https://bugzilla.mozilla.org/show_bug.cgi?id=1830859, and I hope that this is enough here.

[thiagowfx]

Thanks for the investigation!

[whimboo]

FYI the updated tests will be available once #41841 has been merged.

[thiagowfx]

I made some comments there. Thanks!

[whimboo]

The PR has been merged now. Not sure if your wrapper already supports returning a NavigationResult instead of EmptyResult. If not maybe you can check if it is still flaky?

[thiagowfx]

Let's see: GoogleChromeLabs/chromium-bidi#1264

This version is still using EmptyResult.

[thiagowfx]

frame.py is still flaky. But reload.py seems to be OK now! (pending more runs to increase confidence).

See https://github.com/GoogleChromeLabs/chromium-bidi/actions/runs/6109757468/job/16581412407?pr=1265 and GoogleChromeLabs/chromium-bidi#1265

[whimboo]

@thiagowfx please note that from the log output it is not clear why or where it is timing out. Are these both tests or maybe only the cross-origin test case?

[thiagowfx]

My understanding is: same-origin seems to be executed first (as it's listed first), and the timeout happens there already.

[whimboo]

Ok, so it only happens in CI and you cannot reproduce locally? If that is the case do you have a chance to somehow enable debug/trace logging for the tests? Where exactly does it hang/timeout?

[thiagowfx]

Running it locally in our mapper repo (npm run wpt -- webdriver/tests/bidi/browsing_context/reload/frame.py) seems to consistently pass. This is a CI-only failure, that's correct.

I am not quite sure how to debug it. I'll try. GoogleChromeLabs/chromium-bidi#1288

[thiagowfx]

I have the impression it's not flaky anymore: GoogleChromeLabs/chromium-bidi#1288. Hope not to be speaking too soon...will reopen if it turns out to be wrong.