Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests: Disable a fragile link e2e test until it is improved #13043

Merged
merged 1 commit into from
Dec 20, 2018

Conversation

gziolo
Copy link
Member

@gziolo gziolo commented Dec 20, 2018

Description

allows autocomplete suggestions to be selected with the mouse test from links spec fails very often on Travis. Let's disable it and try to improve until we enable it again.

When it fails, this is what is printed on the console:

FAIL test/e2e/specs/links.test.js (57.317s)
● Links › allows autocomplete suggestions to be selected with the mouse
expect(received).not.toBeNull()
Received: null
300 | // Expect that clicking on the autocomplete suggestion doesn't dismiss the link popover.
301 | await firstSuggestion.click();
> 302 | expect( await page.$( '.editor-url-popover' ) ).not.toBeNull();
| ^
303 |
304 | // Expect the url input value to have been updated with the post url.
305 | const inputValue = await page.evaluate( () => document.querySelector( '.editor-url-input input[aria-label="URL"]' ).value );
at Object.toBeNull (test/e2e/specs/links.test.js:302:55)
at tryCatch (node_modules/regenerator-runtime/runtime.js:62:40)
at Generator.invoke [as _invoke] (node_modules/regenerator-runtime/runtime.js:288:22)
at Generator.prototype.(anonymous function) [as next] (node_modules/regenerator-runtime/runtime.js:114:21)
at asyncGeneratorStep (test/e2e/specs/links.test.js:15:103)
at _next (test/e2e/specs/links.test.js:17:194)

By the way, we should double check it works as expected at the same time :)

Edit: In my tests, it didn't regress. It would be nice to discover why this test doesn't work as intended with Puppeteer.

@gziolo gziolo added the [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. label Dec 20, 2018
@gziolo gziolo self-assigned this Dec 20, 2018
@gziolo gziolo requested review from talldan and a team December 20, 2018 14:46
Copy link
Member

@tofumatt tofumatt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm cool with skipping unreliable tests. It's not ideal, but I'd rather us trust the test suite and respect failures when they happen than spend time restarting Travis thrice before accepting the failure as legitimate 👍

@gziolo
Copy link
Member Author

gziolo commented Dec 20, 2018

Yes, I had to restart Travis jobs like 10 times today because of this particular test.

@gziolo gziolo merged commit 5ca3383 into master Dec 20, 2018
@gziolo gziolo deleted the update/skip-fragile-e2e-test branch December 20, 2018 15:06
@gziolo
Copy link
Member Author

gziolo commented Dec 20, 2018

Opened a follow-up issue: #13044.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants