Testing: Enable or remove skipped e2e tests

[gziolo]

We have recently disabled a few e2e tests to make Travis job pass consistently. Related PRs: #15059, #14986, #15211.

This issue is opened to coordinate efforts to either fix or remove all skipped e2e tests:

• packages/e2e-tests/specs/block-deletion.test.js
  • results in two remaining blocks and positions the caret at the end of the second block (disabled in Testing: Skip unreliable end-to-end tests #15059), more details in Testing: Skip unreliable end-to-end tests #15059 (comment):

    I've spent some time debugging this specific failure. In doing so, I observe the tedium of debugging on behalf of these other popular plugins (and not knowing which specific one could be a culprit). One lead I have is in the fact that you can observe in standard use of the block settings menu, the "Convert to Reusable Block" button appears only after a short delay (because it fetches user permission to create reusable block). I wonder that there may be a race condition similar to what we've seen previously with animations and the difference in time between when Puppeteer calculates a position and when it enacts a specific interaction (Improve the e2e tests involving animations and waitForAnimation #13739). In this case, I wonder if it calculates the position of "Remove Block", then clicks, but in the time between those two steps the "Add to Reusable Blocks" button appears in the exact location the "Remove Block" button had been. This could explain the mismatch in the snapshot where, for no discernible reason, a reusable block exists in the content.

• packages/e2e-tests/specs/embedding.test.js
  • should transform from video to embed block when YouTube URL is pasted (disabled in Testing: Skip unreliable end-to-end tests #15059)
  • should transform from image to embed block when Instagram URL is pasted (disabled in Testing: Skip unreliable end-to-end tests #15059)
  • should transform from audio to embed block when Soundcloud URL is pasted (disabled in Testing: Skip unreliable end-to-end tests #15059)
• packages/e2e-tests/specs/links.test.js
  • allows autocomplete suggestions to be selected with the mouse (old issue Tests: Enable a disabled fragile link e2e test #13044, previous attempt to fix Fix unreliable link test #13161, disabled in Testing: Skip unreliable end-to-end tests #15059)
  • allows autocomplete suggestions to be navigated with the keyboard (disabled in Tests: Disable randomly failing e2e tests #15211)
• packages/e2e-tests/specs/nux.test.js - the following tests were never enabled even though blocking issues were resolved, they should be removed if they no longer are relevant
  • should show tips as disabled if all tips have been shown
  • should reset tips if all tips have been shown and show tips was unchecked
  • should show tips if "Show tips" was disabled on a draft and then enabled
• packages/e2e-tests/specs/plugins/wp-editor-meta-box.test.js (disabled in Tests: Disable randomly failing e2e tests #15211), more info in Tests: Disable randomly failing e2e tests #15211 (comment):

  sounds familiar to issues I'd seen in the past where sometimes TinyMCE takes some time to initialize, so "click" in the test to activate the field might be delayed. It could possibly be improved by ensuring that whatever DOM changes might occur to indicate that the initialization is complete are accounted for in e.g. page.waitForSelector.

• packages/e2e-tests/specs/taxonomies.test.js

  • should be able to open the tags panel and create a new tag if the user has the right capabilities (disabled in Tests: Disable randomly failing e2e tests #15211), more details in Tests: Disable randomly failing e2e tests #15211 (comment):

    sounds quite similar to issues we'd had recently with the need for this permissions information to be fetched from the REST API: Reusable Blocks: Preload user permissions #15061

    Given that taxonomies are loaded in the sidebar and aren't mandatory on the first load of the editor, it might be enough to refactor test to ensure that it waits until data is loaded with REST API.

Reproducing issues

The biggest challenge we encountered in the past was the fact that all the aforementioned tests were failing randomly on Travis but it was nearly impossible to reproduce it locally. There is a plan to add some debugging capabilities to npm run test-e2e command but for the time being, I recommend using the following patch to slow down CPU and network to simulate conditions similar to Travis. You still might need to run the same test a few times to be able to see failure.

diff --git a/packages/e2e-tests/config/setup-test-framework.js b/packages/e2e-tests/config/setup-test-framework.js
index b14717620..f8e70c62e 100644
--- a/packages/e2e-tests/config/setup-test-framework.js
+++ b/packages/e2e-tests/config/setup-test-framework.js
@@ -198,6 +198,20 @@ async function runAxeTestsForBlockEditor() {
 // other posts/comments/etc. aren't dirtying tests and tests don't depend on
 // each other's side-effects.
 beforeAll( async () => {
+	const client = await page.target().createCDPSession();
+	await client.send( 'Network.emulateNetworkConditions', {
+		// Whether chrome should simulate
+		// the absence of connectivity
+		offline: false,
+		// Simulated download speed (bytes/s)
+		downloadThroughput: 1.5 * 1024 * 1024 / 8,
+		// Simulated upload speed (bytes/s)
+		uploadThroughput: 750 * 1024 / 8,
+		// Simulated latency (ms)
+		latency: 40,
+	} );
+	await client.send( 'Emulation.setCPUThrottlingRate', { rate: 8 } );
+
 	capturePageEventsForTearDown();
 	enablePageDialogAccept();
 	observeConsoleLogging();

It is super useful to run e2e tests in watch mode and with the real browser:

npm run test-e2e:watch -- --puppeteer-interactive --puppeteer-slowmo=0