Failing to connect() to running Firefox via web-ext

[lfilho]

Problem

Ultimately what I want is to programatically test a WebExtension i'm writing with something like jest. Since sideloading doesn't work since FF 73 (https://blog.mozilla.org/addons/2019/10/31/firefox-to-discontinue-sideloaded-extensions/). I can't see a way of automatically installing an extension in my dev environment. web-ext seems to be the only option. Hence I need to connect puppeteer (or playwright) to the FF that web-ext sping up, so finally I could use it with jest to run my automated tests.

I'm trying to use puppeteer to connect to web-ext's Firefox as suggested in #5532 (comment) . Didn't work.

Steps to reproduce

Tell us about your environment:

• Puppeteer version: 3.3.0 (puppeteer-core)
• Firefox version: 76.0.1
• Platform / OS version: macos mojave / 10.14.6
• URLs (if applicable):
• Node.js version: 13.5.0 and 14.4.0

What steps will reproduce the problem?

Save the below to a file an run node my-file.js:

import child_process from 'child_process';
import webExt from 'web-ext';
import pptr from 'puppeteer-core';

(async () => {
  webExt.default.util.logger.consoleStream.makeVerbose();
  const runningInfo = await webExt.default.cmd
    .run(
      {
        sourceDir: `${process.cwd()}/src`,
      },
      {
        shouldExitProgram: false,
      }
    )
    .then((runner) => runner.extensionRunners[0].runningInfo);

  child_process.execSync('sleep 5');

  const browserURL = `http://127.0.0.1:${runningInfo.debuggerPort}`;
  const browser = await pptr.connect({
    browserURL,
  });

  const context = await browser.newContext();
  const page = await context.newPage();

  await page.goto('https://example.org');
})();

What is the expected result?

1. Firefox would open (web-ext)
2. Puppeteer would connect to it
3. Puppeteer would navigate to example.org

What happens instead?

1. Firefox opens (web-ext)
2. Puppeteer error out (see below)

Error:

(node:77010) UnhandledPromiseRejectionWarning: Error: Failed to fetch browser webSocket url from http://127.0.0.1:58779/json/version: Parse Error: Expected HTTP/
    at Socket.socketOnData (_http_client.js:507:22)
    at Socket.emit (events.js:315:20)
    at addChunk (_stream_readable.js:302:12)
    at readableAddChunk (_stream_readable.js:278:9)
    at Socket.Readable.push (_stream_readable.js:217:10)
    at TCP.onStreamRead (internal/stream_base_commons.js:186:23)

What else I tried:

• ws://127.0.0.1:${runningInfo.debuggerPort} instead of http://127.0.0.1:${runningInfo.debuggerPort}
• browserWSEndpoint as a parameter to connect() instead of browserURL
• Specifying a firefox nightly binary

Full log for the above script:

node test.js
[cmd/run.js][info] Running web extension from /Users/santoslg/workspace/ddg-test-project/src
[util/manifest.js][debug] Validating manifest at /Users/santoslg/workspace/ddg-test-project/src/manifest.json
(node:77010) Warning: Accessing non-existent property 'cat' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
(node:77010) Warning: Accessing non-existent property 'cd' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'chmod' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'cp' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'dirs' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'pushd' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'popd' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'echo' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'tempdir' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'pwd' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'exec' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'ls' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'find' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'grep' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'head' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'ln' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'mkdir' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'rm' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'mv' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'sed' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'set' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'sort' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'tail' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'test' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'to' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'toEnd' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'touch' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'uniq' of module exports inside circular dependency
(node:77010) Warning: Accessing non-existent property 'which' of module exports inside circular dependency
[extension-runners/firefox-desktop.js][debug] Creating new Firefox profile
[firefox/index.js][debug] Running Firefox with profile at /var/folders/fm/51pbdqt51bgf480pk0r1w4sdthm6sd/T/012a10fa-052d-4e8a-81c0-fac715ea753e
[firefox/index.js][debug] Executing Firefox binary: /Applications/Firefox.app/Contents/MacOS/firefox-bin
[firefox/index.js][debug] Firefox args: -start-debugger-server 58779 -foreground -no-remote -profile /var/folders/fm/51pbdqt51bgf480pk0r1w4sdthm6sd/T/012a10fa-052d-4e8a-81c0-fac715ea753e
[firefox/index.js][info] Use --verbose or open Tools > Web Developer > Browser Console to see logging
[firefox/remote.js][debug] Connecting to the remote Firefox debugger
[firefox/remote.js][debug] Connecting to Firefox on port 58779
[firefox/remote.js][debug] Retrying Firefox (0); connection error: Error: connect ECONNREFUSED 127.0.0.1:58779
[firefox/remote.js][debug] Connecting to Firefox on port 58779
[firefox/remote.js][debug] Retrying Firefox (1); connection error: Error: connect ECONNREFUSED 127.0.0.1:58779
[firefox/remote.js][debug] Connecting to Firefox on port 58779
[firefox/remote.js][debug] Retrying Firefox (2); connection error: Error: connect ECONNREFUSED 127.0.0.1:58779
[firefox/remote.js][debug] Connecting to Firefox on port 58779
[firefox/index.js][debug] Firefox stdout: 1592034808630	[email protected]	WARN	Loading extension '[email protected]': Reading manifest: Invalid extension permission: networkStatus
[firefox/index.js][debug] Firefox stdout: 1592034808648	[email protected]	WARN	Loading extension '[email protected]': Reading manifest: Invalid extension permission: mozillaAddons
1592034808648	[email protected]	WARN	Loading extension '[email protected]': Reading manifest: Invalid extension permission: telemetry
[firefox/index.js][debug] Firefox stdout: 1592034808648	[email protected]	WARN	Loading extension '[email protected]': Reading manifest: Invalid extension permission: resource://pdf.js/
1592034808648	[email protected]	WARN	Loading extension '[email protected]': Reading manifest: Invalid extension permission: about:reader*
[firefox/remote.js][debug] Retrying Firefox (3); connection error: Error: connect ECONNREFUSED 127.0.0.1:58779
[firefox/remote.js][debug] Connecting to Firefox on port 58779
[firefox/remote.js][debug] Retrying Firefox (4); connection error: Error: connect ECONNREFUSED 127.0.0.1:58779
[firefox/remote.js][debug] Connecting to Firefox on port 58779
[firefox/index.js][debug] Firefox stdout: Started devtools server on 58779
[firefox/remote.js][debug] Retrying Firefox (5); connection error: Error: connect ECONNREFUSED 127.0.0.1:58779
[firefox/remote.js][debug] Connecting to Firefox on port 58779
[firefox/remote.js][debug] Connected to the remote Firefox debugger on port 58779
[firefox/remote.js][debug] installTemporaryAddon: {"addon":{"id":"b72edf14bd71f7760d2c00a1609c9e956f322efb@temporary-addon","actor":false},"from":"server1.conn0.addonsActor2"}
[firefox/remote.js][info] Installed /Users/santoslg/workspace/ddg-test-project/src as a temporary add-on
[cmd/run.js][info] The extension will reload if any source file changes
[util/file-filter.js][debug] Resolved path **/*.xpi with sourceDir /Users/santoslg/workspace/ddg-test-project/src to /Users/santoslg/workspace/ddg-test-project/src/**/*.xpi
[util/file-filter.js][debug] Resolved path **/*.zip with sourceDir /Users/santoslg/workspace/ddg-test-project/src to /Users/santoslg/workspace/ddg-test-project/src/**/*.zip
[util/file-filter.js][debug] Resolved path **/.* with sourceDir /Users/santoslg/workspace/ddg-test-project/src to /Users/santoslg/workspace/ddg-test-project/src/**/.*
[util/file-filter.js][debug] Resolved path **/.*/**/* with sourceDir /Users/santoslg/workspace/ddg-test-project/src to /Users/santoslg/workspace/ddg-test-project/src/**/.*/**/*
[util/file-filter.js][debug] Resolved path **/node_modules with sourceDir /Users/santoslg/workspace/ddg-test-project/src to /Users/santoslg/workspace/ddg-test-project/src/**/node_modules
[util/file-filter.js][debug] Resolved path **/node_modules/**/* with sourceDir /Users/santoslg/workspace/ddg-test-project/src to /Users/santoslg/workspace/ddg-test-project/src/**/node_modules/**/*
[watcher.js][debug] Watching for file changes in /Users/santoslg/workspace/ddg-test-project/src
[extension-runners/index.js][info] Press R to reload (and Ctrl-C to quit)
(node:77010) UnhandledPromiseRejectionWarning: Error: Failed to fetch browser webSocket url from http://127.0.0.1:58779/json/version: Parse Error: Expected HTTP/
    at Socket.socketOnData (_http_client.js:507:22)
    at Socket.emit (events.js:315:20)
    at addChunk (_stream_readable.js:302:12)
    at readableAddChunk (_stream_readable.js:278:9)
    at Socket.Readable.push (_stream_readable.js:217:10)
    at TCP.onStreamRead (internal/stream_base_commons.js:186:23)

[lfilho]

Also posted this at mozilla/web-ext#1927

[stale]

We're marking this issue as unconfirmed because it has not had recent activity and we weren't able to confirm it yet. It will be closed if no further activity occurs within the next 30 days.

[stale]

We are closing this issue. If the issue still persists in the latest version of Puppeteer, please reopen the issue and update the description. We will try our best to accomodate it!