Extension testing in browser / puppeteer

[fregante]

I'll post here issues related to testing, so they can be tracked in one place for the time being.

---

I did some testing while working on #398, you can see that in fregante/content-scripts-register-polyfill#17

Potential Puppeteer issues:

• Puppeteer can't run code in extension contexts (background, content scripts, etc) so we can only command the pages as a user (which might be enough) Support extensions execution contexts puppeteer/puppeteer#1215
• Firefox support in Puppeteer is probably very buggy (one example of a basic feature: Page.goto and Page.evaluate don't resolve in moz-extension pages puppeteer/puppeteer#6616)
• Another example is that Puppeteer Firefox does not support extension loading so it needs to be done through web-ext (not impossible but messy) Firefox extensions with v2.1.0 puppeteer puppeteer/puppeteer#5532 (comment)
• Puppeteer will likely require Xvfb to run on CI (not a huge issue, I have a workflow set up somewhere) Chrome extension is not loaded on headless browser puppeteer/puppeteer#659
• Since the browser to install is determined at npm install time (via ENV, without package/lockfile changes), it might be difficult/impossible to have both browsers installed for testing (a "solution" would be to test Firefox just on CI) (EDIT: may be possible)

When the time comes to discuss testing more in detail, I'll probably look into alternatives to jest-puppeteer (e.g. Selenium?)

[twschiller]

I see the Firefox limitation as the bigger issue. Does Puppeteer support Safari and the other Chromium browsers?

For the extension context limitation, the way I've approached similar limitations in the past is to include custom error tracking in test builds. For our situation it might be:

• In the background page and content script, register listeners for top-level errors and promise rejections
• Expose an "API" that returns the number/list of errors encountered. For example, by replacing the page content with an error message. The API should also probably let you reset the error count
• Puppeteer (or Selenium) can check for the presence of the error indicator on the page and fail the test if it sees one

One benefit to Selenium is there's plenty of services like Browserstack that support it. However, there's probably services for Cypress et al as well now too

Update 6/26: BrowserStack documentation on testing with extensions: https://www.browserstack.com/docs/automate/selenium/add-plugins-extensions-remote-browsers

[fregante]

Does Puppeteer support Safari and the other Chromium browsers?

Chromium, ish… There seems to be this, but last updated in 2019: https://github.com/EasyWebApp/puppeteer-browser

https://github.com/microsoft/playwright is the cross-browser alternative to Puppeteer, but there's even fewer documentation about extensions here.

Maybe hacking around web-ext run can give some results for launching these browsers, but again this needs research.

Safari is in its own league, I published 3/4 Safari extensions but the development/testing process is not clear yet, there's nearly zero documentation and you need to go through Xcode to do anything. Safari testing is more of a Mac developer task than a web extension one.

[twschiller]

• FYI: the PrivacyBadger extension uses Selenium and their tests are open source: request: APIs and infrastructure to simplify cross-browser automated testing of extensions w3c/webextensions#19 (comment)
• 1Password might also use Selenium? https://www.youtube.com/watch?v=S0TdXgh03cI

[fregante]

Just a small note: it appears that everyone is testing the extensions by loading an intermediary page on chrome-extension:// and then using runtime.getBackgroundPage(), which returns the window of said page, but you won't be able to run functions directly in that context. I suppose we could expose the lifted functions. I think the same applies to the content scripts.

Can you post a few examples of what you'd like to test? Maybe it will be easier for me to work towards that instead of just finding what's not possible. A full "first test" example would be great.

[twschiller]

@fregante thanks for the note. With the browser/puppeteer tests, at the beginning it's primarily going to be system tests / smoke tests to expose regressions in behavior (especially around corner cases like frames, etc.). I don't think we'll be calling background/content script methods directly

Example smoke tests

• Load a page that PB has access w/ an extension activated that shows an alert on page load (or something else to indicate PB was able to run the content script). Did the browser show the alert?
• Load a page that PB does not have access to and click the PB icon in the toolbar. Does the sidebar open? Does the alert now show (indicating PB was able to run the "trigger" extensions)

Systems tests

• PixieBrix has support for running actions in other tabs (e.g., "opener", "target"), etc. So testing a brick that opens a tab and then runs an action in target and shows the result in the current tab. Does the correct result come back?

Long-term, I also imagine the testing infrastructure might be used to detect breakages to foundations we've built for popular sites

[twschiller]

The two regressions in this ticket would also be good ones to do: #671

As part of the testing harness we will want/need to:

• Add data-test- attributes to some components to make them easier to target with Selenium
• Expose some programmatic methods for getting the installed bricks into the right state, e.g., programmatically de-activating all bricks
• Create an email/password-based login for the test service account. (It's not worth trying automate Google logins)
• ... I'll brain dump more here when I have a chance

[fregante]

I think none of what you suggested is possible simply because:

• we can't add permissions without user interaction
• we can't click the browser action icon thus triggering activeTab or opening the sidebar

Maybe we can get around the permission issue by adding the hosts in manifest.json, but that changes the loading behavior altogether and our expectations of our permissions and loading.

I will still try, maybe the chromedriver will automatically accept any permission.request() call. For the second part however I'm 99% sure it's simply not possible to control the "browser chrome" (UI outside the web page)

[twschiller]

I think none of what you suggested is possible

Gotcha, I think I understand that restriction better now. The interactions we have with the browser chrome are:

• Permissions prompts
• Context menus
• Clicking the tool bar
• Opening the DevTools and switching to the PixieBrix tab
• Small subset of bricks: prompt, alert, etc.

In the future we could potentially test these using an RPA tool (like UiPath, etc.). That's a small surface area though, that can be handled by manual testing for now (we will just need to write up some test checklists for the wiki)

There's a very large testing surface area that don't require these. For example, instead of using the window alert brick, using another brick that affects the interface (e.g., highlight). So let's focus on those

Maybe we can get around the permission issue by adding the hosts

Yes, I suspect we might have to do this. Fudging the permissions is totally reasonable to do.

Thanks for commenting on w3c/webextensions#19! As we encounter more limitations, let's continue to add to the list there

Initial Tests

@fregante I think we should be able to do both of the following? The don't require using the browser chrome, but may require granting permissions via the manifest when running the test

1. Load a page that PB has access w/ an extension activated that highlights an element on page load (using a trigger). Did the element get highlighted? Are there any errors logged to the console (for content script, background page, etc.?)
2. PixieBrix has support for running actions in other tabs (e.g., "opener", "target"): https://docs.pixiebrix.com/developer-guide/multi-page-automation. So testing a blueprint that opens a tab (using the open-tab brick) and then runs an action in target and shows the result in the current tab (e.g., using the "set input" brick). Does the correct result come back?

[fregante]

Some notes:

• Testing the extension means building it and then loading it into a browser via Jest + Selenium
• This means that any customization/mocking of the manifest or internal modules can only be done once, unless you want to build + run separate tests repeatedly lasting 2.5 minutes each locally or 7 minutes each on CI
• It seems that all browser testing ever happens as a "spectator" of the browser, without any ability to run "test code" in the browser (i.e. code specified in .test.js files) and thus even catching the console/errors seems rare or inexistent if we talk about background scripts. The only exception to this that I found is for easily-bundlable packages in order to unit test them, example: https://github.com/juliangruber/tape-run

Therefore, assuming a single build, to test a brick:

1. Have a local server with custom pages that will be the target of the tests
2. Build the extension
   • with localhost in SERVICE_URL
   • with settings/bricks ready to run
   • already "logged in"
3. Load each tab on specific "localhost" pages to run the bricks

---

Do you think that with all these limitations and difficulty in the setup Selenium is still worth setting up (if UiPath is an alternative)?

I think our options (which can be combined) are:

• The existing unit testing
• The partial Selenium tests we're talking about here, to load the whole extension and see if it works
• End-to-end tests with UiPath, from installation, to brick editing, permissions, sidebar, actions, …

An additional option I could try would be to:

• build a mocked environment in order to run the whole extension as if it was a regular unit test, without actually running it in a browser

I'm not sure exactly what this would entail, but we're less likely to run into walls and hard limitations of Selenium/ChromeDriver.

Maybe we can do unit testing + UiPath + mocked whole-extension tests.

---

My knowledge about testing is extremely limited so correct me if I'm talking nonsense 😃

[fregante]

I might have found ways to:

• execute code in the page context (not extension context, unless we load an extension:// page): https://stackoverflow.com/a/21125803/288906
• get logging and errors: https://advancedweb.hu/detecting-errors-in-the-browser-with-selenium/

[twschiller]

To help move things along on the testing initiative, I started a POC branch for running Selenium tests as part of CI: #711

I invited your @pixiebrix.com email to our BrowserStack account. @fregante Could you take a swing at loading the extension into the remote browser (Chrome first) in the PR?

In the meantime, I'll also start writing up a proposal for the other details. For example:

• Since the server is not open source (yet), we'll target the staging server which is always running CD of the main branch
• I'll provision and account for performing email/password authentication with the server

It seems that all browser testing ever happens as a "spectator" of the browser

It uses the WebDriver API, so we can execute scripts, etc. So many things are possible, they're just annoying

This means that any customization/mocking of the manifest or internal modules can only be done once

I'm OK with this. For E2E tests, there's not much need to mock internal modules/responses. For the manifest, I think the only difference is pre-provisioning permissions?

build a mocked environment in order to run the whole extension

This is going to be more trouble than it's worth, because we'd have to maintain parity with all the vendor quirks

I might have found ways to:

Great! - we'll be creating a library of helpers using tribal knowledge like this. The Privacy Badger repository and the 1Password conference talk linked above are have some good nuggets

My knowledge about testing is extremely limited so correct me if I'm talking nonsense

No worries! A lot of testing is framework-specific so there's a very large surface area and number of quirks to be aware of. We'll split the work into small chunks, as there will be places where it make sense for you to take the lead vs. where me (or someone else joining the team soon) should