Package request: playwright test

[teto]

Project description

The request is not necessarily to package this for nixpkgs but if anyone has already done it so that I can steal the derivation. I think the probability is high considering the recent hype around playwright (there was also a post on this yesterday https://discourse.nixos.org/t/running-playwright-tests/25655/2).

I am trying to help a coworker run "playwright test" but I dont know much about playwright or node. The executable brought by nix shell nixpkgs#playwright doesn't provide it and it's a python one.
Seems like the "true" playwright is in typescript https://github.com/microsoft/playwright and the test command is provided by a submodule @playwright/test . At some point I managed to make it work with node2nix on that last repo but it might have been a fluke since once I started calling the derivation from a flake, it tried to reach out to the network.

Metadata

• homepage URL: https://github.com/microsoft/playwright
• source URL: https://github.com/microsoft/playwright
• license: apache 2
• platforms: unix, linux, darwin

[teto]

We apparently have several (too many) solutions when it comes to packaging npm software:

1. https://github.com/svanderburg/node2nix
2. https://github.com/nix-community/dream2nix
3. https://github.com/nix-community/npmlock2nix
4. https://github.com/serokell/nix-npm-buildpackage
5. https://github.com/nix-community/napalm

I've tried a bunch of them and they all fail with different errors. As I've used node2nix in the past, I've decided to stick with it. From my understanding: playwright (PW) uses the recent "workspace" feature of package.json which isn't supported with node2nix yet (see PR svanderburg/node2nix#200) so you have to go into the "packages" folder and run node2nix on those packages separately or create your own hybrid folder.

I've managed to make playwright test work in an impure setting with some hacks (NODE_PATH, export PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS=true) but it's not extensively tested, trying to put it into a flake too at which point the install fails because playwright tries to install its patched browsers. Apparenty I need to use playwright-core instead microsoft/playwright#1587 . Btw the pkgs.playwright.browsers defined at
pkgs/development/python-modules/playwright/default.nix dont seem to be the patched versions, not sure if that matters or not (still noob to npm and playwright).

[karakunai]

At the moment, I've been looking for a solution myself as well and thanks to the experiments done by @voidus and @ludiosarchive , I've managed to get something that one might would call as a working setup although it's far from ideal. Here's how:

1. Install playwright outside Nix, and skip the browser installations.
2. Build the derivations for the browsers like how it's done here and here.
3. Symlink each browser derivations built to ~/.cache/ms-playwright/{browser_name}-{revision_number}/{chrome-linux | firefox | minibrowser-{gtk | wpe}}.
4. Define PLAYWRIGHT_SKIP_VALIDATE_HOST_REQUIREMENTS variable.
5. Run playwright.

Now, to the caveats:

1. The distributed binaries for chrome will refuse to work outside of the headless mode, but since they are mostly patching webkit and firefox, you may use the ones available from nixpkgs (e.g ungoogled-chromium), and then link the path through an environment variable which will be loaded from the playwright.config.ts. Something like this:

   projects: [
       {
           name: "chromium",
           use: {
               ...devices["Desktop Chrome"],
               launchOptions: {
                   executablePath: process.env.YOUR_VARIABLE_HERE,
               },
           },
       },
   ],

2. We have to link chromium to the ~/.cache/ms-playwright/chromium-{browser_revision} still since the settings above will only apply to the projects only, and the playwright's inspector would not recognize these so it'll check for the browser on the cache folder. Or, you may as well ignore point 1, and do this instead with the available chromium from nixpkgs.

3. Replacing firefox and webkit with the ones available from nixpkgs would never work, hence the idea of using the ones they have and build the derivations on the premises to match the current playwright version installed.

4. Webkit still looks for some dependencies despite the effort.

I might be able to put up some examples, but I'm still new to Nix myself, it might would take some days (promise).

[teto]

some notes to myself as:

• I had read on this tracker something about PLAYWRIGHT_BROWSERS_VERSION but it doesn't seem to be a thing.
• the browser version seems to be hardcoded in playwright-test so when you set wrapProgram $out/bin/playwright --set-default PLAYWRIGHT_BROWSERS_PATH "${pkgs.playwright.browsers}" you need to make sure that your playwright version matches the driver version of your nixpkgs (e.g., driverVersion in pkgs/development/python-modules/playwright/default.nix). node2nix was ignoring my constraint: "dependencies": { "@playwright/test": "^1.27.1", }, but "dependencies": { "@playwright/test": "1.27.1", }, correctly pinned it.

[teto]

I am at this stage

playwright test  test/init/init.test.ts 

Running 0 tests using 0 workers

Error: test/init/init.test.ts:4:16: Playwright Test did not expect test() to be called here.
Most common reasons include:
- You are calling test() in a configuration file.
- You are calling test() in a file that is imported by the configuration file.
- You have two different versions of @playwright/test. This usually happens
  when one of the dependencies in your package.json depends on @playwright/test.

   at test/init/init.test.ts:4

  2 |
  3 | test('init', async ({ page }) => {
> 4 |   await page.goto('/')
    |                ^
  5 | })
  6 |

    at addFatalError (/nix/store/kxj6nk73l450g2xffsv44n37ma3rzsmn-node-dependencies-jinko-e2e-1.0.0/lib/node_modules/@playwright/test/lib/globals.js:51:17)
    at TestTypeImpl._currentSuite (/nix/store/kxj6nk73l450g2xffsv44n37ma3rzsmn-node-dependencies-jinko-e2e-1.0.0/lib/node_modules/@playwright/test/lib/testType.js:70:34)
    at TestTypeImpl._createTest (/nix/store/kxj6nk73l450g2xffsv44n37ma3rzsmn-node-dependencies-jinko-e2e-1.0.0/lib/node_modules/@playwright/test/lib/testType.js:77:24)
    at /nix/store/kxj6nk73l450g2xffsv44n37ma3rzsmn-node-dependencies-jinko-e2e-1.0.0/lib/node_modules/@playwright/test/lib/transform.js:240:12
    at Object.<anonymous> (/home/teto/nova/dango/e2e-candidate/test/init/init.test.ts:4:16)
=================
 no tests found.
=================

and not sure how to debug this. the playwright tracker is useless apart from microsoft/playwright#13102 but not sure what to do. playwright-test is at version 1.30.1

[teto]

ok found a solution, I'll test the package for a while and upstream if satisfactory.

[FeliciousX]

how'd you go with packaging them? I would love to help test or review them 😄

[teto]

I call node2nix on the following node-packages.json

[
 {"@playwright/test": "1.30.0"}
]

then in my flake I have

   playwright-test-raw = (pkgs.callPackage ./default.nix { })."@playwright/test-1.30.0";

   playwright-test = playwright-test-raw.overrideAttrs(oa: {
         nativeBuildInputs = oa.nativeBuildInputs or [] ++ [
           pkgs.makeWrapper
         ];

         postInstall = ''
           # you need to set both the path and version else playwright looks into the wrong one
           wrapProgram $out/bin/playwright \
               --set-default PLAYWRIGHT_BROWSERS_PATH "${playwrightBrowsers}" \
               --prefix NODE_PATH : ${placeholder "out"}/lib/node_modules
         '';
       });

playwright-browsers is a fork of #223382 with versions that match my version of the node module. Note that "playwright-browsers" doesn't expose the browsers expected by playwirght (ie.,with the playwright patches) but the nixpkgs browsers in a custom folder that respects the folder hierarchy expected by playwright. From my experience and other nixpkgs issues: firefox is currently broken but chrome works for basic usage. I haven't had the chance yet to test the more exoctic scenarii. Took me a while to understand all this.

This looks like a popular tool so I expect it will keep improving if we can get a good enough base to start with.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/running-playwright-tests/25655/4