playwright: package with buildNpmPackage #302759

kalekseev · 2024-04-09T05:50:43Z

Description of changes

@playwright/test and playwright-core packages built from source
python3Packages.playwright: 1.44.0 -> 1.45.1
playwright: 1.40.0 -> 1.45.3
updated update script
~~renamed playwright-driver into playwright-core to match original name https://www.npmjs.com/package/playwright-core but we can left it as driver to not break things for users~~
~~removed playwright-test, it's now replaced by playwright https://www.npmjs.com/package/@playwright/test~~
~~made playwright js package a top level package (python version used previously is just a wrapper around js package)~~

/nix/store/nxawidis47cx9v15j2smbhnrlzjg498i-playwright-core-1.43.0/cli.js
Usage: npx playwright [options] [command]

Options:
  -V, --version                          output the version number
  -h, --help                             display help for command

Commands:
  open [options] [url]                   open page in browser specified via -b, --browser
  codegen [options] [url]                open page and generate code for user actions
  install [options] [browser...]         ensure browsers necessary for this version of Playwright are installed
  uninstall [options]                    Removes browsers used by this installation of Playwright from the system
                                         (chromium, firefox, webkit, ffmpeg). This does not include branded channels.
  install-deps [options] [browser...]    install dependencies necessary to run browsers (will ask for sudo permissions)
  cr [options] [url]                     open page in Chromium
  ff [options] [url]                     open page in Firefox
  wk [options] [url]                     open page in WebKit
  screenshot [options] <url> <filename>  capture a page screenshot
  pdf [options] <url> <filename>         save page as pdf
  show-trace [options] [trace...]        show trace viewer
  test                                   Run tests with Playwright Test. Available in @playwright/test package.
  show-report                            Show Playwright Test HTML report. Available in @playwright/test package.
  merge-reports                          Merge Playwright Test Blob reports Available in @playwright/test package.
  help [command]                         display help for command

Things done

Add a 👍 reaction to pull requests you find important.

phaer · 2024-04-09T07:51:09Z

@kalekseev Nice, thanks for working on this!

The current implementation, using node2nix has the benefit of automatically updating the hashes. Could we do the same here? i.e. update them in updateScript?

kalekseev · 2024-04-09T08:12:10Z

@kalekseev Nice, thanks for working on this!

The current implementation, using node2nix has the benefit of automatically updating the hashes. Could we do the same here? i.e. update them in updateScript?

we have to write a bunch of bash in update.sh

phaer · 2024-04-09T10:35:45Z

@kalekseev Nice, thanks for working on this!
The current implementation, using node2nix has the benefit of automatically updating the hashes. Could we do the same here? i.e. update them in updateScript?

we have to write a bunch of bash in update.sh

Are you going to do so? Otherwise, I could take a look at it next week.
Ideally, version should be kept in sync with the driver via update.sh as well.

kalekseev · 2024-04-09T11:20:20Z

Are you going to do so? Otherwise, I could take a look at it next week. Ideally, version should be kept in sync with the driver via update.sh as well.

@phaer don't know if I will have time to work on this PR this week, feel free to work on it.

BTW After recent changes it looks like playwright, playwright-test, playwright-driver were unified as single package playwright so we should review these options:

replace playwright-test and playwright-driver with playwright
move python version of playwright out of global packages into pythonPackages and replace it with this package.
playwright-python should use ${playwright}/cli.js as driver
update.sh should use browser.json from original repository

kalekseev · 2024-04-10T20:58:17Z

@phaer I think I'm done with this one, updated update.sh with chatgpt help, not the best code but good enough for start.

teto · 2024-06-08T14:47:52Z

pkgs/development/python-modules/playwright/default.nix

    # Set the correct driver path with the help of a patch in patches
    substituteInPlace playwright/_impl/_driver.py \
-      --replace "@driver@" "${driver}/bin/playwright"
+      --replace "@driver@" "${driver}/cli.js"


teto · 2024-06-17T14:37:43Z

Result of nixpkgs-review pr 302759 run on x86_64-linux 1

52 packages built:

changedetection-io
changedetection-io.dist
lacus
lacus.dist
pentestgpt
pentestgpt.dist
playwright (python311Packages.playwright)
playwright-driver
playwright-test
playwright.dist (python311Packages.playwright.dist)
python311Packages.batchspawner
python311Packages.batchspawner.dist
python311Packages.dockerspawner
python311Packages.dockerspawner.dist
python311Packages.jupyterhub
python311Packages.jupyterhub-ldapauthenticator
python311Packages.jupyterhub-ldapauthenticator.dist
python311Packages.jupyterhub-systemdspawner
python311Packages.jupyterhub-systemdspawner.dist
python311Packages.jupyterhub-tmpauthenticator
python311Packages.jupyterhub-tmpauthenticator.dist
python311Packages.jupyterhub.dist
python311Packages.lacuscore
python311Packages.lacuscore.dist
python311Packages.oauthenticator
python311Packages.oauthenticator.dist
python311Packages.playwright-stealth
python311Packages.playwright-stealth.dist
python311Packages.playwrightcapture
python311Packages.playwrightcapture.dist
python311Packages.pycookiecheat
python311Packages.pycookiecheat.dist
python311Packages.pytest-playwright
python311Packages.pytest-playwright.dist
python312Packages.lacuscore
python312Packages.lacuscore.dist
python312Packages.playwright
python312Packages.playwright-stealth
python312Packages.playwright-stealth.dist
python312Packages.playwright.dist
python312Packages.playwrightcapture
python312Packages.playwrightcapture.dist
python312Packages.pycookiecheat
python312Packages.pycookiecheat.dist
python312Packages.pytest-playwright
python312Packages.pytest-playwright.dist
shot-scraper
shot-scraper.dist
theharvester
theharvester.dist
urlwatch
urlwatch.dist

phaer

I built this successfully on aarch64-darwin, with a disabled sandbox - but that's already been needed before that PR on darwin.
Some smaller style-related comments regarding update.sh. I also think that the update script could maybe be moved to pkgs/development/web/playwright, as it's mostly affecting the driver.nix file there?
I think the bundling solution works well, Thanks :)
As this PR is more up-to-date now, I'll re-base & update Playwright: browser improvements, update #298944 once this one got merged.

phaer · 2024-07-15T10:51:52Z

pkgs/development/python-modules/playwright/update.sh

+update-source-version python3Packages.playwright "$version"
+
+# Update package-lock.json files for all npm deps that are built in playwright
+# TODO: skip if update-source-version reported the same version


That's not checked yet, I believe?

phaer · 2024-07-15T11:00:13Z

pkgs/development/python-modules/playwright/update.sh

-replace_sha() {
-  sed -i "s|$1 = \".\{44,52\}\"|$1 = \"$2\"|" "$driver_file"
+# Function to download `package-lock.json` for a given source path and update hash
+update_hash() {


Suggested change

update_hash() {

update_npm_hash() {

or something like this? Just to show that it's about npm/nodejs packages here, we are in ./pkgs/development/python-modules here.

phaer · 2024-07-27T20:33:36Z

@kalekseev is there anything one could do to to help pushing it forward? Or is it just waiting to a merge by a commuter.

None of my comments should be blocking anything, this is definitely progress in my view!

kalekseev · 2024-07-28T03:19:22Z

@phaer I believe it's ready to be merged, if someone sees any blockers I'm ready to fix them

phaer · 2024-07-28T10:53:20Z

@teto This might be good to merge then, if you don't have any objections? I think you might be the only one with commit bit among the current reviewers :)

teto · 2024-08-05T14:56:34Z

@phaer yes I am interested in merging this. I want to test it first, most likely next monday. Could you rebase to fix the conflict please ?

kalekseev · 2024-08-06T02:50:16Z

@phaer yes I am interested in merging this. I want to test it first, most likely next monday. Could you rebase to fix the conflict please ?

@teto rebased

ibrahimsag · 2024-08-17T17:53:41Z

Can we update the playwright to 1.46?

teto · 2024-08-19T21:45:09Z

@phaer yes I am interested in merging this. I want to test it first, most likely next monday. Could you rebase to fix the conflict please ?

Thanks for rebasing. I am sorry it takes so long. Playwright is something I help with on the side and before merging the PR, I would like to test it on our infrastructure, which needs a patch in staging-next. Once my patch reaches nixos-unstable, I can check this with our playwright code and if all goes well I will merge. So this may take a few more days. Other nixpkgs contributors may merge it before me. I will rebase if need be so you dont spend more time on this.

SuperSandro2000 · 2024-08-23T11:25:35Z

pkgs/development/web/playwright/driver.nix

I couldn't find any removed deps in this commit

deps were removed in master, updated commit message

SuperSandro2000 · 2024-08-23T11:26:18Z

pkgs/development/web/playwright/driver.nix

+    pname = "playwright";
+    inherit version src;
+
+    sourceRoot = "${src.name}"; # update.sh depends on sourceRoot presence


Suggested change

sourceRoot = "${src.name}"; # update.sh depends on sourceRoot presence

sourceRoot = src.name;

update.sh script depends on this form.

pkgs/development/web/playwright/driver.nix

playwright-driver: 1.40.0 -> 1.46.0 python3Packages.playwright: 1.44.0 -> 1.46.0

kalekseev · 2024-08-24T14:08:45Z

Updated to 1.46.0 and rebased.

teto · 2024-09-03T13:07:46Z

tested here. Looks good. I think we might want to expose more but this can be done later. Merging.

humemm · 2024-09-04T19:03:03Z

@kalekseev @teto

Just pulled and ran this PR. The build passes fine but the runtime blows up with an error

node:internal/modules/cjs/loader:1148
  throw err;
  ^

Error: Cannot find module '/nix/store/mfiy2g44i2lxmj9rdxns3zhvqdf4cv81-playwright-core-1.46.0/package/cli.js'
    at Module._resolveFilename (node:internal/modules/cjs/loader:1145:15)
    at Module._load (node:internal/modules/cjs/loader:986:27)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:174:12)
    at node:internal/main/run_main_module:28:49 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}

Just reviewed the locations and code and it seems like cli.js is no longer found in /package/cli.js but instead it's jut in the top level directory.

The following is the fix in playwright/default.nix:

# Set the correct driver path with the help of a patch in patches
substituteInPlace playwright/_impl/_driver.py \
  --replace-fail "@node@" "${lib.getExe nodejs}" \
-  --replace-fail "@driver@" "${driver}/package/cli.js"
+  --replace-fail "@driver@" "${driver}/cli.js"

maybe in the future we should add a simple test that runs playwright --version after build.

kalekseev · 2024-09-05T14:19:40Z

maybe in the future we should add a simple test that runs playwright --version after build.

this is python package not playwright itself so playwright --version won't catch this, I'll create PR with fix in a couple of hours

humemm · 2024-09-05T21:55:28Z

@kalekseev

maybe in the future we should add a simple test that runs playwright --version after build.

this is python package not playwright itself so playwright --version won't catch this, I'll create PR with fix in a couple of hours

Not sure what you're referring to. The error above was triggered by simply doing playwright --version. This "python" package also exposes the playwright executable/cli when installed.

phaer · 2024-09-05T23:26:44Z

@humemm now that this pr is merged, I just need to find time for a rebase of #298944 which includes a NixOS test

kalekseev mentioned this pull request Apr 9, 2024

Playwright: browser improvements, update #298944

Merged

15 tasks

ofborg bot added 8.has: package (new) 10.rebuild-darwin: 1-10 10.rebuild-darwin: 1 10.rebuild-linux: 1-10 10.rebuild-linux: 1 labels Apr 9, 2024

kalekseev force-pushed the playwright-core branch from 2557d20 to 4fd5530 Compare April 9, 2024 08:10

github-actions bot added the 6.topic: python label Apr 9, 2024

kalekseev changed the title ~~[POC] playwright-core: package with buildNpmPackage~~ playwright-test: package with buildNpmPackage Apr 9, 2024

kalekseev force-pushed the playwright-core branch 2 times, most recently from 195c2aa to 429aae2 Compare April 10, 2024 11:58

kalekseev changed the title ~~playwright-test: package with buildNpmPackage~~ playwright: package with buildNpmPackage Apr 10, 2024

kalekseev force-pushed the playwright-core branch from 429aae2 to 5aed73f Compare April 10, 2024 12:22

ofborg bot added the 8.has: clean-up label Apr 10, 2024

ofborg bot requested review from yrd, techknowlogick and sephii April 10, 2024 12:46

ofborg bot added 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 and removed 10.rebuild-darwin: 1 10.rebuild-darwin: 1-10 10.rebuild-linux: 1 10.rebuild-linux: 1-10 labels Apr 10, 2024

kalekseev force-pushed the playwright-core branch from 5aed73f to fbc7571 Compare April 10, 2024 20:54

kalekseev marked this pull request as ready for review April 10, 2024 20:58

teto reviewed Jun 8, 2024

View reviewed changes

kalekseev force-pushed the playwright-core branch 2 times, most recently from 279e7af to 84dce4a Compare June 10, 2024 08:26

kalekseev force-pushed the playwright-core branch from 84dce4a to ff625d5 Compare July 6, 2024 08:39

kalekseev requested a review from teto July 6, 2024 08:52

phaer approved these changes Jul 15, 2024

View reviewed changes

kalekseev force-pushed the playwright-core branch 2 times, most recently from b8a8575 to 64de115 Compare August 6, 2024 02:46

SuperSandro2000 reviewed Aug 23, 2024

View reviewed changes

playwright: apply nixfmt

46da647

kalekseev force-pushed the playwright-core branch from 64de115 to 0f341d6 Compare August 24, 2024 08:10

playwright: build from source

4519015

playwright-driver: 1.40.0 -> 1.46.0 python3Packages.playwright: 1.44.0 -> 1.46.0

kalekseev force-pushed the playwright-core branch from 0f341d6 to 4519015 Compare August 24, 2024 14:08

teto merged commit 23f4d2e into NixOS:master Sep 3, 2024
26 checks passed

kalekseev mentioned this pull request Sep 5, 2024

python3Packages.playwright: fix cli path #339866

Merged

13 tasks

kalekseev mentioned this pull request Sep 22, 2024

playwright-test 1.47.0 -> 1.47.2 plus update script #343383

Open

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

playwright: package with buildNpmPackage #302759

playwright: package with buildNpmPackage #302759

kalekseev commented Apr 9, 2024 •

edited

Loading

phaer commented Apr 9, 2024

kalekseev commented Apr 9, 2024

phaer commented Apr 9, 2024

kalekseev commented Apr 9, 2024

kalekseev commented Apr 10, 2024

teto Jun 8, 2024

teto commented Jun 17, 2024

phaer left a comment

phaer Jul 15, 2024

phaer Jul 15, 2024

phaer commented Jul 27, 2024

kalekseev commented Jul 28, 2024

phaer commented Jul 28, 2024

teto commented Aug 5, 2024

kalekseev commented Aug 6, 2024

ibrahimsag commented Aug 17, 2024

teto commented Aug 19, 2024

SuperSandro2000 Aug 23, 2024

kalekseev Aug 24, 2024

SuperSandro2000 Aug 23, 2024

kalekseev Aug 24, 2024

kalekseev commented Aug 24, 2024

teto commented Sep 3, 2024

humemm commented Sep 4, 2024 •

edited

Loading

kalekseev commented Sep 5, 2024

humemm commented Sep 5, 2024 •

edited

Loading

phaer commented Sep 5, 2024

	sourceRoot = "${src.name}"; # update.sh depends on sourceRoot presence
	sourceRoot = src.name;

playwright: package with buildNpmPackage #302759

playwright: package with buildNpmPackage #302759

Conversation

kalekseev commented Apr 9, 2024 • edited Loading

Description of changes

Things done

phaer commented Apr 9, 2024

kalekseev commented Apr 9, 2024

phaer commented Apr 9, 2024

kalekseev commented Apr 9, 2024

kalekseev commented Apr 10, 2024

teto Jun 8, 2024

Choose a reason for hiding this comment

teto commented Jun 17, 2024

phaer left a comment

Choose a reason for hiding this comment

phaer Jul 15, 2024

Choose a reason for hiding this comment

phaer Jul 15, 2024

Choose a reason for hiding this comment

phaer commented Jul 27, 2024

kalekseev commented Jul 28, 2024

phaer commented Jul 28, 2024

teto commented Aug 5, 2024

kalekseev commented Aug 6, 2024

ibrahimsag commented Aug 17, 2024

teto commented Aug 19, 2024

SuperSandro2000 Aug 23, 2024

Choose a reason for hiding this comment

kalekseev Aug 24, 2024

Choose a reason for hiding this comment

SuperSandro2000 Aug 23, 2024

Choose a reason for hiding this comment

kalekseev Aug 24, 2024

Choose a reason for hiding this comment

kalekseev commented Aug 24, 2024

teto commented Sep 3, 2024

humemm commented Sep 4, 2024 • edited Loading

kalekseev commented Sep 5, 2024

humemm commented Sep 5, 2024 • edited Loading

phaer commented Sep 5, 2024

kalekseev commented Apr 9, 2024 •

edited

Loading

humemm commented Sep 4, 2024 •

edited

Loading

humemm commented Sep 5, 2024 •

edited

Loading