Repackage node2nix generated packages using buildNpmPackage #229475
Comments
|
Feel free to move packages out of the set, this is a net good. Not really sure how/if we want to add aliases for the old names, though. (cc @lilyinstarlight) As a rule of thumb, more or less anything with a package-lock.json can be packaged, but some are impossible due to npm cursedness. Feel free to request my review directly on packages, I'm happy to land things quickly and work with you on this :) |
|
Ah, forgot about the overrides file. That looks good, do that. Again, please request my review on these, I'm more than happy to review and get them landed. Please do keep in mind that we expect you to maintain a package if you add it, so only migrate packages that you are okay with maintaining and testing. |
|
@winterqt Should we make a That way they are easy to find and turn into throws later (as for other aliases) and we can just make the node-packages |
|
Great catch, yes, let's do it. |
|
I've gone over a couple of the packages so far, and it seems to be not as straightforward as I initially thought it would be 😛
|
|
For a start, I have categorized the packages around the ones I use. Repackage candidate (has package-lock.json)
Repackaged
Can't be repackaged
|
|
I think we should start with everything that is easy to get done and has a top-level alias. Maybe that is even enough to get nodePackages into shape and we can save ourselves time fixing the really hard and weird cases. |
|
@natsukium thanks a lot for compiling these packages. I've edited the original post |
|
I've added aliases for |
|
Just want to note that a missing The nixpkgs/pkgs/tools/misc/github-copilot-cli/default.nix Lines 14 to 16 in 1e237b7 (There is precedent for this type of approach in |
|
Can we please NOT add 5000 lines of generated packages.json locks per node package? |
|
Last time I checked node2nix was doing a lot of the fetching in serial. However one could speed it up by pulling with 32-threads instead of just one (7min instead of 4hours based on my napkin math) |
|
Indeed, in the example above, it was undesirable to commit the 5,000 lines of package-lock.json. Also, even if we could fetch in parallel, my laptop (macbook pro) only has eight threads, so it is unlikely to be that fast. (although it is enough compared to 4 hours). |
|
Can we get statistics / figures on what it would add to nixpkgs before anything? |
|
FWIW @Mic92, not every package requires that we add a |
|
Apologies for my confusion, I didn't understand it was about adding only those who had package-lock.json in tree. 👍 for the approach! |
|
I am ok with this approach when we do this for packages that provide a |
|
We can also upload the package-lock.json to somewhere with buildNpmPackage.
That doesn't fix that it is broken in other ways and confuses dev and prod dependencies. |
That's definitely one way to do it, but we'd need to decide on a hosting platform for said lock files. It's probably better not to leave the choice up to the individual contributor as that would possibly make maintenance even harder than it is now, with the upstream fetcher endpoints going offline and needing to be updated. I'm also not sure if we wouldn't run into IFD issues by doing that. But that seems to be a general issue with monorepo approach that Nixpkgs has. Any user of the repository needs to clone the entire repository regardless of how much of it they will actually use. It's obviously not cloning the binaries, but Nixpkgs is continuously growing, and I don't know if that approach is sustainable long term. However that question is beyond the scope of this issue and should probably be addressed in an RFC. I feel like sticking to packages with pre-existing As a side note, I apologise for starting the issue and then disappearing out of the blue. I had just undergone surgery and didn't have much time in the days leading up to it. I should be able to contribute a bit more now that I'm stuck in a hospital bed however ^^ |
tomodachi94
added
the
5. scope: tracking
Issue description
Currently a vast majority of NPM packages available in Nixpkgs are generates using node2nix. It generates a nix expression form the
node-packages.jsonfile. However it's very hard to maintain. Any time a new package is added to that list or updated, node2nix has to update every single package listed innode-packages.jsonusing a code-gen script. That forces PRs that touch that part of the ecosystem to rerun the code-gen script each time only other PR is merged. This is further exacerbated by the time the script takes to run. As of writing this post, there are 408 packages innode-packages.json, and it takes ~4h to run the generation script.Proposed solution
Nowadays an alternative tool,
buildNpmPackage, is available, but plenty of the packages already listed there were added there before its creation. Not all of the packages can be build using the new tool, but I think it would be worthwhile to review how many of them could be repackaged that way. That should at the very least make the time of running the code-gen script shorter.As such, I've listed all the packages that are build using
node2nix. My intention is this issue could be used to collaborate on the repackaging efforts. If any package gets successfully repackaged, or determined to not be possible to be built usingbuiltNpmPackage, I'd appreciate having that mentioned here so that I can update the list and move that package to an appropriate location.Not checked
Repackage candidate (has package-lock.json)
Repackaged
Can't be repackaged
Usually a package can't be built with
buildNpmPackagewhen it has nopackage-lock.jsonchecked out in its vcs. So that's when it's being built with something other thannpmor when the upstream maintainers just don't track it with vcs.Built with
yarnBuilt with
pnpmBuilt with
npm, but no package-lock.json in vcsWant to contribute? Here's a general walkthrough of how you can do that
package-lock.jsonit might be a good candidatecallPackagederivation usingbuildNpmPackageand put it somewhere appropriate in the nixpkgs package tree.node-packages.jsonand add it toaliases.nixIn case the package does not follow the happy scenario you can mention it here so I can move it to the appropriate group of packages. For an example of a package repackaged this way you can check #229512
I've discussed this idea with @SuperSandro2000 in another PR briefly, but I'd appreciate other people responsible for maintaining the node ecosystem to weigh in as well. @NixOS/node would this be the right way to go about this? Where should the repackaged packages be put? Is doing this worthwhile?
The text was updated successfully, but these errors were encountered: