Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dedupe libvips-42.dll for windows users #3402

Closed
Pomax opened this issue Oct 7, 2022 · 10 comments
Closed

Dedupe libvips-42.dll for windows users #3402

Pomax opened this issue Oct 7, 2022 · 10 comments

Comments

@Pomax
Copy link

Pomax commented Oct 7, 2022

Feature request

As Windows doesn't come with libvips, Sharp convenient bundles it in as a .dll file, but it bundles it in twice: once in build/release and then again in vendor/8.13.2/win32-x64/lib. Given that it's a whopping 21Mb, deduplicating that would be an excellent idea. (The same applies to libglib-2.0-0.dll, libvips-cpp.dll, and libgobject-2.0-0.dll, which take up a combined additional 2Mb)

The two most obvious options I can think of would be:

  • remove the vendor dir from the Windows build before packing up the release, or
  • make the .dll files in build/release Windows symlinks instead of real files.

Either would cut the Windows release size in half.

@lovell
Copy link
Owner

lovell commented Oct 7, 2022

At npm install time, sharp ensures the DLLs have been extracted to vendor then copies them to build (so they exist in the same directory as the prebuilt binary for sharp itself), which is primarily to ensure these files survive a rebuild.

The logic for this Windows-specific behaviour is in https://github.com/lovell/sharp/blob/main/install/dll-copy.js

There used to be (permission?) problems in the past with Node.js and symlinks on Windows, but perhaps we could experiment using a "junction" for this?

@Pomax
Copy link
Author

Pomax commented Oct 7, 2022

how long ago are we talking? Because windows has two types of symlinks - junctions, which generally need admin rights to create, and "plain symlinks" which don't.

@lovell
Copy link
Owner

lovell commented Oct 8, 2022

My Windows knowledge is rather limited, but my understanding is that junctions can be created by a local, non-elevated user, but symlinks might require local admin rights, at least on NTFS.

Sadly "junctions" won't help here as they are for directories only, where we are dealing with individual files. (The fix for the Node.js ecosystem was via libuv/libuv#1706 and has been available since version 10.)

If we are to use symlinking, it would be best to retain the copying logic as a fallback should it fail.

@Pomax
Copy link
Author

Pomax commented Oct 8, 2022

In that case I guess the question is "why does the vendor directory need to hang around?". The dll files are not tied to Node version, so making the node-gyp process look for them in the build/release dir should be fine?

@lovell
Copy link
Owner

lovell commented Oct 9, 2022

When npm install --build-from-source is run, the build directory is removed, so we need to have these somewhere else on the filesystem to support this development use case.

We could test for process.env.npm_config_build_from_source and use the existing copy approach when present, otherwise use a rename? Happy to accept a PR for this, if you're able.

@Pomax
Copy link
Author

Pomax commented Oct 9, 2022

never tried anything gyp related (other than filing issues to please add python 3 support back in the 2.7 days =) but I can have a look?

@Julusian
Copy link

How will removing these duplicate files interact with prebuild-install?

It is not uncommon for prebuild-install to be run each time an electron build is produced.
In one project I have made it manually invoke the postinstall commands, to allow for cross-compiling the electron app https://github.com/bitfocus/companion/blob/master/tools/build_prepare.mjs

@lovell
Copy link
Owner

lovell commented Oct 19, 2022

@Julusian I think it will be OK as prebuild-install won't delete anything. Running node-gyp will delete build, hence the current implementation that always copies the latest DLL files.

Perhaps, as part of this proposal, we should also add logic to dll-copy that verifies build has the expected files, especially if vendor is empty?

@lovell
Copy link
Owner

lovell commented Aug 15, 2023

This will probably be superseded by #3750

@lovell
Copy link
Owner

lovell commented Dec 1, 2023

This problem has gone away with the release of sharp v0.33.0 - all that custom download and file copying logic has been removed.

@lovell lovell closed this as completed Dec 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants