Bundling sharp doesn't work anymore since updating to 4.5.0 #1075

pkeuter · 2021-03-23T12:48:19Z

Since updating to version 4.5.0 it is not possible to bundle sharp anymore. Previously this was fairly simple, by just copying the node_modules/sharp folder next to the build output. But this doesn't work anymore giving the following error:

Error:
Something went wrong installing the "sharp" module

File '/**/test/node_modules/sharp/build/Release/libvips-cpp.so.42' was not included into executable at compilation stage. Please recompile adding it as asset or script.

- Remove the "node_modules/sharp" directory then run
  "npm install --ignore-scripts=false --verbose" and look for errors
- Consult the installation documentation at https://sharp.pixelplumbing.com/install
- Search for this error at https://github.com/lovell/sharp/issues

    at Object.<anonymous> (/snapshot/test/node_modules/sharp/lib/constructor.js:34:9)
    at Module._compile (pkg/prelude/bootstrap.js:1329:22)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1218:10)
    at Module.load (internal/modules/cjs/loader.js:1047:32)
    at Function.Module._load (internal/modules/cjs/loader.js:935:14)
    at Module.require (internal/modules/cjs/loader.js:1087:19)
    at Module.require (pkg/prelude/bootstrap.js:1234:31)
    at require (internal/modules/cjs/helpers.js:73:18)
    at Object.<anonymous> (/snapshot/test/node_modules/sharp/lib/index.js:3:15)
    at Module._compile (pkg/prelude/bootstrap.js:1329:22)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1218:10)
    at Module.load (internal/modules/cjs/loader.js:1047:32)
    at Function.Module._load (internal/modules/cjs/loader.js:935:14)
    at Module.require (internal/modules/cjs/loader.js:1087:19)
    at Module.require (pkg/prelude/bootstrap.js:1234:31)
    at require (internal/modules/cjs/helpers.js:73:18)

I'm fairly convinced this has to do with #837 or #1066 or maybe a combination of both.

Looking at the below error message running in CI, it seems that prebuild-install is not correctly installed or referenced, this has been added in #1066:

> [email protected]
> Fetching base Node.js binaries to PKG_CACHE_PATH
> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules/sharp/build/Release
  %2: path-to-executable/sharp/build/Release
> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules/sharp/vendor/lib
  %2: path-to-executable/sharp/vendor/lib
> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules/sharp/build/Release
  %2: path-to-executable/sharp/build/Release
> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules/sharp/vendor/lib
  %2: path-to-executable/sharp/vendor/lib
/bin/sh: 1: /builds/test/node_modules/pkg/node_modules/.bin/prebuild-install: not found
/bin/sh: 1: /builds/test/node_modules/pkg/node_modules/.bin/prebuild-install: not found

I'm mentioning both @david-mohr as well as @geekuillaume hoping they have a suggestion on how to fix this.

On a sidenote; this seems to me a breaking change so 5.0.0 would have been a more logical version for this release I suppose.

The text was updated successfully, but these errors were encountered:

david-mohr · 2021-03-23T13:16:07Z

have you run yarn to ensure that prebuild-install is actually installed?

jesec · 2021-03-23T13:29:32Z

If the mentioned changes are indeed breaking, I think we would have to revert them immediately and release a new version.

First, I would wait for a while for responses from the related parties, and see if there is a misconfiguration or if the changes can be tweaked to be non-breaking.

@pkeuter mentioned just copying the node_modules/sharp folder next to the build output. Is this "load from outside" way previously supported?

Any thoughts? @leerob @robertsLando

leerob · 2021-03-23T13:41:54Z

Before doing any reverting, would need to verify and confirm it is indeed a breaking change. I'm not convinced those two PRs are related yet but would love to see the investigation here.

david-mohr · 2021-03-23T13:52:18Z

prebuild-install is definitely in the package.json. @pkeuter how did you upgrade and does running 'yarn' or npm i fix the issue?

pkeuter · 2021-03-23T14:44:13Z

I see it is in the package.json and it is also in the package-lock.json (I'm using npm):

"pkg": {
      "version": "4.5.0",
      "resolved": "https://registry.npmjs.org/pkg/-/pkg-4.5.0.tgz",
      "integrity": "sha512-TEAQAbrd4PKyEydg28OLzg61L/NsGN1NcCqBFzY9TxzpPAds/Nogd+9bc2r/nZxBQhHLL1l+LldVNRFoQhdPWg==",
      "dev": true,
      "requires": {
        "@babel/parser": "^7.9.4",
        "@babel/runtime": "^7.9.2",
        "chalk": "^3.0.0",
        "escodegen": "^1.14.1",
        "fs-extra": "^8.1.0",
        "globby": "^11.0.0",
        "into-stream": "^5.1.1",
        "minimist": "^1.2.5",
        "multistream": "^2.1.1",
        "pkg-fetch": "^2.6.9",
        "prebuild-install": "6.0.1",
        "progress": "^2.0.3",
        "resolve": "^1.15.1",
        "stream-meter": "^1.0.4"
      },
...

and this is top-level the prebuild-install:

    "prebuild-install": {
      "version": "6.0.1",
      "resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-6.0.1.tgz",
      "integrity": "sha512-7GOJrLuow8yeiyv75rmvZyeMGzl8mdEX5gY69d6a6bHWmiPevwqFw+tQavhK0EYMaSg3/KD24cWqeQv1EWsqDQ==",
      "requires": {
        "detect-libc": "^1.0.3",
        "expand-template": "^2.0.3",
        "github-from-package": "0.0.0",
        "minimist": "^1.2.3",
        "mkdirp-classic": "^0.5.3",
        "napi-build-utils": "^1.0.1",
        "node-abi": "^2.7.0",
        "noop-logger": "^0.1.1",
        "npmlog": "^4.0.1",
        "pump": "^3.0.0",
        "rc": "^1.2.7",
        "simple-get": "^3.0.3",
        "tar-fs": "^2.0.0",
        "tunnel-agent": "^0.6.0",
        "which-pm-runs": "^1.0.0"
      },
      "dependencies": {
        "simple-get": {
          "version": "3.1.0",
          "resolved": "https://registry.npmjs.org/simple-get/-/simple-get-3.1.0.tgz",
          "integrity": "sha512-bCR6cP+aTdScaQCnQKbPKtJOKDp/hj9EDLJo3Nw4y1QksqaovlW/bnptB6/c1e+qmNIDHRK+oXFDdEqBT8WzUA==",
          "requires": {
            "decompress-response": "^4.2.0",
            "once": "^1.3.1",
            "simple-concat": "^1.0.0"
          }
        }
      }
    },

This was running in CI so there was no node_modules folder. No cache or something either.

When I look in the folder referenced /home/peter/test/node_modules/pkg/node_modules/.bin/ I don't see the binary either. I do see it a bit higher in /home/peter/test/node_modules/.bin. This probably is because of the flattening procedure that npm uses (see https://docs.npmjs.com/cli/v7/configuring-npm/folders). I am not sure if this differs from yarn but this seems to be a logical structure.

david-mohr · 2021-03-23T21:26:33Z

@pkeuter ah yes, I see what you mean now. Here's the target reference https://github.com/vercel/pkg/blob/master/lib/producer.js#L100. As you say, yarn handles this differently which is why it worked fine during testing. I also have my pkg install globally.

If you know how to change the path to handle any combination of situations (local/global, yarn/npm) that would be great.

pkeuter · 2021-03-24T17:01:07Z

@david-mohr I do not know a solution right away, but the fact is that this is, indeed, a breaking change so it might be useful to revert the change in a new release until this has been resolved. Because pkg is currently broken for npm users that use native modules.

david-mohr · 2021-03-24T23:10:01Z

@pkeuter I think this can be patched to work with both npm and yarn, but if it needs to be backed out, that's fine too.

@leerob guess it's your call. To fix this, the function can use existsSync to hunt around for the path. Otherwise just back it out, it doesn't seem that other users are interested in cross-platform support, so it's not really a helpful patch.

leerob · 2021-03-25T00:44:05Z

I'm sorry you're having this issue @pkeuter ☹️ It currently appears it's isolated to one report. Given that, I'd say we patch forward. Cross-platform support seems like something we would want to support.

JeffJassky · 2021-03-26T11:52:58Z

The prebuild-install problem:

The path to the prebuild script is being built incorrectly. The error is in producer.js

pkg/lib/producer.js

Line 135 in f08d083

const prebuild = path.join(

A Workaround:

I got it to resolve the correct path by changing that line to: const prebuild = _path.default.join(process.cwd(), './node_modules/.bin/prebuild-install');. After that, the prebuild-install script path was correct - and it was being executed successfully.

However... The prebuild script shows a warning prebuild-install WARN install No prebuilt binaries found (target=v12.2.0 runtime=node arch=x64 libc= platform=win32)

Manually Installing cross-platform prebuilt binaries:

So I installed my native module (sqlite3) manually using npm install sqlite3 --arch=x64 --platform=win32 which resulted in a very happy looking Success: "/Users/jeffjassky/Projects/StoryFolderDesktop/node_modules/sqlite3/lib/binding/napi-v3-win32-x64/node_sqlite3.node" is installed via remote

Unfortunately:

When running pkg I still got the same error: prebuild-install WARN install No prebuilt binaries found (target=v12.2.0 runtime=node arch=x64 libc= platform=win32) as if it doesn't recognize or include the newly installed module.

My guess is that the target version was incorrect - but it's hard to know for sure. When I tried adding the --target=v12.2.0 flag I got a nasty Unsupported target version: v12.2.0 error.

So I tried manually added the module as an asset in package.json

  "pkg": {
    "assets": [
      "./node_modules/sqlite3/lib/binding/napi-v3-win32-x64/node_sqlite3.node"
    ]
  }

When I run my packaged .exe on Windows I can see that the file indeed gets included, but I get this error: Caught exception: Error: The specified module could not be found. C:\snapshot\StoryFolderDesktop\node_modules\sqlite3\lib\binding\napi-v3-win32-x64\node_sqlite3.node

So, lastly, I copied the node_sqlite3.node file into path-to-executable/node_modules/sqlite3/lib/binding/napi-v3-win32-x64/node_sqlite3.node along side the executable and that finally seems to work, as inelegant as it is.

So, for some reason, it can't locate the module even though it's right there in the snapshot - but it works when the module is placed alongside it.

robertsLando · 2021-03-26T12:47:50Z

@JeffJassky Usually this kind of problems are fixed by dictionaries. You need to edit this https://github.com/vercel/pkg/blob/master/dictionary/sqlite3.js see an example here: https://github.com/vercel/pkg/blob/master/dictionary/puppeteer.js

JeffJassky · 2021-03-26T13:24:10Z

@robertsLando I sort-of understand. These are package-specific configs for pkg - though I don't precisely know what I should put in there - 'cause I don't quite understand what the problem is, to begin with.

robertsLando · 2021-03-26T13:32:27Z

I'm not the author of this package I have just take a general lookup of the code and from what I have understand that's a collection of package-specific fixes. With that you can specify patches (replace/add/remove code from files) and or additional assets/scripts to add for a specific package like:

'use strict';

module.exports = {
  pkg: {
    scripts: ['lib/middleware/*.js'],
    assets: [
      'lib/public/**/*', // for [email protected]
    ],
  },
};

In your case I think a possible fix could be adding the assets needed here in the sqlite3 dictionary, something like:

'use strict';

module.exports = {
  pkg: {
    assets: [
      'lib/binding/**/*',
    ],
  },
};

Give this a try and let me know :)

JeffJassky · 2021-03-26T14:32:05Z

@robertsLando After looking into it further, the .node module is actually getting included in the snapshot - in the exact correct location. Hmm... the plot thickens.

Seems to be related directly to #1075, possibly #894, and #687

robertsLando · 2021-03-26T14:36:47Z

We might check where the lookup fails so. Based on what I understand native node modules behaves different then normal assets, them are copied in a temporary folder iin order to be used, maybe the lookup or the copy fails somewhere. That feature has been added in #837. Did you alo try to put the node addon in the same root dir of the application to see if that works?

The magic happens here: https://github.com/vercel/pkg/pull/837/files#diff-0c3f42fd6825d8d5b4d0422402a89ff11dc1d1cfb52ee5901f4774e503783c27R1598

JeffJassky · 2021-03-26T14:41:44Z

@robertsLando That's exactly what I was thinking it could be. I'll look into your references now.

Putting the .node module directly next to the executable didn't work. It only worked when a node_modules folder was placed next to the executable with the .node module nested deeply inside.

JeffJassky · 2021-03-26T15:29:08Z

@robertsLando you were right on with the dlopen function. I think I've found the gremlin we're dealing with.

The Extended-Length Path Prefix problem

It seems that in some (possibly all) cases, on Windows (and possibly linux?), the paths of manually included assets in the snapshot include "extended length prefixes".

For example, the path to my file was: \\?\\C:\\snapshot\\StoryFolderDesktop\\node_modules\\sqlite3\\lib\\binding\\napi-v3-win32-x64\\node_sqlite3.node notice the strange little \\?\\ prefix.

I noticed in the original post by @pkeuter there seemed to be a strange prefix on his path too: /**/test/node_modules/sharp/build/Release/libvips-cpp.so.42 notice /**/.

Here's someone discussing (for Windows) it on SuperUser:
https://superuser.com/questions/1208362/what-does-the-following-notation-mean-c-mean

Because of the prefix, it causes this check to fail:

pkg/prelude/bootstrap.js

Line 1714 in f08d083

if (insideSnapshot(modulePath)) {

making it proceed as if the file doesn't exist in the snapshot, resulting in this error: (filename) was not included into executable at compilation stage. Please recompile adding it as asset or script.

Then, if you try to actually load the file using the prefix it results in this error: Error: The specified module could not be found.

Workaround

By stripping the prefix off of the modulePath value before the if(insideSnapshot(modulePath)){...} check, the sqlite3 module loaded successfully and my app fully started as expected.

Suggested Solution

Account for, disable, remove, or otherwise negate extended-length prefixes for manually included assets.

robertsLando · 2021-03-26T15:40:45Z

@JeffJassky Maybe this function could fix the issue?

JeffJassky · 2021-03-26T15:45:06Z

@JeffJassky Maybe this function could fix the issue?

Something close to that, yes. Though it seems like the function is assuming a prefix of 4 characters, whereas on my machine it was 5 (\\?\\) and on @pkeuter it seemed to be 4 (/**/).

Perhaps you could tweak the function to work for both, when we can both test your branch before PR/merge.

robertsLando · 2021-03-26T15:45:32Z

@JeffJassky By looking again the code I think this has been fixed in #1095 Could you give a try to [email protected] just to be sure?

JeffJassky · 2021-03-26T15:49:52Z

@robertsLando @4.5.1 gives me this:Uncaught Exception: 2 Caught exception: Error: Directory 'C:\**\StoryFolderDesktop\node_modules\sqlite3\lib\binding\napi-v3-win32-x64' was not included into executable at compilation stage. Please recompile adding it as asset or script.

which is interesting for a few reasons. The prefix changed, and it moved, and double slashes are gone.

Fixes #1075

robertsLando · 2021-03-26T15:54:38Z

@JeffJassky Maybe: #1103 ?

JeffJassky · 2021-03-26T15:55:41Z

@JeffJassky Maybe: #1103 ?

How can I install and test a specific commit like that?

robertsLando · 2021-03-26T15:59:20Z

@JeffJassky You should

git clone -b fix-native-addons https://github.com/vercel/pkg.git
cd pkg
npm install
npm link

Then inside your repo:

npm link pkg

JeffJassky · 2021-03-26T16:06:06Z

@robertsLando Identical error to the above: #1075 (comment)

I pulled down the branch and made sure to uninstall pkg from my main repo to ensure it built using the locally linked package. It did build - but still has a messed-up path.

To clarify, the function seems to be handling all of the extended prefixes with the \\?\ style - however, it seems like C:\**\... is a different animal and i don't know where that's coming from.

robertsLando · 2021-03-26T16:29:09Z

@JeffJassky the sanitize function should also take care of that and replace it too, are you using latest changes from the branch? Could you retry to pull it?

JeffJassky · 2021-03-26T16:30:46Z

@robertsLando It's up to date. The function looks like this:

  function revertMakingLong(f) {

    const prefixRegex = /^\\\\\?[\\]+|\/\*\*/
    if (prefixRegex.test(f)) return f.replace(prefixRegex, '');
    return f;
  }

I'm running tests with it and the regex test doesn't pass:

JeffJassky · 2021-03-26T18:04:37Z

@robertsLando The extra **\ stuff is getting added here:

pkg/prelude/common.js

Line 188 in cceb78a

exports.stripSnapshot = function stripSnapshot(f) {

I'm having trouble understanding why it won't build in 4.5.1 but I can easily work around the issues in 4.5.0.

robertsLando · 2021-04-07T13:14:50Z

I think that the easier way to fix this for now would be to completely unpublish pkg-fetc 2.6.10 and just work on the 3.0.0 release @leerob

rightaway · 2021-06-17T08:29:18Z

Is there a recommended workaround for latest version of pkg if you need to use sharp?

rightaway · 2021-07-18T16:10:31Z

Is there a recommended workaround for latest version of pkg if you need to use sharp?

Anyone?

What versions of pkg and pkg-fetch are needed to make this work?

robertsLando · 2021-07-19T07:09:11Z

latest pkg version

rightaway · 2021-07-19T09:32:16Z

@robertsLando This issue was already fixed in latest 5.3.0? I tried with this version and it's failing. What versions of pkg and pkg-fetch can I roll back to where it was working?

In latest version I added "assets": ["./node_modules/sharp/build/Release/sharp.node"] to package.json under "pkg" and when I ran the pkg command I got

> [email protected]
compression:  GZip
> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules/sharp/build/Release
  %2: path-to-executable/sharp/build/Release
> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules/sharp/vendor/lib
  %2: path-to-executable/sharp/vendor/lib
> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules/sharp/build/Release
  %2: path-to-executable/sharp/build/Release
> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules/sharp/vendor/lib
  %2: path-to-executable/sharp/vendor/lib
prebuild-install WARN This package does not support N-API version v14.17.0 
prebuild-install WARN install No prebuilt binaries found (target=v14.17.0 runtime=napi arch=x64 libc= platform=linux)

Then when running the package I get

pkg/prelude/bootstrap.js:1697
      throw error;
      ^

Error: 
Something went wrong installing the "sharp" module

INTERNAL ERROR this file doesn't exist in the virtual file system :/snapshot/project/node_modules/sharp/build/Release/libvips-cpp.so.42

- Remove the "node_modules/sharp" directory then run
  "npm install --ignore-scripts=false --verbose sharp" and look for errors
- Consult the installation documentation at https://sharp.pixelplumbing.com/install
- Search for this error at https://github.com/lovell/sharp/issues

    at Object.<anonymous> (/snapshot/project/node_modules/sharp/lib/constructor.js:32:9)
    at Module._compile (pkg/prelude/bootstrap.js:1751:22)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1097:10)
    at Module.load (internal/modules/cjs/loader.js:933:32)
    at Function.Module._load (internal/modules/cjs/loader.js:774:14)
    at Module.require (internal/modules/cjs/loader.js:957:19)
    at Module.require (pkg/prelude/bootstrap.js:1676:31)
    at require (internal/modules/cjs/helpers.js:88:18)
    at Object.<anonymous> (/snapshot/project/node_modules/sharp/lib/index.js:3:15)
    at Module._compile (pkg/prelude/bootstrap.js:1751:22)

avpavp · 2021-07-23T18:56:42Z

@rightaway Did you have any luck using pkg with sharp? I'm in the same situation... TIA

rightaway · 2021-07-24T12:32:16Z

@avpavp No luck. @robertsLando says it should work in the latest version but that's definitely not true.

avpavp · 2021-07-24T13:08:36Z

@rightaway Same experience here. No success with [email protected] (current version at time of writing)

By using bits and pieces of @webdevog repo mentioned above, I seem to be making progress using [email protected]. My exe runs now, I just need to make sure everything is actually working...

john-yick · 2021-08-10T09:57:10Z

I see the same issue with [email protected] it throws the as shown below, code runs fine from node directly:

Error: 
Something went wrong installing the "sharp" module

dlopen(/var/folders/tj/pr2229k96fsby1b59lw61s080000gn/T/e6aead472f8db8d35231c777b2f4218832660a817248e233a31dbf49c8ebc167/sharp.node, 1): Library not loaded: @rpath/libvips-cpp.42.dylib
  Referenced from: /var/folders/tj/pr2229k96fsby1b59lw61s080000gn/T/e6aead472f8db8d35231c777b2f4218832660a817248e233a31dbf49c8ebc167/sharp.node
  Reason: image not found

- Remove the "node_modules/sharp" directory then run
  "npm install --ignore-scripts=false --verbose" and look for errors
- Consult the installation documentation at https://sharp.pixelplumbing.com/install
- Search for this error at https://github.com/lovell/sharp/issues

    at Object.<anonymous> (/snapshot/modv-core/node_modules/sharp/lib/constructor.js:34:9)
    at Module._compile (pkg/prelude/bootstrap.js:1751:22)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at Module.require (pkg/prelude/bootstrap.js:1676:31)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/snapshot/modv-core/node_modules/sharp/lib/index.js:3:15)
    at Module._compile (pkg/prelude/bootstrap.js:1751:22)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at Module.require (pkg/prelude/bootstrap.js:1676:31)
    at require (internal/modules/cjs/helpers.js:74:18)
    at Object.<anonymous> (/snapshot/modv-core/api/models/common.js)
    at Module._compile (pkg/prelude/bootstrap.js:1751:22)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Module.require (internal/modules/cjs/loader.js:887:19)

leadwolf · 2021-08-15T11:34:10Z

Looking at the sharp repo, in binding.gyp they link to libvips-cpp.so.42 with a relative path:

'ldflags': [
                # Ensure runtime linking is relative to sharp.node
                '-Wl,-s -Wl,--disable-new-dtags -Wl,-rpath=\'$$ORIGIN/../../vendor/<(vips_version)/lib\''
              ]

The problem is that when importing sharp.node, pkg copies it to a temporary folder and libvips can't be resolved from that location. As fallback, pkg tries to find the missing file in the same virtual directory as the .node file, and copies it to the temporary folder. This won't work because at build time libvips isn't in the same directory as sharp.node, so it won't end up in the same virtual directory either. But even if we did manage to get libvips into the same virtual directory and have pkg copy it to the temporary folder, the import will still not work because it is not at the expected relative path: ../../vendor....

robertsLando · 2021-09-29T08:30:30Z

@leadwolf I think you detected the real problem. I also tried to move sharp into the same directory with no luck, I also tried adding both folders to assets:

"assets": [
       "./node_modules/sharp/build/Release/**",
      "./node_modules/sharp/vendor/**",
    ]

and the entire folder:

"assets": [
       "./node_modules/sharp/**",
    ]

Did you find a workaround to this? @jesec @erossignon suggestions?

This should be handled here: https://github.com/vercel/pkg/blob/main/prelude/bootstrap.js#L2035 but for some reason it's not working as expected

robertsLando · 2021-09-29T10:01:53Z

IMO what we should do is to copy the entire module folder instead of just the module, I will see if I can make a PR

Fixes #1075

robertsLando · 2021-09-29T14:19:28Z

For anyone interested in this a fix is available with #1321

When a `.node` file is required `pkg` copies it in a temporary folder like `/tmp/<hash>/addon.node` the problem is that for modules like `sharp` this isn't working as them are statically linked to other `.so` files using relative paths. This pr will copy the entire module folder inside the temporary directory, this allows also to remove the `tryImport` function as there is no reason to try again if that fails. I tested this on my end and it's working I also grouped all hash folders inside `pkg` folder to be sure that the copy will run when users will update pkg, otherwise the copy may not be done when the folder already exists but was created with a previous pkg version Fixes #1075

see vercel/pkg#1319, this makes the package fail because of vercel/pkg#1075 rather than fail in the same way because of the missinng dependency

Fixes vercel#1075

rightaway · 2022-07-15T06:12:44Z

@robertsLando Is it still working for you? An error is happening again #1692.

robertsLando · 2022-07-15T06:14:28Z

It's working for me. Try using

"assets": [
  "node_modules/sharp/**",
]

rightaway · 2022-07-16T07:46:58Z

@robertsLando I tried again and it's actually working even without "assets". But when building I get these warnings. The same happened in #229. Do you know the cause?

> [email protected]
compression:  GZip
> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules/sharp/build/Release
  %2: path-to-executable/sharp/build/Release
> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules/sharp/vendor/lib
  %2: path-to-executable/sharp/vendor/lib
> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules/sharp/build/Release
  %2: path-to-executable/sharp/build/Release
> Warning Cannot include directory %1 into executable.
  The directory must be distributed with executable as %2.
  %1: node_modules/sharp/vendor/lib
  %2: path-to-executable/sharp/vendor/lib

robertsLando added a commit that referenced this issue Mar 26, 2021

better sanitize modules path

bfa7cfa

Fixes #1075

robertsLando mentioned this issue Mar 26, 2021

better sanitize modules path #1103

Closed

jesec mentioned this issue Apr 8, 2021

Support RegExp mountpoints #1121

Merged

YiannisGkoufas mentioned this issue May 7, 2021

Cannot build using npm noobaa/noobaa-core#6489

Closed

tonywoode mentioned this issue Sep 27, 2021

cross platform native modules didn't work for me #1319

Closed

robertsLando added a commit that referenced this issue Sep 29, 2021

fix: better support of .node files

988397d

Fixes #1075

robertsLando mentioned this issue Sep 29, 2021

fix: better support of .node files #1321

Merged

jesec closed this as completed in #1321 Sep 30, 2021

tonywoode added a commit to tonywoode/quickPlayNode that referenced this issue Oct 1, 2021

[pkg] add yarn/npm build issue to script...

a128265

see vercel/pkg#1319, this makes the package fail because of vercel/pkg#1075 rather than fail in the same way because of the missinng dependency

mplutka pushed a commit to mplutka/pkg that referenced this issue Dec 18, 2021

better sanitize modules path

a23b3ae

Fixes vercel#1075

This was referenced Jul 11, 2022

Not working with sharp again "Cannot include directory %1 into executable." #1692

Closed

Can't use sharp with pkg lovell/sharp#3289

Closed

rightaway mentioned this issue Dec 16, 2022

Not working with sharp again "Cannot include directory %1 into executable." #1826

Closed

rightaway mentioned this issue Apr 20, 2023

Not working with sharp again "Cannot include directory %1 into executable." #1918

Closed

robertsLando mentioned this issue Jun 10, 2024

pkg packing uneeded build files for native modules yao-pkg/pkg#67

Closed

Bundling sharp doesn't work anymore since updating to 4.5.0 #1075

Bundling sharp doesn't work anymore since updating to 4.5.0 #1075

Comments

pkeuter commented Mar 23, 2021

david-mohr commented Mar 23, 2021

jesec commented Mar 23, 2021

leerob commented Mar 23, 2021

david-mohr commented Mar 23, 2021

pkeuter commented Mar 23, 2021

david-mohr commented Mar 23, 2021

pkeuter commented Mar 24, 2021

david-mohr commented Mar 24, 2021

leerob commented Mar 25, 2021

JeffJassky commented Mar 26, 2021 • edited Loading

The prebuild-install problem:

A Workaround:

Manually Installing cross-platform prebuilt binaries:

Unfortunately:

robertsLando commented Mar 26, 2021

JeffJassky commented Mar 26, 2021 • edited Loading

robertsLando commented Mar 26, 2021 • edited Loading

JeffJassky commented Mar 26, 2021 • edited Loading

robertsLando commented Mar 26, 2021 • edited Loading

JeffJassky commented Mar 26, 2021

JeffJassky commented Mar 26, 2021 • edited Loading

The Extended-Length Path Prefix problem

Workaround

Suggested Solution

robertsLando commented Mar 26, 2021 • edited Loading

JeffJassky commented Mar 26, 2021 • edited Loading

robertsLando commented Mar 26, 2021

JeffJassky commented Mar 26, 2021 • edited Loading

robertsLando commented Mar 26, 2021

JeffJassky commented Mar 26, 2021 • edited Loading

robertsLando commented Mar 26, 2021

JeffJassky commented Mar 26, 2021 • edited Loading

robertsLando commented Mar 26, 2021

JeffJassky commented Mar 26, 2021 • edited Loading

JeffJassky commented Mar 26, 2021 • edited Loading

robertsLando commented Apr 7, 2021 • edited Loading

rightaway commented Jun 17, 2021

rightaway commented Jul 18, 2021

robertsLando commented Jul 19, 2021

rightaway commented Jul 19, 2021

avpavp commented Jul 23, 2021

rightaway commented Jul 24, 2021

avpavp commented Jul 24, 2021

john-yick commented Aug 10, 2021

leadwolf commented Aug 15, 2021

robertsLando commented Sep 29, 2021 • edited Loading

robertsLando commented Sep 29, 2021

robertsLando commented Sep 29, 2021

rightaway commented Jul 15, 2022

robertsLando commented Jul 15, 2022

rightaway commented Jul 16, 2022

JeffJassky commented Mar 26, 2021 •

edited

Loading

JeffJassky commented Mar 26, 2021 •

edited

Loading

robertsLando commented Mar 26, 2021 •

edited

Loading

JeffJassky commented Mar 26, 2021 •

edited

Loading

robertsLando commented Mar 26, 2021 •

edited

Loading

JeffJassky commented Mar 26, 2021 •

edited

Loading

robertsLando commented Mar 26, 2021 •

edited

Loading

JeffJassky commented Mar 26, 2021 •

edited

Loading

JeffJassky commented Mar 26, 2021 •

edited

Loading

JeffJassky commented Mar 26, 2021 •

edited

Loading

JeffJassky commented Mar 26, 2021 •

edited

Loading

JeffJassky commented Mar 26, 2021 •

edited

Loading

JeffJassky commented Mar 26, 2021 •

edited

Loading

robertsLando commented Apr 7, 2021 •

edited

Loading

robertsLando commented Sep 29, 2021 •

edited

Loading