Provide pre-built "sharp.node" binary for common platforms and Node versions

[mikemorris]

https://github.com/mapbox/node-pre-gyp

[lovell]

Hi Mike, node-pre-gyp looks interesting, thank you for the suggestion.

From scanning the mapnik examples, it looks like it statically links dependencies into a single pre-compiled .node binary for each combination of architecture, platform and node version.

Would we also need to build and provide statically compiled versions of libvips and its dependants first?

Is there any licensing impact to consider when static linking both LGPL and GPL2 dependencies with Apache 2.0 sources.

I'd be happy to accept a PR that integrates with node-pre-gyp.

The existing preinstall.sh script could be used to populate a VM/container environment in which to build a statically linked version of libvips.

I'm able to provide an S3 bucket to host pre-compiled files in, if that's the easiest way to achieve this.

There are plenty of people to help test this amongst the contributors to the "meaning of life" issue that is #42.

[lovell]

Handy guide to using various CI providers as the build infrastructure for compiling binaries:
http://cylonjs.com/blog/2014/11/19/creating-multiplatform-precompiled-binaries-for-node-modules/

[lovell]

NPM's "Roadmap area of focus: native modules" is relevant here.

[lovell]

Most of my comment #186 (comment) about static linking is no longer relevant as sharp ships with pre-built shared libraries that can be used to generate a sharp.node shared library.

Windows should probably be the priority/test case for this work.

[lovell]

I'm going to investigate the use of prebuild-ci and (a possibly-modified version of) prebuild-install.

The libvips-downloading logic in the binding.* files will need to be extracted as node-gyp build may no longer be run.

[lovell]

Work-in-progress branch at https://github.com/lovell/sharp/tree/prebuild

• Move pkgconfig-based libvips detection logic from binding.gyp to JS
• Move download logic from binding.gyp to JS
• Move Windows-only DLL copy logic from binding.gyp to JS
• Integrate with prebuild-ci
• Integrate with prebuild-install
• Document
• Spend more time on Windows support
• Test, test, test

[elijahbee]

Can't wait for this, building from source is rather problematic

Failed at the [email protected] install script 'node-gyp rebuild'.

[lovell]

Should anyone be able to help test this, especially on Windows, please try running the following in a newly-created temporary directory.

npm install lovell/sharp#prebuild

yarn add lovell/sharp#prebuild

This should download and use pre-compiled libvips and sharp binaries. Please note that git needs to be available as this method is installing via GitHub but won't be required when published via npm.

If a globally-installed copy of libvips is detected then it will intentionally fallback to building sharp from source.

If it works, please +1 this comment.

If it doesn't work, please add a comment with full details of the platform used, all relevant version numbers and a link to a Gist with the install log.

[lovell]

Success on Windows, finally.

Here's the log when installing on a fresh Windows machine without MSVC etc.

C:\Temp>node -v
v9.5.0

C:\Temp>npm install lovell/sharp#prebuild

> [email protected] install C:\Temp\node_modules\sharp
> (node install/libvips && node install/dll-copy && prebuild-install) || (node-gyp rebuild && node install/dll-copy)

info sharp Downloading https://github.com/lovell/sharp-libvips/releases/download/v8.6.1/libvips-8.6.1-win32-x64.tar.gz
info sharp Creating C:\Temp\node_modules\sharp\build\Release
info sharp Copying DLLs from C:\Temp\node_modules\sharp\vendor\lib to C:\Temp\node_modules\sharp\build\Release
prebuild-install info begin Prebuild-install version 2.5.0
prebuild-install info looking for local prebuild @ prebuilds\sharp-v0.20.0-prebuild.test.1-node-v59-win32-x64.tar.gz
prebuild-install info looking for cached prebuild @ C:\...\AppData\Roaming\npm-cache\_prebuilds\https-github.com-lovell-sharp-releases-download-v0.20.0-prebuild.test.1-sharp-v0.20.0-prebuild.test.1-node-v59-win32-x64.tar.gz
prebuild-install info found cached prebuild
prebuild-install info unpacking @ C:\...\AppData\Roaming\npm-cache\_prebuilds\https-github.com-lovell-sharp-releases-download-v0.20.0-prebuild.test.1-sharp-v0.20.0-prebuild.test.1-node-v59-win32-x64.tar.gz
prebuild-install info unpack resolved to C:\Temp\node_modules\sharp\build\Release\sharp.node
prebuild-install info unpack required C:\Temp\node_modules\sharp\build\Release\sharp.node successfully
prebuild-install info install Successfully installed prebuilt binary!

+ [email protected]
added 70 packages in 43.907s

[xcb6]

win7, node -v (v8.9.3) => install ok
thanks

[MatousLeadPages]

win10, node -v (v8.9.4) => works great

[KyleAMathews]

Tested on mac and it worked perfectly (well... once I deleted the global libvips install I forgot I had :-))

[kolpav]

win10, v9.6.0, works perfectly

[Prozi]

Win10 finally without problems <3

[pronebird]

Does one still need to compile libvips for prebuilt to work?

[lovell]

@pronebird sharp has provided a prebuilt libvips since v0.15.0 for most platforms.

"If a globally-installed copy of libvips is detected then it will intentionally fallback to building sharp from source." #186 (comment)

So if for whatever reason you are still compiling libvips and installing it globally then the prebuilt sharp has to be ignored and instead will be compiled against and linked with the global libvips version.

[pronebird]

@lovell I wasn't aware of this.

Not that long ago I started seeing this error on my heroku-16 instance:

Error: libvips-cpp.so.42: cannot open shared object file: No such file or directory

ldd emits:

ldd node_modules/sharp/build/Release/sharp.node  on ⬢ staging... up, run.7438 (Free)
	linux-vdso.so.1 =>  (0x00007ffd11f85000)
	libvips-cpp.so.42 => not found
	libvips.so.42 => not found
	libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007fc9484d1000)
	libgobject-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x00007fc94827e000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc947efc000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc947bf3000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc9479dd000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc947613000)
	libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007fc9473a3000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc947186000)
	libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007fc946f7e000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fc948a1b000)

I think what happens is that node_modules/sharp/vendor/lib is being cleaned up by the heroku-postbuild script that I have, such as

yarn install --production=false && yarn run build && yarn install --production=true

Since I do some compilation on server I install dev dependencies first then compile and finally clean them up with yarn install --production=true. But that doesn't seem to be the case when using npm instead of yarn. Also I can't reproduce this locally on my mac.

[lovell]

The prebuild branch has now been merged to the master branch and will shortly be released as v0.20.0.

npm install lovell/sharp

yarn add lovell/sharp

[pronebird]

@lovell does it change anything in terms of configuration or what's the benefit?

[lovell]

@pronebird The primary benefit of providing prebuilt sharp.node binaries is that the current Python and C++ compiler install-time dependencies will no longer be required for most users, making installation simpler and easier.

The npm install --build-from-source flag (a "standard" used by both node-pre-gyp and prebuild-install) will be respected if you still wish to compile from source.

[pronebird]

@lovell ah right so it was still compiling sharp even though vips binaries were prebuilt. 👍

[lovell]

v0.20.0 is now available, thanks everyone for feedback/testing, and a big thank you to the sponsor of this work (that wishes to remain anonymous for now).