[FIX] Segmentation fault on CentOS 7 due to outdated sharp

[tassoevan]

Proposed changes (including videos or screenshots)

Upgrades sharp to avoid a segmentation fault on CentOS 7 during startup related to sharp.node being loaded via process.dlopen().

Suggested as a fix for versions 4.0.x and 4.1.x.

Issue(s)

Closes #23346
Closes #23427

Steps to test or reproduce

1. Build Rocket.Chat in a temporary directory:

   meteor build --server-only --directory /tmp/meteor-build

2. Setup a VirtualBox VM with CentOS 7 (minimal), SSH access (this requires a Bridged Adapter on networking), and rsync
3. Connect into VM via SSH (assuming user user and host IP 192.168.0.123)
4. Follow this section: https://docs.rocket.chat/quick-start/installing-and-updating/manual-installation/centos#install-necessary-dependency-packages
5. Follow only the steps related to MongoDB in this section: https://docs.rocket.chat/quick-start/installing-and-updating/manual-installation/centos#configure-the-rocket.chat-service
6. Create /opt/Rocket.Chat directory:

   sudo mkdir /opt/Rocket.Chat
   sudo chown tasso:tasso /opt/Rocket.Chat
   sudo chmod a+rwX /opt/Rocket.Chat

7. Logout from VM and copy the bundled files to it:

   rsync -avP --del /tmp/meteor-build/bundle/ [email protected]:/opt/Rocket.Chat/

8. Connect back into VM via SSH
9. Install dependencies and run Rocket.Chat:

   cd /opt/Rocket.Chat/programs/server
   npm install
   
   ROOT_URL=http://192.168.0.123:3000/ \
   MONGO_URL=mongodb://localhost:27017/rocketchat?replicaSet=rs01 \
   MONGO_OPLOG_URL=mongodb://localhost:27017/local?replicaSet=rs01 \
   PORT=3000 \
   node /opt/Rocket.Chat/main.js

Further comments

Found it. The segfault occurs when delete is called here https://github.com/libvips/libvips/blob/master/cplusplus/VImage.cpp#L113-L119 on a std::list, which immediately made me suspect https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html

The prebuilt sharp binary is built with C++11 "old" ABI but it looks like the libvips binaries for musl-based Linux were built with C++11 "new" ABI. This hasn't been a problem with the "oldmalloc" allocator, but it is incorrect behaviour between sharp and libvips, and musl's new "mallocng" allocator has exposed this.

I'm going to switch all prebuilt Linux binaries to use "new" ABI for the next release, so the forthcoming libvips v8.10.6 and sharp v0.28.0 binaries will be aligned for both glibc and musl Linux.

lovell/sharp-libvips@07de78c

Originally posted by @lovell in lovell/sharp#2570 (comment)

[mausballklauer]

We still have this problem on a Centos7 VM with 4.2.1 as well as 4.2.2.

Steps to reproduce:

Extract bundle

$ tar -xzf rocket.chat-4.2.2.tgz

Run npm

$ cd bundle/programs/server
$ npm install
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: this library is no longer supported

> [email protected] install /export/home/kvbapp/oli/bundle/programs/server/node_modules/fibers
> node build.js || nodejs build.js

[...]

added 147 packages from 122 contributors and audited 148 packages in 14.486s

3 packages are looking for funding
  run `npm fund` for details

found 1 high severity vulnerability
  run `npm audit fix` to fix them, or `npm audit` for details

Start Server

$ /opt/node-v12.22.1-linux-x64/bin/node bundle/main.js
Change Stream is available for your installation, give admin permissions to your database user to use this improved version.
LocalStore: store created at
LocalStore: store created at
LocalStore: store created at
{"level":51,"time":"2021-12-21T11:11:49.059Z","pid":16436,"hostname":"xxx","name":"Migrations","msg":"Migrating from version 232 -> 249"}
{"level":51,"time":"2021-12-21T11:11:49.060Z","pid":16436,"hostname":"xxx","name":"Migrations","msg":"Running up() on version 233"}
{"level":51,"time":"2021-12-21T11:11:49.066Z","pid":16436,"hostname":"xxx","name":"Migrations","msg":"Running up() on version 234"}
{"level":51,"time":"2021-12-21T11:11:49.083Z","pid":16436,"hostname":"xxx","name":"Migrations","msg":"Running up() on version 235"}
{"level":51,"time":"2021-12-21T11:11:49.100Z","pid":16436,"xxx","name":"Migrations","msg":"Running up() on version 236"}
{"level":51,"time":"2021-12-21T11:11:49.104Z","pid":16436,"xxx","name":"Migrations","msg":"Running up() on version 237"}
{"level":51,"time":"2021-12-21T11:11:49.145Z","pid":16436,"xxx","name":"Migrations","msg":"Running up() on version 238"}
{"level":51,"time":"2021-12-21T11:11:49.164Z","pid":16436,"xxx","name":"Migrations","msg":"Running up() on version 239"}
{"level":51,"time":"2021-12-21T11:11:49.166Z","pid":16436,"xxx","name":"Migrations","msg":"Running up() on version 240"}
{"level":51,"time":"2021-12-21T11:11:49.170Z","pid":16436,"hostname":"xxx","name":"Migrations","msg":"Running up() on version 241"}
{"level":51,"time":"2021-12-21T11:11:49.173Z","pid":16436,"hostname":"xxx","name":"Migrations","msg":"Running up() on version 242"}
{"level":51,"time":"2021-12-21T11:11:49.177Z","pid":16436,"hostname":"xxx","name":"Migrations","msg":"Running up() on version 243"}
{"level":51,"time":"2021-12-21T11:11:49.189Z","pid":16436,"hostname":"xxx","name":"Migrations","msg":"Running up() on version 244"}
{"level":51,"time":"2021-12-21T11:11:49.336Z","pid":16436,"hostname":"xxx","name":"Migrations","msg":"Running up() on version 245"}
{"level":51,"time":"2021-12-21T11:11:49.339Z","pid":16436,"hostname":"xxx","name":"Migrations","msg":"Running up() on version 246"}
{"level":51,"time":"2021-12-21T11:11:49.564Z","pid":16436,"hostname":"xxx","name":"Migrations","msg":"Running up() on version 247"}
{"level":51,"time":"2021-12-21T11:11:49.813Z","pid":16436,"hostname":"xxx","name":"Migrations","msg":"Running up() on version 248"}
{"level":51,"time":"2021-12-21T11:11:50.751Z","pid":16436,"hostname":"xxx","name":"Migrations","msg":"Running up() on version 249"}
{"level":51,"time":"2021-12-21T11:11:50.972Z","pid":16436,"hostname":"xxx","name":"Migrations","msg":"Finished migrating."}
Loaded the Apps Framework and loaded a total of 0 Apps!
+-------------------------------------------------------+
|                     SERVER RUNNING                    |
+-------------------------------------------------------+
|                                                       |
|  Rocket.Chat Version: 4.2.2                           |
|       NodeJS Version: 12.22.1 - x64                   |
|      MongoDB Version: 4.0.18                          |
|       MongoDB Engine: unknown                         |
|             Platform: linux                           |
|         Process Port: 11001                           |
|             Site URL: https://xxx.local  |
|     ReplicaSet OpLog: Enabled                         |
|          Commit Hash: ab3baf2f31                      |
|        Commit Branch: HEAD                            |
|                                                       |
+-------------------------------------------------------+
+----------------------------------------------------------------------+
|                              DEPRECATION                             |
+----------------------------------------------------------------------+
|                                                                      |
|  YOUR CURRENT MONGODB VERSION (4.0.18) IS DEPRECATED.                |
|  IT WILL NOT BE SUPPORTED ON ROCKET.CHAT VERSION 5.0.0 AND GREATER,  |
|  PLEASE UPGRADE MONGODB TO VERSION 4.2 OR GREATER                    |
|                                                                      |
+----------------------------------------------------------------------+
Segmentation fault (core dumped)

$ cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)

Are we missing something obvious here?

On former installs we had to set LD_PRELOAD="bundle/programs/server/npm/node_modules/sharp/vendor/lib/libz.so" but this file is not present any more.

[WPTechnician]

We are also having this issue moving from 3.17.1. Exactly as mausballklauer described in the previous post. Thanks!

[sampaiodiego]

@mausballklauer what node and npm versions are you running?

node -v
npm -v

[WPTechnician]

We are running:
npm -v 7.12.1
node -v v12.18.4

Sorry, I should mention that I did update node before attempting the update to 12.22.1 but it rolled back when I rolled back the server after the failed update.

[sampaiodiego]

please make sure to use the exact same versions as stated on the release notes.. in case of release 4.2.2: https://github.com/RocketChat/Rocket.Chat/releases/tag/4.2.2

Node: 12.22.1
NPM: 6.14.1

[mausballklauer]

We use:

$ node -v
v12.22.1
$ npm -v
6.14.12

NPM 6.14.12 is bundled with Node 12.22.1 so that's version we are using. Do you think it is neccessary to downgrade specifically to 6.14.1?

Nevertheless, I'll try to build it tomorrow and report back then.

[sampaiodiego]

NPM 6.14.12 is bundled with Node 12.22.1 so that's version we are using. Do you think it is neccessary to downgrade specifically to 6.14.1?

I'd say using the same minor version (so 6.14.x) should be enough.. but if it doesn't work with 6.14.12, it might be worth a try with 6.14.1

[mausballklauer]

Ok, this was unexpected... I downgraded NPM to 6.14.1 and now it works. The segmentation fault error is gone.

This resolves my actual issue, nevertheless it would be good fixing this so that the install procedure works with the NPM version that comes bundled with the Node version stated in the release notes.

Anyways, thanks for having a look into this!

[tassoevan]

@mausballklauer See #23346 (comment)

[mausballklauer]

Today, I wanted to start the instance of Rocket.Chat from my yesterdays comment where I stated that it was finally working.

To my great surprise, it was throwing the segmentation fault error on startup. What happened? It was working yesterday, why not today? Where was the difference between yesterday and today?

After some investigation and carefully retaking all the steps I did yesterday I finally found the difference. You must know, we are behind a corporate proxy with our servers, no direct communication towards "the internet" is allowed. During the downgrade procedure of NPM (from 16.14.12 to 6.14.1) I had to set the HTTPS_PROXY environment variable in order to get the NPM install.sh script working.

Setting this variable finally made a difference. When I try to start Rocket.Chat without this variable being set, it throws a segmentation fault. If I simply enter

export HTTPS_PROXY="http://our.corporate.proxy.local:8080/"

and then start RocketChat again - it starts and works fine, no segmentation faults!

So it wasn't the NPM version all along, it was the HTTPS_PROXY variable that had to be set.

Can you tell what is happening during startup and what could possibly be the reason why this environment variable has to be set in the shell starting the Rocket.Chat instance?