Latest NodeJS 4.9.0, 6.14.0, 8.11.0, 9.9.0 and 9.10.0 not running on PPCLE Red Hat Enterprise Linux Server release 7.3 #19530

Flarna · 2018-03-22T10:29:37Z

Version: 4.9.0, 6.14.0, 8.11.0, 9.9.0 and 9.10.0
Platform: PPCLE
Subsystem: build

I found that NodeJS 9.9.0 is not running on our PPCLE machine, NodeJS 9.8.0 was fine.

Problem reported: /lib64/libstdc++.so.6: version 'GLIBCXX_3.4.20' not found

I did some objdump and I found following symbol in 9.9.0 which was not used in 9.8.0:

0000000000000000      DF *UND*	0000000000000000  GLIBCXX_3.4.20 _ZSt24__throw_out_of_range_fmtPKcz
demangled:
0000000000000000      DF *UND*	0000000000000000  GLIBCXX_3.4.20 std::__throw_out_of_range_fmt(char const*, ...)

Is a new compiler version used starting with 9.9.0? And is this intended?

Some details about our machine:
Red Hat Enterprise Linux Server release 7.3
ldd (GNU libc) 2.17
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16)

The text was updated successfully, but these errors were encountered:

gireeshpunathil · 2018-03-22T10:47:04Z

/cc @nodejs/build

gibfahn · 2018-03-22T15:52:04Z

cc/ @jBarz @mhdawson @refack , but I think we have moved up to gcc 4.9 from gcc 4.8 recently.

See also nodejs/build#1020

Flarna · 2018-03-22T16:21:22Z

I think use of gcc 4.9 could explain this.
But I don't see this dependency for x86 binaries. Are x86 binaries still built with 4.8?

bnoordhuis · 2018-03-22T16:39:16Z

Hm, the problem is this (from here):

GCC 4.8.0: GLIBCXX_3.4.18, CXXABI_1.3.7
GCC 4.8.3: GLIBCXX_3.4.19, CXXABI_1.3.7
GCC 4.9.0: GLIBCXX_3.4.20, CXXABI_1.3.8  <- bump

I suppose the solution is to revert the 4.8->4.9 upgrade in <= v9.x.

jBarz · 2018-03-22T17:44:49Z

Using rpath option at the link step will force the binary to use the 4.9 libraries?
We should be using the following in the environment.

export CC=/usr/bin/gcc-4.9
export CXX=/usr/bin/g++-4.9
export LINK=/usr/bin/g++-4.9
export LDFLAGS=“-Wl,-rpath,$(dirname $($CC --print-file-name libgcc_s.so))”

Flarna · 2018-03-23T09:38:26Z

@jBarz The problem is that there are no 4.9 libs for PPCLE Red Hat Enterprise Linux Server 7.x.

Seems 9.9.0 was at least partly built with GCC 4.9.4:

$ readelf -p .comment portable-nodejs-linux-ppcle_64-9.9.0/node 

String dump of section '.comment':
  [     0]  GCC: (Ubuntu 4.9.4-2ubuntu1~14.04.1) 4.9.4
  [    2b]  GCC: (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4

$ readelf -p .comment portable-nodejs-linux-ppcle_64-9.8.0/node 

String dump of section '.comment':
  [     0]  GCC: (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4

As reference for x86 I get:

$ readelf -p .comment portable-nodejs-linux-x86_64-9.9.0/node 

String dump of section '.comment':
  [     0]  GCC: (GNU) 4.4.7 20120313 (Red Hat 4.4.7-18)
  [    2d]  GCC: (GNU) 4.8.2 20140120 (Red Hat 4.8.2-15)

$ readelf -p .comment portable-nodejs-linux-x86_64-9.8.0/node 

String dump of section '.comment':
  [     0]  GCC: (GNU) 4.4.7 20120313 (Red Hat 4.4.7-18)
  [    2d]  GCC: (GNU) 4.8.2 20140120 (Red Hat 4.8.2-15)

Flarna · 2018-03-23T09:40:13Z

I suppose the solution is to revert the 4.8->4.9 upgrade in <= v9.x.

Does this indicate that NodeJS 10 will use GCC 4.9 and as a result doesn't work out of the box on our Red Hat Enterprise Linux Server 7.x?

bnoordhuis · 2018-03-23T14:05:00Z

That seems likely.

gibfahn · 2018-03-23T20:12:32Z

Does this indicate that NodeJS 10 will use GCC 4.9 and as a result doesn't work out of the box on our Red Hat Enterprise Linux Server 7.x?

V8 has started to use C++14 features (AIUI) which will necessitate gcc 4.9. However if it's not straightforward to get the gcc 4.9 libstdc++ on RHEL 7, then we probably need to document how it should be done for Node users.

I guess alternatively it might be a good time to consider statically linking against libstdc++.

bnoordhuis · 2018-03-24T09:09:30Z

I guess alternatively it might be a good time to consider statically linking against libstdc++.

I've thought about that (but libc++ because license) but I foresee an endless stream of issues with native modules.

For example, consider a module foo that links against /usr/lib/libfoo.so. If the library links against a different C++ runtime, the spectrum of things that can go wrong runs from link-time errors to segfaults and data corruption.

Longer term it might be worth it but it's probably not going to be an easy transition. I don't know if we could get it ready in time for Node.js 10 and have all the kinks worked out.

cosinusoidally · 2018-03-25T12:07:04Z

I guess alternatively it might be a good time to consider statically linking against libstdc++.

I've thought about that (but libc++ because license) but I foresee an endless stream of issues with native modules.

Don't you build your x86/x64 release binaries with some version of devtoolset on CentOS? If so, you are already statically linking to parts of libstdc++ on x86/x64. In devtoolset its copy of libstdc++.so looks something like:

cat /opt/rh/devtoolset-4/root/usr/lib/gcc/x86_64-redhat-linux/5.3.1/libstdc++.so
/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf64-x86-64)
INPUT ( /usr/lib64/libstdc++.so.6 -lstdc++_nonshared )

which is a linker script that dynamically links to the system libstdc++.so (which for CentOS 6 will be the one from GCC 4.4) and then statically links in any missing libstdc++ symbols using libstdc++_nonshared.a (the symbols in libstdc++_nonshared.a are the ones introduced by the newer version of GCC that ships with a given devtoolset release). This allows binaries to run on CentOS 6 without needing to ship a newer version of libstdc++.so with your program.

Maybe the solution for PPCLE RHEL would be to build with a devtoolset (if available)?

Flarna · 2018-03-28T21:15:45Z

Toolchain seems to be unchanged; 9.10.0 seems to be like 9.9.0.
I already ordered an Ubuntu 16.04 machine to run test with NodeJs >=9 tests 😀
For building our native addon we keep the old one to avoid raising glibc requirements. We don't use node in build step so it's no issue there.

So from my point of view this issue can be closed as the root cause is understood and it's clear that there is no way around for NodeJS 10 to raise the requirements. I don't care much about the final NodeJS 9 releases.

Flarna · 2018-03-29T09:06:55Z

Just found that all fresh NodeJS versions are effected including 4.9.0, 6.14.0, 8.11.0.

mhdawson · 2018-03-29T16:18:35Z

Ok, the problem seems to be that we were staging the 4.9 compiler and I thought the default would stay as 4.8, but it looks like it is actually 4.9.,

I'll add the selection logic and we can validate that it resolves the issue on 4.X and 6.x. The lowest stated version was 4.9 for 8.X so the selection logic uses 4.9 for 8.X and above. However, we'll have to consider that again given the info in this thread.

Did anybody check if this affects x86 as well ?

Flarna · 2018-03-29T16:20:53Z

x86 is not effected

mhdawson · 2018-03-29T16:31:24Z

@Flarna so on x86 4.9 is compatible with 4.8 but on PPCLE that is not the case?

mhdawson · 2018-03-29T16:33:36Z

@Flarna is there a version newer than 4.9 that is easier to get on RHEL 7.X ?

Flarna · 2018-03-29T16:43:05Z

so on x86 4.9 is compatible with 4.8 but on PPCLE that is not the case?

To me it looks more like x86 is still built with GCC 4.8. See my #19530 (comment) above where I used readelf to get GCC infos.

is there a version newer than 4.9 that is easier to get on RHEL 7.X ?

Honestly speaking I don't know what is available as I'm just an user.

In our addon build we use GCC 6.4.0 but it's a self built GCC patched for our needs. We link statically to avoid issues like this but I don't think our solution is directly applicable to NodeJS. In our addon we take care to not expose any C++ symbol but NodeJS can't do that as it has to export V8 API.

mhdawson · 2018-03-29T16:43:26Z

@Flarna can you test out this version:

https://nodejs.org/download/test/v6.14.0-test2c93af2da3/node-v6.14.0-test2c93af2da3-linux-ppc64le.tar.xz

It is the 6.14.0 release rebuilt with the detection logic so that it built with the 4.8 compiler.

If that works then we can decide if we should change the logic for 8.x and work on respining the builds.

MylesBorins · 2018-03-29T17:04:02Z

Should we be planning for a member patch release for all lines?

Flarna · 2018-03-29T17:09:04Z

@mhdawson Yes, this version works.

mhdawson · 2018-03-29T17:18:59Z

@Flarna thanks

@MylesBorins, not sure what you mean by a member patch release, but what we need is to rebuild the le releases for 4, 6, and 8 (for now I'm just going to make 8 build with 4.8 instead of 4.9 until we figure out the story across platforms).

There are no code changes so I think we could just rebuild and republish for ppcle unless that's not something we want to do.

I still need to do one update to get it right for 8 and then we could rebuild with the ppcle job I created.

mhdawson · 2018-03-29T17:30:00Z

@MylesBorins can you approve nodejs/build#1201. Once that lands and I validate with a build in test, we can respin the builds.

richardlau · 2018-03-29T17:42:52Z

We generally don't replace already published builds to avoid confusion. Usual practice is to do a new patch release.

mhdawson · 2018-03-29T21:19:45Z

Tested the new versions, they all run -version as well as a starting the console and being able to print 'test'. I think they are good.

MylesBorins · 2018-03-30T03:50:43Z

New versions are released. Closing the issue. Please let me know if it isn't fixed and I'll reopen

Flarna · 2018-04-03T08:45:33Z

I can confirm that all four NodeJs versions run fine on our RedHat Enterprise 7.3.

mukeshkmr776 · 2019-07-29T18:13:51Z

@mhdawson
I am having same problem for NodeJS-10.15.3 for PPCLE (RedHat 7.x).
Really exhausted trying finding solution for this.

Please confirm if same problem exists for NodeJS-10.15.3 also.

And, suggest some way to make NodeJS-10.15.3 work for RHEL 7.x (PPCLE) (Power8)

It is making me cry :'(

mukeshkmr776 · 2019-08-04T20:08:48Z

@mhdawson @MylesBorins

Flarna · 2019-08-05T08:59:34Z

@mukeshkmr776 NodeJs 10 and newer uses gcc 4.9 and as a result it wont run on these old platforms.
see #19530 (comment)

mukeshkmr776 · 2019-08-05T14:49:07Z

@Flarna Thanks for confirming this. Now, I am not getting gcc version >=4.9.x even after updating my server rpm through yum. Could you please provide me the binary libstdc++.so.6 or complete rpm or a link to it so that I can move ahead. :(

One observation, then how come Nodejs 10.15.3 is working on my another system of RHEL 7.x (ppc) with gcc-4.8.x ???? :/

Flarna · 2019-08-05T14:59:32Z

For Node.Js < 10 this was fixed in node builds as they stepped back to gcc 4.8.
We haven't found a solution for Red Hat, we moved to Ubuntu instead.

One observation, then how come Nodejs 10.15.3 is working on my another system of RHEL 7.x (ppc) with gcc-4.8.x ???? :/

Sorry, don't know but maybe ppc doesn't require a newer libstdc++ but pccle does. As far as I remember we had no problems on x86 either that time, only ppcle (we never used ppc).

mukeshkmr776 · 2019-08-07T19:21:41Z

@Flarna So what should I do then? :| Nodejs v10 working on ppc64, but on ppc64le. Though both had same gcc 4.8.x versions :/

Flarna · 2019-08-07T22:12:08Z

NodeJs v10 uses gcc 4.9.4 not 4.8. I would say if it works on ppc64 it's just luck. As mentioned above x86 worked also for me that time.
Moving from gcc 4.8.3 to 4.9 moved GLIBCXX from 3.4.19 to 3.4.20. As far as I understood the versioning of glibcxx the exported symbols are versioned. Only changed exports get a new version and which ones are effected depends on the platform and which ones are actually used depends on the application.

I think you have just two options (I assume RedHat does not provide anything helpful here):

try to build gcc 4.9 toolchain by yourself, I expect it's not easy...
move to a different system (like we did)

Even if you get Node 10 running it will be not a future prove solution as NodeJs 12 has moved to gcc 6.3 (GLIBCXX 3.4.22) as far as I know.

justinreock-roguewave · 2019-09-05T12:12:21Z

Was this fix applied to the build process for the ppc64le binaries? nodejs/build#1548

I've been looking into this issue the last couple of days and it really is a mystery that the build for ppc64le is linked to 3.4.20. x86_64 builds are linked to the same version of gcc and work fine with 4.8.5, which provides 3.4.19:

`

It seems to me like a bug in the build process that the ppc64le binaries are linking against 3.4.20.

This only affects 10.x and later versions, as we know. I'm running a build of 10.15.2 manually on a ppc64le machine and seeing if I can force it to use gcc 4.8, will see if that does anything, but in the mean time can we check to see if the fix applied above was applied to the ppc64le build pipeline?

justinreock-roguewave · 2019-09-05T12:51:12Z

Further evidence that this is a broken build process. When I dump out the comment section of the binaries for ppc vs x86 I get very, very different results:

x86_64:
[root@personal-justin-amqload1 bin]# readelf -p .comment node

String dump of section '.comment':
[ 0] GCC: (GNU) 4.4.7 20120313 (Red Hat 4.4.7-23)
[ 2d] GCC: (GNU) 6.3.1 20170216 (Red Hat 6.3.1-3)

[root@personal-justin-amqload1 bin]#

ppc64le:
[jreock@dc3-zsnode1 bin]$ readelf -p .comment node

String dump of section '.comment':
[ 0] GCC: (Ubuntu 4.9.4-2ubuntu114.04.1) 4.9.4
[ 2b] GCC: (Ubuntu 4.8.4-2ubuntu114.04.3) 4.8.4

[jreock@dc3-zsnode1 bin]$

sam-github · 2019-09-05T16:19:31Z

We are in process of beginning to build 10x/ppcle on centos7 with devtools-6, which should extend the backwards compatibility of the official binaries to even older machines than those currently supported. See nodejs/build#1907 among others. We hope this will be released in the next month or so.

justinreock-roguewave · 2019-09-05T17:34:45Z

Thanks for the reply, Sam – this is a great approach. Will look forward to the push, let me know if I can help at all!

…

-Justin From: Sam Roberts <[email protected]> Reply-To: nodejs/node <[email protected]> Date: Thursday, September 5, 2019 at 12:22 PM To: nodejs/node <[email protected]> Cc: Justin Reock <[email protected]>, Comment <[email protected]> Subject: Re: [nodejs/node] Latest NodeJS 4.9.0, 6.14.0, 8.11.0, 9.9.0 and 9.10.0 not running on PPCLE Red Hat Enterprise Linux Server release 7.3 (#19530) We are in process of beginning to build 10x/ppcle on centos7 with devtools-6, which should extend the backwards compatibility of the official binaries to even older machines than those currently supported. See nodejs/build#1907<nodejs/build#1907> among others. We hope this will be released in the next month or so. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#19530?email_source=notifications&email_token=AFUQ75SRYPDIBKY7DUWJPQTQIEW33A5CNFSM4EW3GJ7KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD57XFKA#issuecomment-528446120>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFUQ75WKZOCLPFTIBDBZSGTQIEW33ANCNFSM4EW3GJ7A>.

sam-github · 2019-09-05T17:43:05Z

Once we have builds, it would be useful to confirm for us they help your situation.

justinreock-roguewave · 2019-09-05T17:43:52Z

Absolutely! I'll be keeping an eye on this and will confirm functionality.

mukeshkmr776 · 2019-09-17T14:09:09Z

Please update once this is fixed for Nodejs-v10.x on PPC64LE (RHEL-7.x).
Eagerly waiting on this

mhdawson · 2019-09-25T20:36:05Z

The lastest 10.x Nightly was build so that it should run there now. The next Node.js 10.x release should address this as well. @BethGriggs can you chime in with the target for that?

mukeshkmr776 · 2019-09-25T21:56:16Z

@mhdawson I guess it will be in upcoming release i.e. v10.16.4.... Do we have release date for this upcoming release?

sam-github · 2019-09-26T03:26:39Z

First -rc within next couple days, release in next 2-3 weeks.

mukeshkmr776 · 2019-09-26T08:37:38Z

@sam-github Thanks for the info. : )

mukeshkmr776 · 2019-10-01T12:05:08Z

@sam-github: is -rc available? 😕

sam-github · 2019-10-01T16:00:58Z

@BethGriggs said "soonish" this morning, I believe.

Maybe nightlies would be interesting to you? https://nodejs.org/download/nightly/v10.16.4-nightly20190923859d47593e9fd97e6a9eae6eeb8b6e910acfe434/

mukeshkmr776 · 2019-10-07T09:43:29Z

@sam-github Thanks for the link. But can't be used in production. : (
Please share link once RC/Official one is available.

Flarna · 2019-10-07T11:08:08Z

I tried this in one of our centos 7 containers with following result: 12.11.0, 12.11.1 and above nightly of 10.16.4 work.

so looks good!

as reference older node.js versions in same container:
12.10.0 fails with

/.nvm/versions/node/v12.10.0/bin/node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by ./.nvm/versions/node/v12.10.0/bin/node)
./.nvm/versions/node/v12.10.0/bin/node: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by ./.nvm/versions/node/v12.10.0/bin/node)

10.16.3 fails with

./.nvm/versions/node/v10.16.3/bin/node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by ./.nvm/versions/node/v10.16.3/bin/node)

sam-github · 2019-10-07T14:48:12Z

@Flarna thanks for the feedback

mukeshkmr776 · 2019-10-16T13:10:08Z

@sam-github
@BethGriggs
Not yet released? : (

BethGriggs · 2019-10-16T13:42:49Z

@mukeshkmr776, the proposal is at #29875 and due to be released 22nd October

mukeshkmr776 · 2019-10-16T18:47:37Z

@BethGriggs Thanks Beth! 👍 phew

addaleax added build Issues and PRs related to build files or the CI. ppc Issues and PRs related to the Power architecture. v9.x labels Mar 22, 2018

Flarna changed the title ~~NodeJS 9.9.0 not running on PPCLE Red Hat Enterprise Linux Server release 7.3~~ NodeJS 9.9.0 and 9.10.0 not running on PPCLE Red Hat Enterprise Linux Server release 7.3 Mar 28, 2018

Flarna changed the title ~~NodeJS 9.9.0 and 9.10.0 not running on PPCLE Red Hat Enterprise Linux Server release 7.3~~ Latest NodeJS 4.9.0, 6.14.0, 8.11.0, 9.9.0 and 9.10.0 not running on PPCLE Red Hat Enterprise Linux Server release 7.3 Mar 29, 2018

mhdawson mentioned this issue Mar 29, 2018

jenkins: change compiler level for 8.X, 9.X nodejs/build#1201

Closed

MylesBorins closed this as completed Mar 30, 2018

Latest NodeJS 4.9.0, 6.14.0, 8.11.0, 9.9.0 and 9.10.0 not running on PPCLE Red Hat Enterprise Linux Server release 7.3 #19530

Latest NodeJS 4.9.0, 6.14.0, 8.11.0, 9.9.0 and 9.10.0 not running on PPCLE Red Hat Enterprise Linux Server release 7.3 #19530

Comments

Flarna commented Mar 22, 2018 • edited Loading

gireeshpunathil commented Mar 22, 2018

gibfahn commented Mar 22, 2018

Flarna commented Mar 22, 2018

bnoordhuis commented Mar 22, 2018

jBarz commented Mar 22, 2018

Flarna commented Mar 23, 2018 • edited Loading

Flarna commented Mar 23, 2018

bnoordhuis commented Mar 23, 2018

gibfahn commented Mar 23, 2018

bnoordhuis commented Mar 24, 2018

cosinusoidally commented Mar 25, 2018

Flarna commented Mar 28, 2018

Flarna commented Mar 29, 2018

mhdawson commented Mar 29, 2018

Flarna commented Mar 29, 2018

mhdawson commented Mar 29, 2018

mhdawson commented Mar 29, 2018

Flarna commented Mar 29, 2018

mhdawson commented Mar 29, 2018 • edited Loading

MylesBorins commented Mar 29, 2018

Flarna commented Mar 29, 2018

mhdawson commented Mar 29, 2018

mhdawson commented Mar 29, 2018

richardlau commented Mar 29, 2018

mhdawson commented Mar 29, 2018

MylesBorins commented Mar 30, 2018

Flarna commented Apr 3, 2018

mukeshkmr776 commented Jul 29, 2019

mukeshkmr776 commented Aug 4, 2019

Flarna commented Aug 5, 2019

mukeshkmr776 commented Aug 5, 2019

Flarna commented Aug 5, 2019

mukeshkmr776 commented Aug 7, 2019

Flarna commented Aug 7, 2019

justinreock-roguewave commented Sep 5, 2019

justinreock-roguewave commented Sep 5, 2019

sam-github commented Sep 5, 2019

justinreock-roguewave commented Sep 5, 2019 via email

sam-github commented Sep 5, 2019

justinreock-roguewave commented Sep 5, 2019

mukeshkmr776 commented Sep 17, 2019

mhdawson commented Sep 25, 2019

mukeshkmr776 commented Sep 25, 2019

sam-github commented Sep 26, 2019

mukeshkmr776 commented Sep 26, 2019

mukeshkmr776 commented Oct 1, 2019 • edited Loading

sam-github commented Oct 1, 2019

mukeshkmr776 commented Oct 7, 2019

Flarna commented Oct 7, 2019

sam-github commented Oct 7, 2019

mukeshkmr776 commented Oct 16, 2019

BethGriggs commented Oct 16, 2019

mukeshkmr776 commented Oct 16, 2019 • edited Loading

Flarna commented Mar 22, 2018 •

edited

Loading

Flarna commented Mar 23, 2018 •

edited

Loading

mhdawson commented Mar 29, 2018 •

edited

Loading

mukeshkmr776 commented Oct 1, 2019 •

edited

Loading

mukeshkmr776 commented Oct 16, 2019 •

edited

Loading