[Bug]: Misleading error on old docker version (19), node segfaults

[maaaax]

Describe the bug

At least with 19.03.13 opensearch and dashboards 2.10.0 containers are unable to start.
Dashboards exits with
"Could not find a Node.js runtime binary bundled with OpenSearch Dashboards or on the system" (bin/use_node)

Actually node is segfaulting (nodejs/node#43064)
ulimits have been set, but no effect.

starts:
docker run -it opensearchproject/opensearch-dashboards:2.9.0
doesn't start:
docker run -it opensearchproject/opensearch-dashboards:2.10.0

Furthermore, opensearch container exits with java not found in JAVA_HOME. I haven't looked into this, just upgraded docker to get it working.

To reproduce

Install

Client: Docker Engine - Community
Version: 19.03.13
API version: 1.40
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:03:45 2020
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.13
API version: 1.40 (minimum version 1.12)
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:02:21 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.3.7
GitCommit: 8fba4e9a7d01810a393d5d25a3621dc101981175
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683

Try to run opensearch.

Expected behavior

Print some error message that the binary exists but fails to start.

Screenshots

If applicable, add screenshots to help explain your problem.

Host / Environment

No response

Additional context

No response

Relevant log output

No response

[Divyaasm]

Hi @maaaax, pls let us know if you are still facing the issue or it's resolved. Thanks

[maaaax]

Hi,

I have updated docker to >19 and everything works fine with that. So, for me not a problem. But maybe the start scripts should not write something about missing binaries (node,java) when they are actually just segfaulting.

[prudhvigodithi]

[Untriage]
Hey @maaaax looks to me this is coming from the OpenSearch Dashboards itself and not specific to the docker image.
Related PR's that might cause this issue.
https://github.com/opensearch-project/OpenSearch-Dashboards/pull/4151/files
opensearch-project/OpenSearch-Dashboards#4237
@AMoo-Miki can you please add your thoughts?

Adding @peterzhuamazon @gaiksaya @bbarani should this be moved to OpenSearch Dashboards repo?

[gaiksaya]

Adding @peterzhuamazon @gaiksaya @bbarani should this be moved to OpenSearch Dashboards repo?

Yeah makes sense. If the docker image or any path needs to change in order to resolve this I believe dashboards team would be able to point it out.

[bbarani]

@AMoo-Miki @joshuarrrr @kavilla @ashwin-pc Can you please provide your inputs here?

[AMoo-Miki]

Dashboards exits with
"Could not find a Node.js runtime binary bundled with OpenSearch Dashboards or on the system" (bin/use_node)

use_node logic hasn't changed between OSD 2.9 and 2.10. However, OSD 2.9 shipped with Node.js v16 while OSD 2.10 shipped with Node.js v18. Based on the comments though, i think that neither of the Node.js binaries shipped are working on that environment which is odd. Even if the environment cannot run Node.js v18, the fallback, Node.js v14 should work.

Will need to recreate this on my own to figure what is wrong.

I am not sure why OpenSearch wouldn't work though.

[maaaax]

Seems both are not working:

[opensearch-dashboards@90a3a8c1a2d4 ~]$ ./node/bin/node
./node/bin/node[16]: ../src/node_platform.cc:68:std::unique_ptr<long unsigned int> node::WorkerThreadsTaskRunner::DelayedTaskScheduler::Start(): Assertion `(0) == (uv_thread_create(t.get(), start_thread, this))' failed.
 1: 0xb7a940 node::Abort() [./node/bin/node]
 2: 0xb7a9be  [./node/bin/node]
 3: 0xbe98be  [./node/bin/node]
 4: 0xbe99a1 node::NodePlatform::NodePlatform(int, v8::TracingController*, v8::PageAllocator*) [./node/bin/node]
 5: 0xb38f5b node::InitializeOncePerProcess(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, node::ProcessFlags::Flags) [./node/bin/node]
 6: 0xb395ab node::Start(int, char**) [./node/bin/node]
 7: 0x7fdbb5a80eb0  [/lib64/libc.so.6]
 8: 0x7fdbb5a80f60 __libc_start_main [/lib64/libc.so.6]
 9: 0xabbdee _start [./node/bin/node]
Aborted (core dumped)

[opensearch-dashboards@90a3a8c1a2d4 ~]$ ./node/fallback/bin/node
./node/fallback/bin/node[17]: ../src/node_platform.cc:61:std::unique_ptr<long unsigned int> node::WorkerThreadsTaskRunner::DelayedTaskScheduler::Start(): Assertion `(0) == (uv_thread_create(t.get(), start_thread, this))' failed.
 1: 0xa3ad50 node::Abort() [./node/fallback/bin/node]
 2: 0xa3adce  [./node/fallback/bin/node]
 3: 0xaab67e  [./node/fallback/bin/node]
 4: 0xaab746 node::NodePlatform::NodePlatform(int, v8::TracingController*) [./node/fallback/bin/node]
 5: 0xa03445 node::InitializeOncePerProcess(int, char**) [./node/fallback/bin/node]
 6: 0xa03971 node::Start(int, char**) [./node/fallback/bin/node]
 7: 0x7f9a4afddeb0  [/lib64/libc.so.6]
 8: 0x7f9a4afddf60 __libc_start_main [/lib64/libc.so.6]
 9: 0x98c5bc  [./node/fallback/bin/node]
Aborted (core dumped)

[opensearch-dashboards@90a3a8c1a2d4 ~]$ md5sum ./node/bin/node
3a6340fd551f42e26e9fb4647add7029  ./node/bin/node
[opensearch-dashboards@90a3a8c1a2d4 ~]$ md5sum ./node/fallback/bin/node
14f81269fb91b3765b2d820ac61aba38  ./node/fallback/bin/node

[maaaax]

2.9.0 works within the same setup:

docker run -it --rm opensearchproject/opensearch-dashboards:2.9.0 bash

[opensearch-dashboards@12dd23044b26 ~]$ ./node/bin/node --version
v16.20.0
[opensearch-dashboards@12dd23044b26 ~]$ ./node/fallback/bin/node --version
v14.21.3

[opensearch-dashboards@12dd23044b26 ~]$ md5sum ./node/bin/node
9f99a1ebb0b6258cb289668cb2f7d211  ./node/bin/node
[opensearch-dashboards@12dd23044b26 ~]$ md5sum  ./node/fallback/bin/node
14f81269fb91b3765b2d820ac61aba38  ./node/fallback/bin/node

[maaaax]

I think it's solved.

docker run -it --rm opensearchproject/opensearch:2.10.0 java -version
[0.007s][warning][os,thread] Failed to start thread "GC Thread#0" - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.

this, by googling, led me to adding "--security-opt seccomp=unconfined" to docker run.

java, node 14 and 18 are working with that option!

[jordarlu]

Hi, @maaaax , happy to learn that you have found the way out on this issue.
I wasn't able to mimic the entire enviorment to reproduce the error actually ..but seems you are good to move forward now, I am going to close this issue, please reopen it if you are still facing the same error, thanks a lot!!

$ docker run -it --rm opensearchproject/opensearch-dashboards:2.10.0 bash
Unable to find image 'opensearchproject/opensearch-dashboards:2.10.0' locally
2.10.0: Pulling from opensearchproject/opensearch-dashboards
6a2bb1614d2d: Pull complete 
2a5558b59856: Pull complete 
fee9452241ea: Pull complete 
4cd4143d3f95: Pull complete 
61cc9912a0bd: Pull complete 
4f4fb700ef54: Pull complete 
Digest: sha256:485a0019e5d62c8b98ba43b004656a4f9a0bad3fb397ba18b5ab0b64e3f23e15
Status: Downloaded newer image for opensearchproject/opensearch-dashboards:2.10.0
[opensearch-dashboards@220ef219d1f2 ~]$ ./node/bin/node
Welcome to Node.js v18.16.0.
Type ".help" for more information.
> .exit
[opensearch-dashboards@220ef219d1f2 ~]$ ./node/bin/node --version
v18.16.0
[opensearch-dashboards@220ef219d1f2 ~]$