Make webworker-threads a peer-dependency

[OoDeLally]

I'm running natural in Electron.
Natural crashes when imported on electron, because it tries to use webworker-threads, which only works on node.
It works fine when I delete the webworker-threads folder from node_modules folder. But this is tedious.
As far as I know there is no way to specify in my package.json not to install the webworker-threads optional dependency of natural (this seems to exist only on command-line with --no-optional).

A solution to that could be to set webworker-threads as peer-dependencies instead of optional-dependencies in the package.json. The user would have the choice to include webworker-threads in the case he wants it.

Also, electron also has builtin webworkers, so it would be nice if natural can use them. But for now, it would be quite a low-hanging fruit not to use them at all.

[caseywreed]

This comment helped me a ton. Was trying to get natural running with an AWS Lambda function and AWS crashed because of webworker-threads. Deleted the folder and I was good to go.

For a further explanation of the problem, I found this great blog post. It might help out, too. Thanks again!

[theSiberman]

Same issue running on IBM OpenWhisk

[kkoch986]

makes sense to me ill set that up asap

[rawpixel-vincent]

On server side, I had to delete the webworker-threads folder in node_modules before starting the app. Otherwise I got a runtime error.

[copitz]

👍

However, @OoDeLally, electron-rebuild seems to help with this

[leonardorifeli]

I'm have the same problem in my function (aws lambda).

2018-04-25T03:43:07.586Z	c1686f76-483a-11e8-a644-4958112b2bf6	Error: /var/task/node_modules/webworker-threads/build/Release/WebWorkerThreads.node: undefined symbol: _ZN2v811ArrayBuffer9Allocator7ReserveEm
at Error (native)
at Object.Module._extensions..node (module.js:597:18)
at Module.load (module.js:487:32)
at tryModuleLoad (module.js:446:12)
at Function.Module._load (module.js:438:3)
at Module.require (module.js:497:17)
at require (internal/module.js:20:19)
at bindings (/var/task/node_modules/bindings/bindings.js:81:44)
at Object.<anonymous> (/var/task/node_modules/webworker-threads/index.js:1:105)
at Module._compile (module.js:570:32)

But, I'm using circleci to deploy my function.

I'm can not send node_modules because I'm versioning the function with github. It's not a good pattern.

@caseywreed / @kkoch986 do you have a suggestion for that?

[leonardorifeli]

@caseywreed / @kkoch986 I solved my problem.

In the circleci configuration file, I set it to use the lambci / lambda: build-nodejs6.10 docker image

docker:
     - image: lambci / lambda: build-nodejs6.10

Thanks.

[3rd-Eden]

webworker-threads should be completely opt-in, I don't want to install a native dependency just because I try to a simple NLP task.

[Hugo-ter-Doest]

What does that mean for the dependencies in package.json. Should it be in optional-dependencies or in peer-dependencies?

Hugo

[3rd-Eden]

Optional dependencies are still being installed by default, but npm will not throw a tantrum when it fails to compile or install. With peerDependencies they not installed, but the user is presented with a warning if they do no install the said dependency them selfs upon installation. Last option is non of the above and remove the dependency from the package.json completely. If people want to leverage the webworker-threads package they can install it them selfs. npm flattens the dependency tree by default so if a user installs it, then the natural package will be able to access it as well.

The last option is what we've decided implement for the ws library (which I maintain) as well as it has optional binary modules that can increase speed and validation of websocket frames. It still functions without it. We tried optionalDependencies as well, but found that in rare cases it would still stop the npm installation process when the compilation failed.

[Hugo-ter-Doest]

Ok. I simplified things as follows. I removed the webworker-threads dependency from package.json. The parallel methods of classifier are now only available if the webworker-threads could be loaded. So as a developer, if you want to call a parallel method like trainParallel, you first have to check if it is available. Is this going to work in other frameworks and in browsers? Please let me know.

Hugo

[Hugo-ter-Doest]

PS Downside of this is that the parallel methods are not tested. If you want to test this, you first have to install webworker-threads manually and then run npm test again.

Hugo

[Hugo-ter-Doest]

I think this is issue is now solved. If not, please comment.

Hugo

[3rd-Eden]

You could still run the tests if you want, e.g:

npm uninstall webworker-threads && npm test && npm install webworker-threads && npm test

So at least you test if works with, and without the module.

[dcsan]

is there a specific release of the npm version that fixes this? I still get the problem on ubuntu, but seem to be able to get it to compile on OSX.

it's failing for me with

>   "version": "0.6.3",

the compile ( `npm i` ) does seem to work now but get errirs at startup

npm WARN [email protected] No description
npm WARN [email protected] No repository field.
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/jest-haste-map/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/sentence-similarity/node_modules/natural/node_modules/webworker-threads):
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] install: `node-gyp rebuild`
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: Exit status 1

added 1221 packages from 1102 contributors and audited 877612 packages in 43.617s
found 6 vulnerabilities (1 low, 4 moderate, 1 high)
  run `npm audit fix` to fix them, or `npm audit` for details
root@bot-wiz2:/mnt/ext250/web-apps/rap/server# npm run plain

> [email protected] plain /mnt/ext250/web-apps/rap/server
> APPNAME=rap PORT=6835 DEBUG=*,-express*,-nodemon*,-superagent,-AmpMetrics,-body-parser* node server.js

Importing data...
Combining data...
Building search index...
Ready!
/mnt/ext250/web-apps/rap/server/node_modules/sentence-similarity/node_modules/natural/lib/natural/classifiers/classifier.js:32
    if (e.code !== 'MODULE_NOT_FOUND') throw e;
                                       ^

Error: Could not locate the bindings file. Tried:
 → /mnt/ext250/web-apps/rap/node_modules/webworker-threads/build/WebWorkerThreads.node
 → /mnt/ext250/web-apps/rap/node_modules/webworker-threads/build/Debug/WebWorkerThreads.node
 → /mnt/ext250/web-apps/rap/node_modules/webworker-threads/build/Release/WebWorkerThreads.node
 → /mnt/ext250/web-apps/rap/node_modules/webworker-threads/out/Debug/WebWorkerThreads.node
 → /mnt/ext250/web-apps/rap/node_modules/webworker-threads/Debug/WebWorkerThreads.node

[dcsan]

i tried

npm uninstall webworker-threads
npm rebuild   # which worked surprisingly

but then running my app

Combining data...
Building search index...
Ready!
/mnt/ext250/web-apps/rap/server/node_modules/sentence-similarity/node_modules/natural/lib/natural/classifiers/classifier.js:32
    if (e.code !== 'MODULE_NOT_FOUND') throw e;
                                       ^

Error: Could not locate the bindings file. Tried:
 → /mnt/ext250/web-apps/rap/node_modules/webworker-threads/build/WebWorkerThreads.node
 → /mnt/ext250/web-apps/rap/node_modules/webworker-threads/build/Debug/WebWorkerThreads.node

any other ideas appreciated, otherwise this library is basically unusable.