Allow to require external node modules in a thread

[thomasfr]

It is not possible at the moment to require external modules or files from within a thread. I know this is not a bug it is a feature. But it would be much much more helpful if one could use external modules. It is totally obvious to me that this would mean to start up a thread with a bigger context which would also mean a increase in the start-up time for those threads.

[dionysusxie]

I agree.

[xxorax]

Too.

[neilk]

Likewise, a much needed feature

[tomcss]

I would really like this as well.

[RedCarrottt]

I agree too.

[jnfeinstein]

@thomasfr can you describe what you had in mind? This library comes closest to implementing what I want with the exception of this feature, and I have the time right now to work on it.

[thomasfr]

Hi, it is some time ago since i opened that issue, but it would be still very very helpful. I think i was missing the option to use external modules like redis, request, any helper modules to compute something in the background like natural or just some logging or debugging modules like debug, bunyan or lodash the like in a seperate thread.

[jnfeinstein]

I noticed that you can't include any core module. I wasn't sure what you meant by starting a thread with a larger context. I'm fairly new to node native bindings.

[jescalan]

+1

[samccone]

+1

[jnfeinstein]

I've worked on this a ton recently. I wanted to make a require function that worked exactly like node's, to support things like nested requires.

What I've deduced is that the require function comes from the native Module class built into node. I was able to implement NativeModule and process.binding functions that work in a thread (these are to get access to the native parts of node, which you presumably will need in other files). I then wrote a startup function that does everything a node process does when it starts up, including creating a root module and adding things like global, process, require, etc.

This works, up to a point. The native modules in node are not coded in a thread-safe manner. For example, Buffer::HasInstance in node_buffer.cc (in the node repo) checks if an object is a buffer by comparing the object's constructor to one previously set during node's initialization. Since isolates by their nature cannot share JavaScript objects but have the same native memory space, this check will always fail for tagg threads. Therefore a tagg thread can never require the buffer class, which is pretty damn important. Also, this solution is janky as all hell and wont take advantage of updates to node.

TL:DR I think that if you need to require, you should use a node's cluster or child process. If you want to offload some cpu busy work, you should use tagg and the load function to grab any helpers you need.

Totally open to input and guidance if anyone has suggestions.

[jescalan]

Wow, great report @jnfeinstein - really sad to see that might not be possible, but good to know! I'm so far from even average at C that my input probably wouldn't be useful, but will be watching from the sidelines!

[thomasfr]

Thanks @jnfeinstein for doing that deep research.

[samccone]

+100 🐶 @jnfeinstein for the awesome response and serious info drop

[Redsandro]

TL:DR I think that if you need to require, you should use a node's cluster or child process. If you want to offload some cpu busy work, you should use tagg and the load function to grab any helpers you need.

By helpers you mean CPU heavy pure javascript snippets? Do I understand correctly that for example, using a module that has a binary part, i.e. node-expat for xml processing, cannot be used in a tagg threadPool for using multiple core processing of huge XML files?

I am not entirely sure about the exact workings, but IIRC the tagg-fork node-webworkers has a function importScripts that has loaded modules that are mentioned in package.json.

author of webworker-threads here. [...]
I've mostly relied on onejs to compile all required modules in package.json into a single JS file for importScripts to use, just like one would do when deploying to a client-side web worker environment.

Source: Stack Overflow: Load nodejs module into a web-worker

Unfortunately, afaik, this only works with javascript-only modules, not with partially binary modules (i.e. node-expat).

[jnfeinstein]

I believe the onejs "hack" would also work in tagg. It's the dependency management component that is lacking (i.e. tagg can only import one file at a time and can't tell if it needs other files). If you have all your required modules in one file, you should be good to go!

Edit: Looked at webworker-threads and it reminded me that Node's built-in functionality (file processing, network, etc) will not work, even if you use onejs. And buffers, per my post above.

[Redsandro]

You can use the fs module though with _native_fs iirc.

[abh1kg]

Is this still being pursued somehow? As @jnfeinstein said, onejs/ node packagers do not work with TAGG.

[jnfeinstein]

@abhikco I think this should be possible in nodeJS > 0.11.3, thanks primarily to this commit. I can dig up my old code if you're still interesting in making this work.

[jescalan]

would be amazing!

[thomasfr]

That would be awesome!

[Redsandro]

How is threads-a-gogo better or worse as compared to fork-pool which I have been using in order to do things in my processes that threads-a-gogo doesn't allow, like requiring modules?

If I understand correctly, threads-a-gogo is easier to use but less versatile, am I wrong?

Also, threads-a-gogo uses threads (shared memory) and fork-pool uses processes (private memory), correct?

I have difficulty evaluating the pro's ans con's of both, and my choice for the processes at this point is purely functional.

[jnfeinstein]

fork-pool looks pretty straightforward. TAGG is much faster at "IPC"
(quotes because TAGG doesn't use processes).. Last time I checked (which
was a few months ago) node processes communicated by serializing data to a
string and passing it through a pipe. Sloooooow. TAGG can use a single
processes's memory space to pass data and messages around. FAST. You have
to be heavily using IPC to see the advantage. And the API is good.

Mostly I like TAGG because JavaScript isn't supposed to be threaded...but
it is!
On Tue, Feb 3, 2015 at 8:07 AM Sander AKA Redsandro <
[email protected]> wrote:

How is threads-a-gogo better or worse as compared to fork-pool
https://www.npmjs.com/package/fork-pool which I have been using in
order to do things in my processes that threads-a-gogo doesn't allow,
like requiring modules?

If I understand correctly, threads-a-gogo is easier to use but less
versatile, am I wrong?

Also, threads-a-gogo uses threads (shared memory) and fork-pool uses
processes (private memory).

I have difficulty evaluating the pro's ans con's of both, and my choice
for the processes at this point is purely functional.

—
Reply to this email directly or view it on GitHub
#22 (comment)
.

[sean-hill]

Has this been implemented yet? I'd like to use some modules I built inside of a child thread, but can't figure out how to do it.

[ghost]

yeah, please, "require" keyword in thread