Build instructions, console mode, require calls

[euglv]

Hello,
thank you for work and for great example how to embed node.js into delphi application.
Here https://github.com/ivere27/toby we can find instructions how to build c++ application that embeds node.js.
But there are no instructions how to build tobynode.dll for use with Delphi. It may be hard for some one from Delphi world to build something in Visual Studio.
By the way, main page of this repository suggest downloading debug version of tobynode.dll:
https://github.com/ivere27/archive/tree/master/tobynode/node_v6.10.0_x_toby_v0.1.5_x86_debug
This dll works only if Visual Studio is installed. Or you need to collect and copy to application folder some dll-s from the debug version of visual studio redistributable package.
This solution https://github.com/ivere27/toby/tree/master/vc_tobynode has only debug configuration setup. When I tell Visual Studio to build release version of tobynode.dll - it doesn't builds, because release configuration is not set (folders, linker and compiler settings).
I wanted to build release and optimized version of tobynode.dll that doesn't require installing C++ redistributable packages.
After lots of hours finally I managed to build release version of tobynode.dll, but it was only 53 kb and required node.dll and redistributable package.
Also I figured out that node.js 6 and above does not support Windows XP, but my delphi application needs WinXP support.
The good news is that tobynode support node.js 5.11 and node.js 5.11 support Windows XP!
Finally I managed to build tobynode.dll with node.js 5.11 inside, it is almost 9 Mb in size. I tested it on Windows XP SP3 without any redistributable packages, and it works.
I think it will be good to share this dll with others:
tobynode.zip - release version, node.js 5.11
Here is solution with release configuration to build this dll:
vc_tobynode.zip
Folder structure should be the same as described here for building C++ sample.
Unsure if I configured this solution correctly, because I am not familiar with C++, may be it misses some parts, but all node.js features I tested - worked.

I have some additional questions, unsure if I should create separate issue for each or not.

1. TobyPascal.pas statically links tobynode.dll, because of this tobynode.dll is loaded early when program starts. Because of this delphi application has to be configured as console application. If delphi application is not console - node.js core code throws errors. Allocating console before creating TToby class does not help, console should be allocated before tobynode.dll loads. So the question is:
   How to use tobynode.dll not in console application?
   Core node.js code throws errors when there are no stdout or stderr even if console.log(...) is not called from javascript. For example node.js source file node\lib\net.js has this code:
   if (this !== process.stderr)
   This code only checks if current stream is not stderr, nothing was going to be logged, but code is failing not in console application.
   I suggest modifying TToby class to load tobynode.dll dynamically, so we can setup stdout and stderr pipes before loading tobynode.dll.
2. Is there any way to intercept javascript require call? I want to embed some third party node modules to my application, but I do not want them to be stored in node_modules folder in application directory. I want to store third-party modules that are loaded through require call as resources of main executable (or at least inside tobynode.dll). Can I achieve this without modifying node.js sources and rebuilding tobynode.dll when I need to embed some new module?
3. How can I unload and reload tobynode.dll without restarting Delphi application? (For example in case some runtime error in .js code)

Thanks!

[ivere27]

Hi, many thanks for contributing. it's so glad that you are trying to use this project. :)
as I also am in delphi world, I totally agree that it could be hard to build something in VS.
that's why I uploaded the pre-comiled dll 'tobynode.dll' to https://github.com/ivere27/archive
and the 'node-delphi' project is not yet production ready so I haven't tested the release mode.

the issues and the questions..

Supporting Windows XP

great work! please make Pull-Requests!
binary(dll) files on https://github.com/ivere27/archive
any document/source files or wiki pages on https://github.com/ivere27/node-delphi or https://github.com/ivere27/toby
by the way, I suggest to build node.js 4.x version which is Long-Term Support
there is no LTS in 5.x. https://github.com/nodejs/LTS
(currently, ubuntu linux 16.04 LTS 's default node.js version is v4.2.6)

Q1. static link and stdout

in node.js world(as like other *nix world) stdout/stderr are so much important.
as you mentioned, we need a way to handle stdout in non-console mode(AppType Application).
by the way, non-console delphi application is not working? there is a gui example in /node-delphi/example/Project1.dproj

solution 1 - wrapping 'console'
in tobyOnLoad(), execute the below source by tobyJSCompile()
or the first javascript code(ex, app.js)

var console = {
  log : function() {
    // handle here in js

    // test
    require('fs').appendFileSync('stdout.log', arguments[0]);
  }
};

console.log('hi', 'there');

solution 2 - managing stdout pipe in delphi
I think those code(redirecting pipe stuff) should be in tobynode.dll

solution 3 - making tobynode.dll(including pipe handling) as BPL!
then, load 'node.dll' dynamically. it may take times but best for delphi users.

about 'loading tobynode.dll dynamically' you suggested,
it's also a great point. could you make a patch(or example) and PR?
it's good to provide several options to developers.

Q2. require

it's possible to intercept 'require' but so tricky.
'require' is a modue that lazily loaded
in /node/lib/internal/bootstrap_node.js line 349

    const script = `global.__filename = ${JSON.stringify(name)};\n` +
                   'global.exports = exports;\n' +
                   'global.module = module;\n' +
                   'global.__dirname = __dirname;\n' +
                   'global.require = require;\n' +
                   'return require("vm").runInThisContext(' +
                   `${JSON.stringify(body)}, { filename: ` +
                   `${JSON.stringify(name)}, displayErrors: true });\n`;

tobynode.dll 's loading steps are..
'v8' -> tobyOnload() -> bootstrap_node.js -> tobyInit(yourScript)
in tobyOnload context, 'require' variable IS undefined.
see the source, /toby/toby.cpp : _node()
node's core CreateEnvironment() => process.nextTick(bootstrap_node.js) "in QUEUE"
toby's _node() => global->DefineOwnProperty('toby')
then, process.nextTick(userScript) "in QUEUE"

I also desire that node'js core team supports one tarred/zipped node_modules file.
this question is my question too. I've been trying to solve this problem.
anyway, we could have some solutions..

solution 1 - modify node.js(a long long road) to support kinda memory filesystem(?)

1. make one zip file including business codes and node_modules
   (just like excel's .xlss or java's .jar)
2. embed it as a resource file in PE file(in an exe file)
3. when the exe launch, execute scripts from the zip file.

solution 2 - using require.cache
https://nodejs.org/api/globals.html#globals_require_cache
when a module is once required then, it's cached.
so,

1. make a node_module in to one file using 'browserify'.
   http://browserify.org/
2. store it as String in delphi
3. run it and modify require.cache object.

solution 3 - including modules into custom_tobynode.dll l(hack node.js and rebuild)

• node.js's internal javascript files(net.js, os.js etcs.) are stored as byte array in node.exe
  by /node/tools/js2c.py
• we can add our library to /node/node.gyp

Q3. unload/reload tobynode.dll

ooops. we can add 'uncaughtexception handler' in tobyOnLoad()
(or implement a function to force unload tobynode.dll ex, tobyUnload)
https://nodejs.org/api/process.html#process_event_uncaughtexception
actually, https://github.com/electron/electron use that way.
check out /electron/lib/browser/init.js line 46

// Don't quit on fatal error.
process.on('uncaughtException', function (error) {
  // Do nothing if the user has a custom uncaught exception handler.
  var dialog, message, ref, stack
  if (process.listeners('uncaughtException').length > 1) {
    return
  }

  // Show error in GUI.
  dialog = require('electron').dialog
  stack = (ref = error.stack) != null ? ref : error.name + ': ' + error.message
  message = 'Uncaught Exception:\n' + stack
  dialog.showErrorBox('A JavaScript error occurred in the main process', message)
})

I think it's better to separate them into each issues. as small as possible. :)

using node.js in delphi give us a lots of opportunities.
ex, such as REST Client, Web/Socket Servers, Database Clients(especially!) etc
it would be great if you help the project to make more mature.
therefor please do not hesitate to send Pull Request on any changes.

please, let me know if I'm wrong in some point.
best regards,
Yong

[euglv]

Hello, I have made TobyPascal with dynamic linking, it works well on Windows 10, but for some reason is failing on Windows XP even in console application.
When I load tobynode.dll via loadLibrary on Windows XP I get this error after tobyInit call:

#
# Fatal error in C:\Projects\node\deps\v8\src/api.h, line 400
# Check failed: allow_empty_handle || that != 0.
#

Error is raised in thread created by tobynode.dll
The same tobynode.dll with node.js 5.11 but statically linked works well on Win XP SP2.

So can't fully get rid of using console application. I think tobynode.dll should allocate dummy pipes for stdout and stderr if not present.

The js code from the example works ok in non console application. But this code will fail:

var req = http.get('http://example.com', function (res) {
    toby.hostCall('response', res + "");
});
req.end();

Error happens because core node.js code in net.js will check whether current stream is stdout or stderr.

Also interesting, is it possible to build tobynode as a single .obj file? Delphi can include (link) code from C .obj files (but not .lib).
This way we can try to link node.js inside delphi executable without use of dll.

using node.js in delphi give us a lots of opportunities.
ex, such as REST Client, Web/Socket Servers, Database Clients(especially!) etc

I agree with you. Also I like the idea to share Delphi's VCL or Firemonkey to js code.
But for just Web/Socket Servers for greater performance I think it will be better to use libuv directly https://github.com/HeZiHang/Delphi-libuv.
Some Web servers that are built on top of libuv, as node.js is - are faster than node.js.
If you do not need javascript, but only asynchronous io - consider using libuv.

[euglv]

The reason why tobynode.dll doesn't work when loaded dynamically on WinXP could be that thread local storage callbacks are not called when library is loaded dynamically on Windows XP.
TLS (thread local storage) callbacks always run in statically-linked DLLs, and since Vista, they also run in dynamically-linked DLLs.

[ivere27]

sorry for the late reply. recently, I had no time to code.
I've just committed an workaround patch for stdout issue(for http.get). have a look at the diffs.
308486b

TTimer,for reading redirected stdout, need to be refactored as Thread or Async/Event.
(by the way, please feel free to make a pull request. it's always welcome)

[ivere27]

• dynamic linking - applied :)
  the patch is including 'loadLibrary' in order to create the stdout pipe before to load the tobynode.dll

• Windows XP (Thread local storage issue)
  Thanks for the test tobynode in XP. I haven't used Windows XP for a long time.
  is there any progress?

• tobynode as a single .obj
  It's theoretically possible if we compile the toby.cpp(including node.h, v8.h) by C++ Builder or BCC32C.
  look at the last section in https://github.com/ivere27/node-pascal/blob/master/README.md
  there are two object formats in PE format. OMF and COFF.
  https://en.wikipedia.org/wiki/Relocatable_Object_Module_Format
  it is difficult but very valuable for delphi users.

• libuv
  you are right, it's better to use libuv directly for performance.
  if you are interested, there are few projects of mine using libuv already.
  dorypuppy - multi process manager in Android(JNI)
  SimpleProcessSpawn - multi process manager
  SimpleHttpRequest - http client(also using openssl for TLS)

It's better to open new issues for each subjects.