nwaku c-bindings (NodeJS + Python)

[fryorcraken]

Planned start date: started ~May'23
Due date:

Summary

nwaku is the reference implementation of the Waku protocol. To enable developers to use the reference implementation in their native application, nwaku needs to expose c-bindings.
As Nim is a garbage collected language, integration in other garbage collected languages such as NodeJS or Python may not be straightforward, as one needs to ensure that the garbage collector of either languages to not conflict with each other

Acceptance Criteria

• 1. Expose c-binding API as defined in https://rfc.vac.dev/spec/36/
• 2. Provide a NodeJS package and example using said c-binding API
• 3. Provide a Python package and example using said c-binding API

Tasks

• Important to implement chore: apply thread-safe enhancements in C-bindings #1878 in order to accomplish point 1. from the acceptance criteria.
• [ ] Adapt https://rfc.vac.dev/spec/36/ as per changes in proc signatures.
• [ ] Extend the waku implementation protocols. So far only the relay protocol is supported. Before that, we need to have a solid relay implementation
• Create a separate NodeJs repo within the waku org. This repo should have nwaku as a submodule.
• Create a separate Python repo within the waku org. This repo should have nwaku as a submodule.

Future Tasks

• Extend the library to other platforms ( see Language Bindings #904 )
• Adapt https://rfc.vac.dev/spec/36/ as per changes in proc signatures.
• Extend the waku implementation protocols. So far only the relay protocol is supported. Before that, we need to have a solid relay implementation

RAID (Risks, Assumptions, Issues and Dependencies)

• It is unclear whether this possible with Nim as preliminary research did not find any existing example with NodeJS.
  • A workable approach has been found.
• This would be the second implementation of WAKU2-BINDING RFC, hence it may lead to changes in the RFC.

Original issue

Problem

NodeJS is used by projects such as RAILGUN to build a backend/node.
Currently, developers of such projects have the following choices:

• integrate js-waku: not ideal as js-waku targets the browser environment and hence is lacking service node feature such as:
  • server side implementation/performance optimization of protocols (e.g. store, filter, light push)
  • RLN (currently only exporting zerokit to wasm for the browser, which does not work in NodeJS due to difference in APIs)
• Use nwaku or go-waku as a service node
  • via JSON RPC API (which is limited)
  • via HTTP REST API (not yet delivered)
• Use go-waku c-bindings (which requires work to do as we do not provide a JS wrapper at this stage

None of the choices above are ideal/straightforward.

Acceptance Criteria

A clear strategy to integrate nwaku in NodeJS. Two steps will be required:

1. Select the most efficient way for NodeJS and Waku to communicate (REST API? Unix socket? C-binding? etc)
2. Deliver an off-the-shelf solution/wrapper so that a user can install a npm package and communicate with their nwaku node
3. Deliver an off-the-shelf solution/wrapper so that a user can install a npm package and deploy/run a nwaku node

(3) can be done at a later stage/tracked with a new issue once (1) and (2) are done.

[fryorcraken]

Regarding this. Is c-binding the best solution? Should we just start with providing a go-waku npm packages for NodeJS, similar to the React Native one?

Cc @richard-ramos

[Ivansete-status]

Weekly Update

• achieved:
  • Moved the Waku execution into a secondary working thread. Essential for NodeJs.
  • Adapted the NodeJs example to use the libwaku with the working-thread approach. The example had been receiving relay messages during a weekend. The memory was stable without crashing.
• next: start applying the thread-safety recommendations chore: apply thread-safe enhancements in C-bindings #1878

[Ivansete-status]

Weekly Update

• achieved: Started applying thread-safe recommendations, making the Waku Node instance to be created within the Waku Thread itself.
• next: Carry on with the thread-safety recommendations: avoid using Channel to communicate main thread and the Waku Thread.

[Ivansete-status]

Weekly Update

• achieved: Merged PR that made the Waku Node to be created within the Waku Thread. Submitted a PR that aims to make a safer the communication between the main thread and the Waku Thread.
• next: Merge the PR to enhance communication between threads and start extracting the thread context outside the library (comment: refactor(cbindings): libwaku - run waku node in a secondary working thread  #1865 (comment)).

[Ivansete-status]

Weekly Update

• achieved: improved the thread safeness communication.
  refactor(cbindings): Thread-safe communication between the main thread and the Waku Thread #1978
• next: Once the above PR is merged, avoid the use of global variables, to enhance the thread-safeness ( see refactor(cbindings): libwaku - run waku node in a secondary working thread  #1865 (comment) )

[Ivansete-status]

Weekly Update

• achieved: Use of 'ThreadSignalPtr' instead of loop to handle req/resp.
  refactor(cbindings): waku_thread.nim - Using 'ThreadSignalPtr' instead of loop to handle req/resp #2045
• next: Avoid the use of global variables, to enhance the thread-safeness ( see refactor(cbindings): libwaku - run waku node in a secondary working thread  #1865 (comment) )

[Ivansete-status]

Weekly Update

• achieved:

  • Added a simple cpp example to the main code. chore(cbindings): Adding cpp example that integrates the 'libwaku' #2079.
  • Submitted a PR where we start showing the doability of a Rust integration with the libwaku.
    This PR is currently introducing the thread-safety enhancement of avoiding using global variables. Ideally, this should be in a separate PR. chore(cbindings): cbindings rust simple libwaku integration example #2089.
    Notice that it was important to invest time in the Rust example so that we can carry on with the "callback" technique to exchange information between the host code (any) and the foreign code (Nim.)

• next: Separate the PR mentioned above and submit another one which only avoids using global variables but doesn't add the wip-Rust integration.

[Ivansete-status]

Weekly Update

• achieved:

  • Separate PR to avoid global variables: chore(cbindings): avoid using global var in libwaku.nim #2118
  • Started to document the tasks tackled so far: https://www.notion.so/NWaku-cbindings-FFI-7a9ae6240cfc4caba7c7ff0bf3429a70

• next: Start creating a separate NodeJs and Python repositories, where we will create nodejs-waku and py-waku, respectively.

[Ivansete-status]

@fryorcraken, @chair28980 - This can be considered as completed because:

The NodeJs repo has been created, and from there nodejs-waku packages can be built: https://github.com/waku-org/waku-nodejs-bindings
The Python repo has been created, and from there py-waku packages can be built: https://github.com/waku-org/waku-python-bindings

Besides, I've moved the following tasks to the upcoming epics:

• Extend the library to other platforms ( see Language Bindings #904 )
• Adapt https://rfc.vac.dev/spec/36/ as per changes in proc signatures.

[Ivansete-status]

@fryorcraken - I'm closing this because we have NodeJs and Python repos available as stated in my previous comment.

@chair28980:

I've added the following points to the waku-org/pm#141 issue, and I removed the point that referred to this issue that I'm closing (nwaku/issues/1332)

• Extend the library to other platforms ( see Language Bindings #904 )
• Adapt https://rfc.vac.dev/spec/36/ as per changes in proc signatures.
• Support filter from libwaku
• Support lightpush from libwaku