WebRTC for Waku

[weboko]

This is a feature request

Problem

We have some scenarios that can be improved by Waku having WebRTC capabilities.
Discussions around WebRTC usually are concentrated around two points:

• browser to browser connection;
• browser to server connection;

These problems have different solutions and they give different benefits.

Browser to server:

• main benefit is to avoid TLS encryption and necessity of certificates;
• ability not to use web sockets all the time;

Browser to browser:

• ability to offload Waku network;
• possibility to implement completely new features (collaborative editing, etc);

Note: there is another server-to-browser approach which I didn't cover here and have limited knowledge of.

Browser to server

As far as I know there were some efforts to have it implemented in nwaku and go-waku:

• Add WebRTC Direct Transport nwaku#872
• Implement webrtc-direct go-waku#203

But there is no implementation for it yet.
In order to move forward with it we need to determine priority of this and action accordingly.
There are no obvious blockers that I can identify now.

Browser to browser

We have an example that shows how WebRTC connection can be established between two clients by using Waku Noise.

Work on it revealed some road blockers:

• for WebRTC channel to be created client should be able to retrieve its SDP coordinates which forces client to rely on some other server to get it. In the example I used dummy Google STUN server (here);
• in case clients are not able to be connected directly usage of TURN server is unavoidable. Clients won't be able to connect directly in case both of them are behind NAT and hole-punching failed, see;

These points are identified as blocker because usage of some STUN/TURN servers kills the purpose of Waku usage and creates censorship risks if it is done not in an intentional way.

What we can do to overcome it:

• implement some of the STUN capabilities in waku nodes;
• as mentioned in the spec we can operate a STUN server/servers for a particular network or an application such as Status;
• give an opportunity to configure used servers as part of public API;

My opinion is that it makes sense to stick to similar approach as libp2p and let consumers set and possibly operate their servers. Later on when libp2p releases their browser-to-browser API we can just move to using it.
This approach can let such consumers as Status use WebRTC from Waku fairly easily and soon.

As a long term goal it might make sense to extend waku network with STUN capabilities. This will require considerate amount of work and planning. I believe we can do it after understanding how usable it is to developers.

References

Previous research - #20
libp2p browser-to-browser - libp2p/specs#497
libp2p browser-to-server - https://github.com/libp2p/specs/blob/master/webrtc/README.md

Other links

Waku WebRTC PoC - https://github.com/waku-org/js-waku-examples/tree/master/examples/noise-rtc
libp2p browser-to-browser PR - libp2p/js-libp2p-webrtc#90
WebRTC security overview - https://webrtc-security.github.io/
nwaku issue on WebRTC - waku-org/nwaku#872
go-waku issue on WebRTC - waku-org/go-waku#203
libp2p direct connection approach - https://github.com/libp2p/specs/blob/master/relay/DCUtR.md

UPD: thanks to @Menduist were able to clarify some points.

[fryorcraken]

To keep a small incremental scope for domain, I suggest we focus on browser-to-browser connections to enable further relay connections. Why?

• This can keep the experiment/PoC contained to js-waku to progress more freely (no deps on webrtc direct in go-waku or nwaku)
• The output would be usable by a current platform: RAILGUN uses Relay in the browser (whereas an application level usage would mean building more PoC/example for it).

---

As there is currently no decentralized solution, I think it is fine to further investigate the usage of STUN and TURN servers so we can progress while keeping an eye on the decentralized option described above.

I think there are 2 next steps to be done in parallel-ish because they feed from each other:

1. An RFC
2. A PoC

RFC

Let's write an RFC to clearly state the privacy (lack of) guarantees of using WebRTC transport for Waku with a centralized STUN and TURN server.

In said RFC, we can clarify what metadata is leaked and possible linkabilty between two parties using the same STUN server to connect to each other.
Same for TURN.

Noting that there are a number of scenarios that are of interest:

1. Using WebRTC for the purposes of increase Waku Relay connections: in this case, Alice and Bob both use Waku but may or may not use the same application over Waku. Hence, linkability of Alice and Bob may not matter as such.
2. Using WebRTC to have a higher bandwidth/lower latency for specific application purposes such as video calls, collaborating editing, etc: In this case, Alice and Bob are interacting from a user POV and the linkability would matter and be a potential privacy breach.

This should be done with consideration for both STUN and TURN servers.

From this, we can then draw some guidelines. E.g.: prefer to use Google STUN server as no linkability is possible except for timing attacks but as the Google server is used by many application then the chance of it happening are lower.
E.g. Platform should use their own log-less TURN server as using a 3rd party TURN server link two users collaborating.

PoC

In terms of PoC, the next step would be to get several browser in the same Relay mesh by connecting to each other via WebRTC (using centralized servers).

I can see two current blockers for this:

1. discovery: we need the sibling of peer exchange protocl to enable short-lived WebRTC nodes to advertise themselves.
   • We could go around this as a first step by displaying the ENR and enable entering a peer ENR to initiate a WebRTC connection (in the web page)
2. handshake confirmation: noise currently has a step where both parties need to confirm they see the same authcode. This is undesirable when we want the Waku node to connect to other nodes via WebRTC in the background to form a healthy gossipsub mesh
   • Can we somehow skip this step (authcode)?
   • Is noise even appropriate to exchange SDP data? Or should we be using noise but not noise device pairing?
   • Note that I assume a node would advertise their WebRTC address using ENR, which includes a secp256k1 key.
   • Can we use this information to do a noise handshake without authcode (it would be a XK1 handshake where Alice got Bob's static key from the ENR)?
   • Or should we simply use ECIES?

[fryorcraken]

@s1fr0 Can you check my comments on noise and share your thoughts on the matter?

[fryorcraken]

Cc @kaiserd re peer exchange/advertisement of short-lived WebRTC nodes

[Menduist]

Note that most of this issues have been solved in the libp2p webrtc (wip) implementation

The only thing that is specific to waku is that if hole-punching fails, you apparently want to use a relay server to enable "full relayed connections" between the peers (whereas libp2p will just fail)
That seems very hard to do in a decentralized manner, may be possible with something like a mixnet / tor? But then you loose the "high bandwidth / low latency"

[fryorcraken]

Note that most of this issues have been solved in the libp2p webrtc (wip) implementation

Great to hear

The only thing that is specific to waku is that if hole-punching fails, you apparently want to use a relay server to enable "full relayed connections" between the peers (whereas libp2p will just fail)
That seems very hard to do in a decentralized manner, may be possible with something like a mixnet / tor? But then you loose the "high bandwidth / low latency"

I wonder if it would be possible to agree on a nwaku node to use as a relay server.
I imagine there would be a number of security/privacy compromises but it could be a balanced solution.

[Menduist]

You could use libp2p's circuit relay, but the real question is who is going to allocate bandwidth for private peers to use

For a simple 1-1 video call with 0.5 mbit/s per stream (360p 30 fps) stream, the relay has to

• Receive stream from A (0.5 mbit/s incoming) +
• Send A's stream to B (0.5 mbit/s outgoing) +
• Receive stream from B (0.5 mbit/s incoming) +
• Send B's stream to A (0.5 mbit/s outgoing)
  = 2 mbit/s total
  (And 0.5mbit/s per stream is the bare minimum, and will look bad)

Who is ready to waste 2mbit/s of bandwidth for two random people to call each other? That's almost 1GB/hour for a single 1-1, that would never scale. From a random internet source, 86% of webrtc connections succeeds without a relay, that leaves 14% of connections that needs to be relayed.
If you have 10000 users on your 1-1 video call app, that's 1400 users that need relaying, or 1400 mbit/s of bandwidth required. Let's say 5% of the network are nwaku node (which seems high for a decentralized app), each of them would need to relay 28 mbit/s just for this (or 12.6gb per hour, which would cost 843$ / month just in bandwidth in EC2*.).

I really don't see an easy way to get a "fat pipe" between peers that cannot connect directly to each other, unfortunately. FYI, when protocol labs started the circuit relay protocol in IPFS, they hoped that other people would also run relay nodes in the network. But they ended up being the only one willing to bear the cost, so they instead tuned the relay protocol to be cheap enough that everyone in the network runs it (and the "light version" means "only using it to try to hole-punch, not to actually relay data")

* I use aws to get an idea of real bandwidth cost, not because I think people should host things on AWS

[fryorcraken]

@weboko to split the use WebRTC for Waku Relay milestone in smaller chunks of works.

[fryorcraken]

API improvement of XK1 pattern tracked with waku-org/js-noise#3 (master issue for noise handshake API improvements).

[weboko]

The goal: enable WebRTC as transport for Relay.

To achieve it we need to have following parts in place:

1. peer discovery;
2. secure credentials exchange;
3. on success: libp2p transport;

Roughly we want 1 and 2 to work like (success path):

• Alice declares she is able to talk thought WebRTC by publishing her peerId over dedicated contentTopic like `/relay-webrtc;
• Bob by seeing Alice's ability published invite to initiate WebRTC to contentTopic /relay-webrtc/AlicePeerId (Alice is subscribed to it already);
• Alice responds to it (in /relay-webrtc/AlicePeerId) and initiates XK1 handshake;
• by using secure channel they exchange SDP coordinates to establish WebRTC connection
  • alternatively Alice might as well just publish her SDP unencrypted in /relay-webrtc/AlicePeerId and wait for others to be connected
  • or exchange it in another /relay-webrtc/AlicePeerId/BobPeerId;
  • I believe they good enough for experiments and nothing else;

note: because browser tab is short lived there are many edge cases to consider such as intention of Alice can be outdated, cap number of rtc connections, spam during etc.

Once connection is established we want to add it to the mesh of GossipSub.

---

Experiment#1

At this point libp2p team has implemented webRTC transport which I use in an experiment.
It does not work in a way we want because discovery is absent and exchange happens over /p2p-circuit protocol.

Next attempts

In following experiments will try to showcase broadcasting approach to establishing connection in a way approximate to what we want to see.

---

Following work for js-waku:

• discover peers through broadcasting;
• extract and use XK1 handshake from js-noise for coordinates exchange;
• create js-libp2p native transport;

Following work for examples:

• make an example using @libp2p/webrtc where 3 tabs connect to each other and only one keeps connection to nwaku node;
• make an example with broadcasting way of establishing connection;

[fryorcraken]

Before doing discovery via broadcasting, maybe a QR code (phone to laptop)

[fryorcraken]

Blocked on #1315

[weboko]

Unblocked now, working on integrating transport again and iterating towards providing example based on QR code / broadcasting

[DougAnderson444]

You could use libp2p's circuit relay, but the real question is who is going to allocate bandwidth for private peers to use

Update: I believe with libp2p circuit-relay-v2 it is very cheap to connect peers to each other with little cost to the relay

See https://github.com/libp2p/specs/blob/master/relay/circuit-v2.md#rationale

[weboko]

You could use libp2p's circuit relay, but the real question is who is going to allocate bandwidth for private peers to use

Update: I believe with libp2p circuit-relay-v2 it is very cheap to connect peers to each other with little cost to the relay

See https://github.com/libp2p/specs/blob/master/relay/circuit-v2.md#rationale

Totally agree! So this is actually used for a first rough example it is going to be the implementation. For the next iterations similar approach will be used as in this noise-rtc example.

[DougAnderson444]

Also if Waku is focused on scalability with this issue, then eventually you'll likely consider Gossipsub over webrtc if you haven't already (I'm not that familiar with the Waku architecture).

I mention this because it doesn't work so well browser-to-browser right now in JS. I've already opened an issue on the javascript side for this here ChainSafe/js-libp2p-gossipsub#448 but another option will soon also be to use wasm once libp2p/rust-libp2p#4248 is merged. 4248 is browser-to-server now but will eventually be upgraded for browser-to-browser

At that point I envision a mesh of browser webrtc gossipsub nodes, which would be fantastic!

[danisharora099]

These points are identified as blocker because usage of some STUN/TURN servers kills the purpose of Waku usage and creates censorship risks if it is done not in an intentional way.

What we can do to overcome it:

implement some of the STUN capabilities in waku nodes;
as mentioned in the spec we can operate a STUN server/servers for a particular network or an application such as Status;
give an opportunity to configure used servers as part of public API;
My opinion is that it makes sense to stick to similar approach as libp2p and let consumers set and possibly operate their servers. Later on when libp2p releases their browser-to-browser API we can just move to using it.
This approach can let such consumers as Status use WebRTC from Waku fairly easily and soon.

The js-libp2p team has done some work on decentralised hole punching (tracked with libp2p/js-libp2p#1461) which might be helpful here

[weboko]

Iceboxing as this track is on hold while we work on other milestones.