Open Problem: Routing at Scale (1M, 10M, 100M, 1B.. nodes)

[daviddias]

No description provided.

[daviddias]

Do not forget to link to:

• DHT 2.0 ipfs/notes#291
• Unreliable crypto channels ipfs/notes#198
• Thoughts on the next level of content routing for ipfs ipfs/notes#162
• service discovery on ipfs ipfs/notes#15
• Closed DHT Based Routing notes#10
• Local Peer Exchange Protocol notes#3
• https://github.com/libp2p/research-dht/issues/6

[daviddias]

This is already a potential solution. A requirement should not constrain the solutions to a box, rather define what we believe it is needed to achieve the scale we want (for example: handle churn of multiple millions of nodes)

[daviddias]

Ah!! @yiannisbot just realized now that I was missing to click the box "submit review". Now I understand why you were asking for my review last week and I gave you my surprised answer saying "I already gave you the review". My apologies

[yiannisbot]

:)) No problem and thanks for the comments. Fair points! Will update asap and release next version.

[yiannisbot]

I have incorporated all comments. This reads very nicely now I believe, it covers all essential points. @daviddias: could you do another pass?

[yiannisbot]

@aschmahmann: I've assigned a review for this open problem to you - if you could have a quick look to see if we're missing something essential, that'd be fantastic!

[daviddias]

Awesome work @yiannisbot!! :D

[aschmahmann]

@yiannisbot looks pretty good to me!

A few brief thoughts/points (mostly about Kademlia):

1. We have no idea what Kademlia's current scalability is because of the massive number of undialable nodes on the network. We will hopefully be able to see with ongoing testing (where we can remove undialable nodes), and with some minor DHT fixes to remove undialable nodes, what the numbers look like.
2. Everything is a numbers game. There are ways to tweak even algorithms like Kademlia to reduce latency (and still keep bandwidth usage low) by just increasing routing table size. Without doing some simulations on what we already have and comparing with what we need there's no reasonable way to approach a "solution".
3. While Kadmelia does in some ways scale/adapt better with many nodes, latency is Log(N) so that's not really the case.
4. There's a related problem of trying to figure out how to balance making all content available with making only the roots of content available. If all my data is zip files then there's no reason for the routing network to really help route more than one object/zip file. However, if my data is a directory with 1000 files in it, then the routing network should help move around all 1000 files + the directory.