[libp2p-kad] Prefixed Keys

[romanb]

Context & Motivation

Abstractly, there is a desire for node discovery that is targeted at a specific subset of nodes in the DHT, whereby this subset is associated with some identifier. That is, knowing such an identifier for a subset of nodes, a particular node wants to discover other peers in the DHT within this subset.

Concretely, in #1087, it is mentioned that a validator in Polkadot wants to discover (at least some of) the nodes of a specific parachain it is (temporarily) assigned to, and be able to do so in a targeted manner, i.e. without randomly walking the DHT until it finds some. In that context, the validator knows about an identifier for the parachain and the general idea described in that issue seems to be around ways to tag or group nodes of a parachain together in the DHT based on such an identifier.

To that end, this PR is the result of exploring the option of permitting keys in the Kademlia DHT to be assigned prefixes, corresponding to such an identifier / namespace shared by multiple nodes and thus by multiple DHT keys.

I should say upfront that, as it stands, I am not in favor of actually merging this PR, as I am not convinced that this is a worthwhile addition. Nevertheless I found it educational to explore this path in more detail and hereby document that (admittedly laborious) experiment. There are furthermore some aspects of this PR which may be extracted as individual improvements. See also the conclusion at the end.

Overview

The basic concept is to permit partitioning of DHT keys into two parts: a prefix and a suffix, each obtained by running some input through an (approximate) random oracle (the SHA256 hash function in this case) whose outputs are truncated according to the desired length of the prefix such that the result is a single 256-bit identifier in the DHT keyspace. In this way, the keyspace can remain unchanged, which is one factor for (backward-)compatibility with the libp2p Kademlia protocol (this implementation maintains that compatibility). A 256-bit keyspace is deemed large enough to allow for such partitioning of keys into prefix and suffix.

For example, by choosing a 12 byte (96 bit) prefix, the remaining suffix is a 20 byte (160 bit) identifier space per prefix. Such a prefix length should also have a sufficiently low probability for random collisions (~ a collision after 2^48 prefixes by the birthday paradox with a 96 bit prefix).

The local key of a peer's DHT routing table is thereby tied to a particular prefix. Changing the prefix requires re-initialization of the Kademlia behaviour in order to rebuild the routing table.

Using a prefixed key

A peer wishing to opt-in to a prefix, generates its local key as follows:

let prefix = kbucket::KeyPrefix::new("foo", 12);
let local_key = kbucket::Key::new(local_peer_id).with_prefix(prefix);

This local_key can then be fed into Kademlia::new or Kademlia::with_config.

This implementation is limited to the use of prefixed DHT keys for peer IDs. It does not currently support prefixed DHT keys for record keys.

Looking up a (random) prefixed key

Similarly, looking up the closest peers to a (random) prefixed key looks as follows:

let prefix = kbucket::KeyPrefix::new("foo", 12);
let rand_key = kbucket::Key::new(PeerId::random()).with_prefix(prefix);
kademlia.get_closest_peers(rand_key);

Note that such a query may also return peers with a different prefix, since the lookup occurs on the entire DHT, just giving preference to peers that share the prefix with the random key, as per the nature of the XOR distance.

Implications

There are a number of important implications both to the implementation and the user-facing API:

1. Since a peer ID alone no longer uniquely determines the DHT key of that peer, it is now necessary to both "identify" newly connected peers to obtain their prefix, and to keep track of a mapping from peer IDs to DHT keys (for peers which are either connected, or in the routing table, or both). The previous connected_peers mapping has been repurposed to a peers mapping to that effect.

2. To faciliate distribution of node's DHT key prefixes in order to correctly build the routing tables, the network protocol messages needed to be further extended beyond the libp2p Kademlia spec with a key_prefix field.

3. As mentioned earlier, the local peer ID is tied to a prefix upon initialization of the Kademlia behaviour and changing the prefix requires reinitialization.

4. To my understanding of at least some of the implications resulting from the intentional non-uniform distribution of the keys in this approach, a peer with a particular prefix has detailed knowledge (via the routing table) of other peers sharing that prefix, since it essentially has two clusters of populated k-buckets. On the other hand, all nodes with a different (shared) prefix fall in the same k-bucket. Thus generally a node will at most have k peers sharing a particular prefix in its routing table, for any prefix that is different from the prefix of the local node.

5. The Kademlia::bootstrap operation can become much more expensive for nodes with prefixed keys, due to the appearance of much closer non-empty buckets from which onwards all farther buckets are refreshed after the initial lookup of the local key.

The fact that the simple bijective mapping between peer IDs and (a subset of) the DHT keyspace is lost by the very nature of this approach is the primary source of implementation and API complexity and potential pitfalls. In particular Kademlia::add_address must be used with correctly prefixed kbucket::Key<PeerId>s. To avoid surprises with a naive combination with the Identify protocol resulting in continuously replacing the entry with the correctly prefixed key in the routing table with
a key computed just from the peer ID, add_address prefers to keep the known key if the peer is
already in the routing table. There are other options that are more intrusive on the API, see the inline
comments.

Correct maintenance of the peers mapping from peer IDs to DHT keys is a source of new internal implementation complexity and raises further questions on the API surface wherever currently a PeerId appears, e.g. in many emitted events, as such a peer ID that does not communicate the prefix needed to potentially construct the right kbucket::Key<PeerId> for that PeerId.

Alternatives

The following is a short discussion of two alternatives that come to mind which could also address the initial use-case motivating this PR.

Provider Records

In this approach, a node who wants to advertise that it participates in a namespace announces itself as a publisher of one or more records with known keys. It does so by looking up the k closest nodes to these keys and subsequently registering itself as a provider on these nodes.

When another node wants to "discover" nodes in a particular namespace, it knows about the corresponding record keys (e.g. derived from a chain ID) and performs a DHT lookup for the providers of these keys, which is a regular iterative Kademlia lookup towards the closest nodes to the key(s).

This functionality is covered by the existing implementation of provider records and has the following key advantages over the prefix-based approach:

1. The peer IDs and associated DHT keys are not tied to a particular namespace. A change of namespace does not require reinitialization of a node's routing table.
2. A peer can easily participate in multiple namespaces and add/remove itself at any point by starting or stopping its publication of the respective provider records.
3. The simple bijective map between peer IDs and (a subset of) the DHT keyspace is retained.

It has been suggested that this approach may be subject to an easier form of eclipse attack than the prefix-based approach, whereby a node can be isolated from all (honest) nodes within a particular namespace. However, apart from the fact that standard Kademlia suffers from a general ease of sybil attacks leading to eclipse attacks, I don't think there is actually any noteworthy difference w.r.t.
the vulnerability between these approaches: It is generally sufficient for a DHT lookup (for a random prefixed node key or a fixed record key) to hit upon a single "malicious" node in order to be served with a chosen set of closest peers. Due to the fixed prefix, the "random" node key lookup will largely hit the same buckets in the routing tables of all nodes discovered during the lookup, just like in the case of the lookup for the fixed record key(s).

The provider lookup may even be at an advantage, because the lookup for the "closest peers to the key" is merely a means to an end, which is the collection of provider records along the way: As long as a single honest node is on the lookup path which has at least one provider record of an honest node,
an honest provider may be successfully contacted, since all provider records found during the iterative query are returned, in contrast to a prefixed-key lookup only returning the k closest nodes to the key. It also requires some constant effort for an adversary to keep all provider records for a key stored in the DHT under his control (i.e. referring to malicious provider peers), as new provider nodes whose IDs happen to be closer to the key(s) in question (than any of the current keys under control of the adversary) may appear at any time and all provider records are regularly republished.

In summary, sybil and eclipse attacks are a general concern for Kademlia and require additional measures to raise the bar of protection, some of which are described in S/Kademlia and other papers, and as such I don't see these considerations as important criteria for deciding between e.g. prefix-based routing and the use of provider records.

Multiple DHTs

In this approach, each prefix or namespace gives rise to a separate DHT, isolated from others. A peer may participate in any number of DHTs, which may be as simple as using multiple instantiations of the Kademlia behaviour, each configured with a distinct protocol ID.

Similarly to the provider records, the advantage here is that each DHT remains a "standard", unmodified (libp2p) Kademlia DHT with a uniformly distributed keyspace, and a peer ID is not intrinsically tied to a particular DHT.

The most obvious disadvantage is that it probably does not scale well to many (possibly small) DHTs, but is rather suited to use-cases involving only a few, larger DHTs. An implementation that incorporates support for multiple routing tables and protocol IDs in a single instantiation of a Kademlia behaviour could offer an improvement at the cost of (I think) significant additional implementation complexity.

I have not explored this option further beyond these initial thoughts.

Conclusion

All things considered I am currently not in favour of the approach pursued in this PR, due to the significant added complexity primarily resulting from the removal of the unique mapping between peer IDs and DHT keys, together with the intentional violation of the Kademlia assumption of a uniformly distributed keyspace. It is also unsatisfactory that as it stands, only peer IDs can be prefixed in the DHT keyspace, not record keys but that is likely another significant effort.

My personal preference w.r.t the initial use-case motivating this PR is simply the use of provider records.

There are, however, a few aspects of this PR that should be considered as general improvements and which could be extracted from it:

1. The fact that libp2p-kad does its own "identification" of newly connected peers to obtain routing information, instead of relying heavily on the user feeding in addresses for peers through Kademlia::add_address upon receiving inbound connections, e.g. via the additional use of the Identify protocol. Importantly, this is based on standard libp2p Kademlia protocol requests, requiring no protocol changes if key prefixes need not be transmitted. It does imply an additional request on every established connection to a peer that is not in the routing table, which can add up in iterative queries. However, in situations where the use of the Identify protocol just for integration with Kademlia is otherwise used, these mostly just replace the identify requests.

2. The ability to explicity remove addresses (or entire peers) from the routing table (in this PR needed only internally) would seem to be a useful addition to the public API. Such removal could be triggered by judgments of addresses or peers based on factors outside the scope of Kademlia, thus enabling explicit curation of the routing table (though that needs to be done with care, of course).

[tomaka]

Thanks for the detailed analysis!

The most obvious disadvantage is that it probably does not scale well to many (possibly small) DHTs, but is rather suited to use-cases involving only a few, larger DHTs. An implementation that incorporates support for multiple routing tables and protocol IDs in a single instantiation of a Kademlia behaviour could offer an improvement at the cost of (I think) significant additional implementation complexity.

This can also preferably be done by creating multiple instances of Kademlia (assuming you give a different protocol ID to each of them) and grouping them together.

[romanb]

I assume by now that the idea of prefixed keys is off the table, but are there any comments on the independent suggestions for possible improvements I made at the end of the PR description? In particular, I see some relevance of the ability to explicitly remove entries from buckets (point 2) to use-cases such as paritytech/substrate#3303. By allowing client code to curate buckets in this way, upper-layer protocols can thusly make sure that specific (types of) nodes do not stay in buckets for long. In the particular case of light clients, if I understand correctly, the code using libp2p could react to Kademlia::RoutingUpdated events by identifying whether the remote peer is a light client and if so explicitly remove it from the routing table.

The objection may be raised that it is preferable to not add such clients to the DHT in the first place, but given how the k-bucket node retention policy works with the preference for nodes with sustained uptime and availability, I think that after an initiial bootstrapping phase the buckets will be relatively stable anyway and retain mostly the "full nodes", with relatively little churn (from light clients or otherwise), and the simplicity of such an approach seems attractive compared to the outlook of more intricate hooks into libp2p-kad.

[mxinden]

given how the k-bucket node retention policy works with the preference for nodes with sustained uptime and availability, I think that after an initiial bootstrapping phase the buckets will be relatively stable anyway and retain mostly the "full nodes"

Full nodes: So this would eventually have long running nodes be connected to other long running nodes without us having to include the upper layer in the decision process. Not needing to cross the abstraction boundaries sounds attractive to me, at least until we can see the problem (full nodes trying to connect to light nodes) on our test clusters.

Light clients: The above would not solve the problem for light clients, correct? I guess depending on the fact that k-buckets of other nodes are likely filled with full nodes when discovering the nodes on the DHT as a light client is not enough, right?

[romanb]

Light clients: The above would not solve the problem for light clients, correct? I guess depending on the fact that k-buckets of other nodes are likely filled with full nodes when discovering the nodes on the DHT as a light client is not enough, right?

What is "the problem for light clients"? If all nodes (including light clients) remove nodes identified as light clients (or any other undesirable node type) from their routing table in a timely manner after discovery, then all of these nodes have a routing table that contains mostly those nodes that are supposed to be there. Can you elaborate on the problem(s) you're seeing for a node that is a light client?

[infinity0]

Just as a note I am working on the general topics of access policies for various network subcomponents and this will be part of that. Give me a week or so, I'll get back with some high-level ideas.

[mxinden]

This is a follow up to #1229 (comment) @romanb. I am not sure my initial thought is of much help, so feel free to ignore this comment.

In #1229 (comment) I was exploring the option of not involving a higher level in the decision who-to-include-in-ones-routing-table, thus leaving the DHT clueless in regards to the existing roles (light client, collator, validator, observer, ...).

Given that full nodes run for a long sustained amount of time and Kademlia's preference for nodes with sustained uptime and availability, one would expect a full node's routing table to eventually contain mostly other full nodes, without the DHT logic being aware of the different roles.

Light clients on the other hand don't have this eventual filtering, given that they are only online for fragmented short amount of times. One could argue though that once they connect to a full node, this full node will likely return other full nodes under the property mentioned in the previous paragraph.

What is "the problem for light clients"?

I don't know whether light clients will face the issue of only discovering other light clients in the first place. Thus I would suggest delaying the decision of introducing the additional logic of upper layers being able to influence the routing table until we see the issue in a test setup.

[romanb]

In #1229 (comment) I was exploring the option of not involving a higher level in the decision who-to-include-in-ones-routing-table, thus leaving the DHT clueless in regards to the existing roles (light client, collator, validator, observer, ...).

I'm not sure I see what you mean. If by "the DHT" you mean the libp2p-kad implementation, then it will always have to be clueless w.r.t. specific roles since it sits one layer below. The only question is what level of customizability is required in libp2p-kad in order to allow upper layers to make such distinctions based on roles known only to these upper layers. And in that context I have simply been suggesting that it may be preferable to keep that customizability to a minimum, concretely by just allowing explicit removal from buckets. But I guess we're on the same page, please let me know if I misunderstood you!

[..] One could argue though that once they connect to a full node, this full node will likely return other full nodes under the property mentioned in the previous paragraph.

Yes, that is my hypothesis, since a client's bootnodes are assumed to be some well-known full nodes.