Rfm 17.1 - Sharing Provider Records with Multiaddress

RFMs.md

@@ -8,6 +8,8 @@ This page lists measurements that are useful to understand the dynamics and the
 
 **[RFM 17 | Provider Record Liveness](#rfm-17--provider-record-liveness)**
 
+**[RFM 17.1 | Sharing Provider Records with Multiaddress](#rfm-171--sharing-provider-records-with-multiaddress)**
+
 **[RFM 19 | DHT Routing Table Health](#rfm-19--dht-routing-table-health)**
 
 ## Ongoing 
@@ -615,6 +617,33 @@ Additional context can be found [here](https://pl-strflt.notion.site/Provider-Re
 
 We have numbers to justify how often do provider records expire and have carried out experiments with alternative replication settings to justify new proposed settings.
 
+## RFM 17.1 | Sharing Provider Records with Multiaddress
+
+* _Status:_ **complete**
+* _DRI/Team:_ [`@cortze`](https://github.com/cortze)
+* _Prerequisite(s):_ **NONE**
+* _Value:_ **Medium**
+* _Report:_ [`rfm17.1-sharing-prs-with-multiaddresses.md`](./results/rfm17.1-sharing-prs-with-multiaddresses.md)
+
+#### Proposal
+Achieving fast content retrieval time in IPFS is a key milestone in order to place the platform as a face to face competitor to centralized services. At the moment, in the process of retrieving content from the IPFS network, the user first needs to find the content provider that hosts it. To do so with the `kubo` implementation, the interested client will try to retrieve the content using the Bitswap protocol and ask its immediately connected peers if they have the content of that CID. If this process fails, `kubo` falls back to the public DHT lookup process to find the Provider Records for the CID (the timeout for the Bitswap discovery is currently set to [1s](https://github.com/protocol/network-measurements/blob/master/RFMs.md#rfm-16--effectiveness-of-bitswap-discovery-process)). 
+
+However, If this process of walking the DHT looking for the PR succeeds, the `kubo` client will get the link between the CID and the PeerID that host the content. Thus, the user still has to make a second DHT lookup to find the latest public multiaddress of that specific peer. 
+
+Each public multiaddress for any peer in the network has an allocated Time To Live (TTL) duration, which can vary between `go-ipfs` or `kubo` versions. It was initially set to 10 mins but was incremented to 30 mins in the `[email protected]` update on August 18, 2022. In some occasions, if the user fetches the PRs inside the time window where the multiaddress of the provider hasn't yet expired, the provider's multiaddress will be shared among the PRs so that the client can fetch the content directly from it.
+
+This RFM, which is an extension of the RFM17 for its close relation to the PR retrievability aspect, aims to measure whether the shared PRs for a given CID actually contain the multiaddress of the provider and for how long they are shared. The final intention of the RFM is to discuss whether we can avoid this second DHT lookup by increasing the TTL of the multiaddress to match the [expiration time](https://github.com/libp2p/specs/blob/9464d50f4e08337d0be4dc15a72761b92215747c/kad-dht/README.md#content-provider-advertisement-and-discoveryhttps://github.com/libp2p/specs/tree/master/kad-dht#content-provider-advertisement-and-discovery) of the PRs. 
+
+#### Measurement Plan
+
+- Spin up a node that generates random CIDs and publishes provider records.
+- Periodically attempt to fetch the PR from the DHT, tracking whether they are retrievable and whether the multiaddresses of all content providers are included in the PR.
+
+#### Success Criteria
+
+- The measurements should show that the multiaddresses are shared together with the PRs for 10-30 mins after the publication of the CIDs (depending on the go-version the remote peers use)
+- If so, consider increasing the expiration time of the PeerID-Multiaddress records to match the PR expiration time. 
+
 ## RFM 18 | TTFB through different architecture components
 
 * _Status:_ **ready**

results/rfm17.1-sharing-prs-with-multiaddresses.md

@@ -0,0 +1,127 @@
+
+# RFM 17.1 | Sharing Provider Records with Multiaddresses
+
+_DRI/Team_: [`Mikel Cortes-Goicoechea`](https://github.com/cortze) & [`Leonardo Bautista-Gomez`](https://github.com/leobago) ([`Barcelona Supercomputing Center`](https://bsc.es/))
+_Date_: 11/11/2022
+
+## Table of contents
+
+1. [Introduction](#1-Introduction)
+2. [Findings](#2-Findings)
+3. [Methodology](#3-Methodology)
+4. [Discussion](#4-Discussion)
+5. [Conclusion](#5-Conclusion)
+6. [References](#6-References)
+
+## 1-Introduction
+**_Context_**: [Slack Discussion](https://filecoinproject.slack.com/archives/C03FFEVK30F/p1662995123912119) & [GitHub Issue](https://github.com/ipfs/kubo/issues/9264) 
+
+Currently, content routing through DHT lookups in IPFS involves two different steps:
+
+1. Routing the CID to nodes hosting the content - find the Provider Records (PR) in the public Provider Records DHT
+2. Mapping the ID of the nodes hosting the content to their public multiaddress  - find the multiaddress of a peer in the Public Address DHT
+
+In this context, we understand the PR as the link between a CID and a Peer ID of the content provider, which is how the PRs are stored in the public DHT. However, in the networking layer, when a host requests or looks for the Providers of a given CID, the method [`dht.FindProviders()`](https://github.com/libp2p/go-libp2p-kad-dht/blob/dae5a9a5bd9c7cc8cfb5073c711bc308efad0ada/routing.go#L445) returns two arrays of `peer.AddrInfo`:
+
+- The first array relates to the `[]AddrInfo` of content providers that the remote host is aware of.
+- The second array relates to peers that are closer to the CID from the remote peer's perspective (from its routing table).
+
+Given that the structure of the `peer.AddrInfo` is:
+
+```go
+type AddrInfo struct {
+	ID    ID
+	Addrs []Multiaddr
+}
+```
+
+one could think: Why do we split the retrieval process into two different steps or lookups when we could simply return the `ID` and `[]Multiaddr` of the peer in the same response?
+
+- The simple answer is: it already happens (sometimes).
+
+Each node in the network keeps locally a peerstore with the information of each currently and previously connected peer. In this local peerstore, nodes generally keep items like the `UserAgent`, `Latency`, `Multiaddresses`, and so on from the remote peer. But most importantly, hosts keep locally an expiration time or Time To Live (TTL) for those `Multiaddresses` associated with that peer. This TTL gets updated or extended each time the local hosts interact with that remote peer, which generally happens when the local host refreshes its routing table or when a Provider publishes or republishes the PR.   
+
+In the current IPFS network, the TTL of those `Multiaddresses` is between 10 and 30 mins (depending on the `go-libp2p` [version](https://github.com/libp2p/go-libp2p/commit/c2821791bac7d638890accc98798bf4cbfe122e7)), while the TTL for the PR is ~24 hours (still pending to be re-adjusted to 48 hours after the submission of [RFM17](https://github.com/protocol/network-measurements/blob/master/results/rfm17-provider-record-liveness.md)).
+
+So, when some node requests the PR of a given CID, the node that stores the PR will only reply with the `[]Multiaddr` together with the `PeerID` of the content provider if the TTL for the `Multiaddr` hasn't expired yet. This prevents sharing non-updated records for any peer in the network. Effectively, it avoids sending the client to the wrong multiaddress.
+
+However, since the PRs have a TTL of 24 hours, and we intend to extend it up to 48h, is it worth shortening the current process that performs 2 lookups to a single one? In other words: 
+- Should we extend the TTL of the `ID` - `Multiaddr` mapping? (30-10 mins VS 24 hours matching the PR TTL)
+- Should we always share the `Multiaddr` with the `ID` if any peer asks for a PR that we know?
+
+This extension of RFM17 aims to prove that the mapping between `ProviderID` and `[]Multiaddr` is actually shared for those 10 to 30 mins after the connection with a PR Holder to store the records. Ultimately, and perhaps as an item of future work, proving that extending that TTL to match the PR expiration time, would remove the need to make a second DHT lookup to map the provider's `ID` and its `Multiaddress`. This will bring a significant performance increase to the DHT Lookup process, as it will reduce the latency by half, i.e., clients will need to "walk" the DHT once, instead of twice as is the case today.
+
+## 2-Findings
+- As we were expecting, peers no longer share the provider’s `ID` + `Multiaddress` after 30 mins (some of them still share them after the expected 10 mins depending on the IPFS client that they are using. The TTL increase was included in `[email protected]` August 18, 2022)
+- Even though the lookup process can find the `Multaddress` for the provider over those 10-30 mins, the lookup process sometimes returns an empty `Multaddress` field in the `peer.AddrInfo`. The problem? The `dht.FindProviders()` method only reports once each content provider, so if the first peer that reports the PR only includes the content provider's `PeerID`, later coming `PeerID` + `Multiaddress` mapping will no longer be notified. 
+
+
+## 3-Methodology
+
+The study uses as a basis the CID-Hoarder from RFM17. It includes some complementary modifications that can track the retrieval side of the DHT lookup in more detail.
+
+The tool uses two hosts, one for publishing the content (gets closed after the publication process), and the second one for pinging the PR Holders individually asking for the PRs and for performing a public DHT lookup for the content.
+
+The result of each ping for the PR holders and the result of the lookup for the PRs are written in the `stdout` as logs. The logs contain both the peer reporting the PR and the content of the received `AddrInfo` response. These logs are then parsed in a Python script to produce the plots analyzed in Section [4](#4-Discussion).
+
+**_Notes_**: *Holder: Peers elected to store the PR in the publication process for the CIDs.*
+
+## 4-Discussion
+As previously introduced, this RFM aims to measure how PRs are shared when someone tries to find the content provider for a given CID. To do so, we will divide this section into three different chapters, i) the outcome of a request for a PR to the PR holders (successful, failed), ii) the reply from those peers that share the PR during the lookup process, and iii) the final result of the `dht.FindProviders()` method form the `kubo` implementation. 
+
+The experiment was done for a total of 100 CIDs for over 50 minutes on the 23rd of September, 2022.
+
+### 4.1-PR holder's direct reply
+
+Looking closely at the ratio of PR holders that reply the PR from Figure [1], we see that continuously 16 to 17 PR holders are sharing them over the entire study. Please keep in mind that there is a delay of 3 mins between ping rounds. 
+
+![img](./../implementations/rmf17.1-sharing-prs-with-multiaddresses/plots/pr_from_prholders_pings.png)
+
+_Figure 1: Number of PR holders replying with the PR._
+
+However, if we check the actual content of the `AddrInfo` that we receive back from the remote peers as it is displayed in Figure [2], we can observe that those 16 to 17 stable PR holders only share both `PeerID` + `Multiaddress` for around three ping rounds or 9 to 12 minutes. Afterwards, the median drops to 7 stable peers sharing the combo until ping round 10, i.e., 30 mins, which is then followed by a period of only `PeerID` replies. 
+
+![img](./../implementations/rmf17.1-sharing-prs-with-multiaddresses/plots/id_plus_multiaddres_per_prholders.png)
+
+_Figure 2: Number of PR Holders replying with the `PeerID` + `Multiaddress` combo._
+
+### 4.2-Reply of peers reporting the PR during the DHT lookup
+
+Results are similar when we analyze the replies of the peers that report back the PR from the DHT lookup process. We increased the number of content providers we were looking for to track the multiple remote peers. Figure [3] represents the number of remote peers reporting the PR for the CIDs we were looking for, where we can see a stable 20 peers by median over the entire study. 
+
+For those wondering why more than 20 peers (k replication value when publishing the CIDs) are reporting the PR, we must remind you that Hydra-Boosters share the PR database among different `PeerID` heads. Which means that if one hydra hears about a PR, all the heads of that common database will also share it.  
+
+![img](./../implementations/rmf17.1-sharing-prs-with-multiaddresses/plots/pr_from_lookup_process.png)
+
+_Figure 3: Number of remote peers replying with the PR during the DHT lookup._
+
+We spotted no difference when comparing the number of PRs that contained the `Multiaddress` of the content's provider. Figure [4] shows the exact drop of peers sharing the combo after ~9-12 mins, with a sudden decline after round 10 (~30 mins). It is clear from these results that the network is quite segmented in terms of client versions, where the TTL of the `Multiaddress` records varies from 10 to 30 mins. 
+
+![img](./../implementations/rmf17.1-sharing-prs-with-multiaddresses/plots/id_plus_multiaddres_from_lookup.png)
+
+_Figure 4: Number or remote peers replying with the `PeerID` + `Multiaddress` combo during the DHT lookup process._
+
+
+### 4.3-Result from the DHT lookup
+
+It is essential to notice that despite being able to show empirically what we already knew, the combo of `PeerID` + `Multiaddress` gets shared only over the TTL of the `Multiaddress` records, the result that we got from the DHT lookup process doesn't fully match the previous results. 
+
+Figure [5] displays the final result of the `dht.FindProviders()` method, distinguishing the content of the received PR. In the figure, we can appreciate that despite round one, two and three are inside the first 10 minutes, there are still some DHT lookup that only contain the `PeerID`. 
+
+![img](./../implementations/rmf17.1-sharing-prs-with-multiaddresses/plots/lookup_result.png)
+
+_Figure 5: Result of the `dht.FindProviders` method, together with the filtered content of the received provider's `AddrInfo`._
+
+We would expect to retrieve the combo over the current TTL of the multiaddress records. However, the fragmentation between IPFS nodes in the network, with different TTLs for the multiaddress, makes some inconsistent replies when asking for the PRs. In the current [`libp2p/go-libp2p-kad-dht`](https://github.com/libp2p/go-libp2p-kad-dht) implementation, the return value of the lookup gets defined by the first `AddrInfo` response we get for each provider in the network (code [here](https://github.com/libp2p/go-libp2p-kad-dht/blob/e33a4be6e9a3a8fb603d21126e2d8a42c5e37d1b/routing.go#L490)). This means that if after 15 minutes of publishing a content some client wants to retrieve it through a DHT lookup, if the first peer that we connect from the closest ones replies with only the `PeerID`, that requester client will have to perform a second DHT lookup to map the `PeerID` with the providers `Multiaddress`. This phenomena will still happen even though a second reply might arrive with the entire combo 20ms later. 
+
+
+## 5-Conclusion
+
+With this study, we have demonstrated empirically that the combo of `PeerID` + `Multiaddress` are, in fact, shared for as long as the TTL of the `Multiaddress` records don't expire. This means that if we increase this TTL to match the PR expiration time, we could be reducing to a single DHT lookup the process of retrieving a CID's content using the public DHT. This would be a significant improvement to the overall DHT lookup time, as it would decrease latency by half.
+
+On the other hand, we have identified some code limitations that could affect the impact of such improvements as far as the network keeps fragmented based on client versions and mismatches between configurations. Thus, the final intention of removing that extra second lookup might not achieve the expected result unless the majority of the network accepts and upgrades the TTL of provider's multiaddresses.   
+
+## 6-References
+
+* [RFM17](https://github.com/protocol/network-measurements/blob/master/results/rfm17-provider-record-liveness.md)
+* [CID-Hoarder](https://github.com/cortze/ipfs-cid-hoarder)