Define libp2p protocol for light client sync #2802

etan-status · 2022-01-13T11:38:40Z

While the current Altair specs define structures to aid light client
development, one missing key aspect is the network protocol definition.

Certain implementations have started defining their own REST based APIs,
e.g., Lodestar at https://github.com/ChainSafe/lodestar/blob/master/packages/api/src/routes/lightclient.ts
While such APIs are useful, REST does not seem to be the ideomatic
choice as the sole API available at such a low level for Ethereum.

This patch introduces a libp2p based protocol to allow light clients to
sync to the latest BeaconBlockHeader in a trustless and decentralized
manner, building on top of prior work from:

@hwwhww at [WIP] Light client protocol + Req/Resp interaction #2267
@jinfwhuang at (WIP) Light client p2p interface #2786
Lodestar's REST API (also has an endpoint to fetch merkle proofs!)

specs/altair/sync-protocol.md

jinfwhuang · 2022-01-18T04:02:37Z

I have a couple of high level comments/questions:
First, should we define the networking layer on libp2p? I have mentioned a few times elsewhere in the discussion, e.g. github PRs and light-client channel in Discord. We basically have three options on networking:

rpc
libp2p
portal network

The main drawback with the libp2p approach is what "LES" has experienced so far. There are very few clients that will serve the light client protocols. Participation of light client protocols are not required for the full node clients.

The portal network experiment will likely gather more momentum in the months ahead. If this network indeed becomes what it is intended to become, then that might be the long term winner.

In the short term, rpc is the easiest to implement and get bootstrapped.

Second, we should write the sync-protocol in such a way that the client could choose any of the three networking options, even if they are not yet defined. It could be possible that the light client ecosystem will accommodate all three networking options. In fact, that could be an likely outcome in the intermediate term when light clients users experiment and try things out.

specs/altair/sync-protocol.md

etan-status · 2022-01-18T11:17:19Z

The main drawback with the libp2p approach is what "LES" has experienced so far. There are very few clients that will serve the light client protocols. Participation of light client protocols are not required for the full node clients.

Lodestar's prototype REST API proved to be very lightweight to serve in both Lodestar and Nimbus prototype versions. This libp2p proposal is similarly scoped in the kind and amount of data gathered and served. Having the major implementations serve those protocols by default should be achievable.

In the short term, rpc is the easiest to implement and get bootstrapped.

Yes, but rpc requires a known node, and also an externally reachable rpc endpoint. The light client should be able to sync with the network autonomously.

Second, we should write the sync-protocol in such a way that the client could choose any of the three networking options, even if they are not yet defined.

Yes, that is still possible. The libp2p proposal here defines portable functions that are not transport dependent. It is possible to also serve the same requests over a different transport. For example, the GossipSub topic can be mapped to the REST events API, and the other endpoints could just become standard requests. The processing overhead for the servers should be comparable, i.e., not noticeable. The functions defined in this proposal could be reused.

etan-status · 2022-01-18T11:45:18Z

Thanks for the thorough review @jinfwhuang ! Really appreciate it.

specs/altair/sync-protocol.md

etan-status · 2022-01-20T19:14:30Z

I have updated this to better support the sync committee period boundary. It should now be possible to have the optimistic_header sync past the current sync committee period, and then obtain a full LightClientHeader at any point throughout the next sync committee period to advance finalized_header (and the accompanying next_sync_committee).

OptimisticLightClientUpdate are no longer considered for best_valid_header because they lack next_sync_committee. Technically, they could be used to advance to a later slot within the current sync committee period, but that does not help the overall sync progress, and it would only apply via UPDATE_TIMEOUT within the next period when the next_sync_committee information must already be available to reliably continue syncing.

Note that I have temporarily merged additional PRs into this one to avoid countinuously fighting merge conflicts while I'm working on this PR. If you are interested in reviewing, please focus only on the latest commit of this PR.
The other commits are from PR #2805 and PR #2810 – please use those other PRs to comment on these.

etan-status · 2022-01-21T12:56:28Z

Added another endpoint to fetch the latest LightClientUpdate (in additional the existing one for the latest OptimisticLightClientUpdate and the best LightClientUpdate.

This simplifies the sync process, and also enables use cases such as tools that solely intend to update from a given checkpoint block root to the latest finalized checkpoint, optimizing sync times.

etan-status · 2022-01-21T17:09:39Z

Extended the test suite to cover all flavors of LightClientUpdate and OptimisticLightClientUpdate.

etan-status · 2022-01-24T13:22:34Z

Added checks to reject signatures from next_sync_committee until current_slot has advanced sufficiently, and restructured the validate_light_client_update and validate_optimistic_light_client_update functions to look more similar.

This implements the latest version of the light client sync protocol, and also adds a test runner for the new accompanying tests. See ethereum/consensus-specs#2802

etan-status · 2022-01-25T09:02:40Z

Updated GetLatestLightClientUpdate to no longer produce LightClientUpdates with finalized_header.slot more than UPDATE_TIMEOUT in the past. This forces next_sync_committee to be rooted in attested_header, allowing light clients to advance into the next sync committee period.

For BestLightClientUpdatesByRange the interval was already smaller because there, only updates are considered that have all of finalized_header, attested_header and the signature_slot in the current sync committee period. However, this approach is not feasible for GetLatestLightClientUpdate as it would introduce windows around the sync committee period change where the finalized_header cannot be advanced; hence, that endpoint can have finalized_header from previous sync committee period as well.

Also updated the light client sync process to ask multiple peers for LightClientUpdate to try and get one with a 2/3 or higher sync committee participation. Historic syncs would always trigger the "forced-update" condition even if only signed by an individual sync committee member, as current_slot is far ahead.

This extends the `--serve-light-client-data` launch option to serve locally collected light client data via libp2p. See ethereum/consensus-specs#2802

This extends the `--serve-light-client-data` launch option to serve locally collected light client data via libp2p. See ethereum/consensus-specs#2802 Backfill of historic best `LightClientUpdate` data is not implemented. To test, in `conf.nim` change `serveLightClientData`'s `defaultValue` to `true`, then run this command: ``` scripts/launch_local_testnet.sh --kill-old-processes --preset minimal \ --nodes 4 --disable-htop --stop-at-epoch 7 ``` The log files of the beacon nodes will be in the `local_testnet_data` directory. They are named `log0.txt` through `log3.txt`. The logs can be browsed for light client related messages.

etan-status · 2022-01-27T22:51:33Z

Documented error handling for the various endpoints (3: ResourceUnavailable).

This implements the latest version of the light client sync protocol, and also adds a test runner for the new accompanying tests. See ethereum/consensus-specs#2802

This extends the `--serve-light-client-data` launch option to serve locally collected light client data via libp2p. See ethereum/consensus-specs#2802 Backfill of historic best `LightClientUpdate` data and the client side of the protocol are not yet implemented. To test, in `conf.nim` change `serveLightClientData`'s `defaultValue` to `true`, then run this command: ``` scripts/launch_local_testnet.sh --kill-old-processes --preset minimal \ --nodes 4 --disable-htop --stop-at-epoch 7 ``` The log files of the beacon nodes will be in the `local_testnet_data` directory. They are named `log0.txt` through `log3.txt`. The logs can be browsed for light client related messages.

This implements the latest version of the light client sync protocol, and also adds a test runner for the new accompanying tests. See ethereum/consensus-specs#2802

This extends the `--serve-light-client-data` launch option to serve locally collected light client data via libp2p. See ethereum/consensus-specs#2802 Backfill of historic best `LightClientUpdate` data and the client side of the protocol are not yet implemented. To test, in `conf.nim` change `serveLightClientData`'s `defaultValue` to `true`, then run this command: ``` scripts/launch_local_testnet.sh --kill-old-processes --preset minimal \ --nodes 4 --disable-htop --stop-at-epoch 7 ``` The log files of the beacon nodes will be in the `local_testnet_data` directory. They are named `log0.txt` through `log3.txt`. The logs can be browsed for light client related messages.

etan-status · 2022-01-28T11:28:55Z

Updated protocol ID for GetLightClientBootstrap from /eth2/beacon_chain/req/light_client_bootstrap_by_block_root/1/ to /eth2/beacon_chain/req/light_client_bootstrap/1/

This implements the latest version of the light client sync protocol, and also adds a test runner for the new accompanying tests. See ethereum/consensus-specs#2802

This extends the `--serve-light-client-data` launch option to serve locally collected light client data via libp2p. See ethereum/consensus-specs#2802 Not yet implemented: - Backfill of historic best `LightClientUpdate`. - Client side of the protocol. Known issues: - Heavy forking for more than a sync committee periods may result in incorrect best `LightClientUpdate` computation. To test, in `conf.nim` change `serveLightClientData`'s `defaultValue` to `true`, then run this command: ``` scripts/launch_local_testnet.sh --kill-old-processes --preset minimal \ --nodes 4 --disable-htop --stop-at-epoch 7 ``` The log files of the beacon nodes will be in the `local_testnet_data` directory. They are named `log0.txt` through `log3.txt`. The logs can be browsed for light client related messages.

Incorporates the latest changes to the light client sync protocol based on Devconnect AMS feedback. Note that this breaks compatibility with the previous prototype, due to changes to data structures and endpoints. See ethereum/consensus-specs#2802 Due to the amount of changes, it is best to review the heavily edited files from scratch (likewise, for the spec).

Incorporates the latest changes to the light client sync protocol based on Devconnect AMS feedback. Note that this breaks compatibility with the previous prototype, due to changes to data structures and endpoints. See ethereum/consensus-specs#2802

etan-status · 2022-05-23T13:55:45Z

Nimbus (unstable) supports the latest protocols using _v0 suffix (Kiln / Prater).

Gossipsub:

/eth2/00000000/light_client_finality_update_v0/ssz_snappy
/eth2/00000000/light_client_optimistic_update_v0/ssz_snappy

Req/Resp:

/eth2/beacon_chain/req/light_client_bootstrap/0/
/eth2/beacon_chain/req/light_client_updates_by_range/0/
/eth2/beacon_chain/req/light_client_finality_update/0/
/eth2/beacon_chain/req/light_client_optimistic_update/0/

etan-status · 2022-06-16T17:08:51Z

Rebase to dev (re-ordering of Misc section)
Submitted PR 2/12 for review: Remove fork_version from LightClientUpdate #2916

etan-status · 2022-06-27T13:14:11Z

Rebase to dev (@spec_state_test_with_matching_config decorator)
Submitted PR 3/12 for review: Allow LightClientUpdate with genesis finality #2924

etan-status · 2022-06-29T13:35:33Z

Rebase to dev
Submitted PR 4/12 for review: Allow passing redundant next_sync_committee #2927

etan-status · 2022-06-30T10:31:42Z

Rebase to dev
Submitted PR 5/12 for review: Cleanup light client tests #2929

etan-status · 2022-07-01T22:58:02Z

Rebase to dev
Submitted PR 6/12 for review: Root next_sync_committee in attested_header #2932

etan-status · 2022-07-12T06:25:13Z

Rebase to dev (helper functions order, compute_sync_committee_period_at_slot, pyspec style, prefix update _period variables, forced update comments)
Submitted PR 7/12 for review: Add LightClientBootstrap #2935

etan-status · 2022-07-14T08:46:26Z

Rebase to dev (update_ prefix for _period variables)
Submitted PR 8/12 for review: Add functions for deriving light client data #2938

etan-status · 2022-07-20T12:41:47Z

Rebase to dev (split of LC protocol into multiple files, checks after each test step)
Submitted PR 9/12 for review: Add LightClient(Finality|Optimistic)Update #2944

etan-status · 2022-07-21T14:55:31Z

Rebase to dev (intro text formatting)
Submitted PR 10/12 for review: Manually trigger LightClientStore force updates #2947

etan-status · 2022-07-22T14:55:42Z

Rebase to dev (rename force update function, split into multiple files)
Submitted PR 11/12 for review: Describe light client sync process #2951

@hwwhww

While the current Altair specs define structures to aid light client development, one missing key aspect is the network protocol definition. Certain implementations have started defining their own REST based APIs, e.g., Lodestar at https://github.com/ChainSafe/lodestar/blob/master/packages/api/src/routes/lightclient.ts While such APIs are useful, REST does not seem to be the ideomatic choice as the sole API available at such a low level for Ethereum. This patch introduces a libp2p based protocol to allow light clients to sync to the latest `BeaconBlockHeader` in a trustless and decentralized manner, building on top of prior work from: - @hwwhww at #2267 - @jinfwhuang at #2786 - Lodestar's REST API (also has an endpoint to fetch merkle proofs!)

etan-status · 2022-07-22T16:05:11Z

Rebase to dev (this PR itself is the final part 12/12)

hwwhww

Epic work @etan-status!

Generally look good. 👍

There may be some design space on the light client incentives that we can figure out later.

specs/altair/light-client/p2p-interface.md

hwwhww · 2022-07-25T12:23:51Z

specs/altair/light-client/p2p-interface.md

+
+### Sync committee
+
+Whenever fork choice selects a new head block with a sync aggregate participation `>= MIN_SYNC_COMMITTEE_PARTICIPANTS` and a post-Altair parent block, full nodes with at least one validator assigned to the current sync committee at the block's `slot` SHOULD broadcast derived light client data as follows:


Note: There may be some room to optimize it, based on how many full nodes enable the light client protocol.

e.g., if most nodes are altruistic and enable light client protocol, maybe we only need to assign the sync committee aggregators to publish the updates.

Yeah, it may work with just aggregators, but it's too early to tell. Also have to consider that the topic also extends beyond just full nodes, with all light clients also being subscribed to it, so I erred towards having a few more broadcasts for now. Originally, I implemented it with just the proposer sending the update, but that has the obvious downside of the slot not appearing at all if the proposer does not implement the light client sync protocol – in general, it still worked surprisingly well though.

The messages involved here are tiny (BeaconBlockHeader + SyncAggregate essentially), and the additional broadcast lines up with the general concept to reward the sync committee for enabling light clients to sync. Furthermore, the sync committee already sends messages at 1/3 into the block, so sending the second message around that time also groups up those tasks nicely.

hwwhww · 2022-07-25T12:31:49Z

specs/altair/light-client/p2p-interface.md

+
+##### GetLightClientBootstrap
+
+**Protocol ID:** `/eth2/beacon_chain/req/light_client_bootstrap/1/`


Curious, should it be beacon_chain/ or light_client/?
I see that it's a "beacon chain" light client, but for implementers & service providers, maybe there some benefits of distinguishing them?

@etan-status what do you think?

I don't have a strong opinion on this.

beacon_chain fits for these reasons:

What is being downloaded are partial beacon blocks / beacon states

The other light client related messages, i.e., sync contributions, are also rooted in beacon_chain

The light client still needs to deal with the status, ping, metadata, goodbye messages, so it cannot solely use light_client, at least on libp2p

light_client fits for these reasons:

The topics are not necessary for collecting validator rewards

Easier to reason starting point to direct a client implementation to

Other impacted API should also be considered, e.g., REST, should it have it under the light client umbrella, or the beacon chain one? What about portal?

I'm fine with either way too! Let's keep it to beacon_chain now unless someone has a strong objection one day.

Co-authored-by: Hsiao-Wei Wang <[email protected]>

etan-status mentioned this pull request Jan 13, 2022

Implement light client syncing status-im/nimbus-eth2#2337

Closed

terencechain reviewed Jan 16, 2022

View reviewed changes