use raw SHA256 as message-id and reduce size to 8 bytes

[blacktemplar]

Resolves #2039

Note to implementers: message-id was also reduced to 8-bytes. See discussion on bandwidth analysis below

[protolambda]

Suggestion, if we are changing the message ID function: should we truncate to e.g. 20 out of 32 bytes of the hash? It's neat to see the full hash-tree-root hash go around the network, but it's not necessary. If the chance of collision is low, and the effect of a collision is low as well, then it may be worth the gain in efficiency. If anyone likes to help test how large the current ratio between message IDs and messages is on medalla network, we can make a more educated decision here.

[djrtwo]

Seems like the "." at the end was a typo

Suggested change

	message-id: SHA256(message.data).
	message-id: SHA256(message.data)

[blacktemplar]

I am used to finish a sentence (even after a colon) with a dot within the math environment if the sentence is not continued afterwards, but we can remove that of course if you prefer it.

[djrtwo]

I lean toward shortening the byte sequence

Why 20 instead of say 16?

[protolambda]

Why 20 instead of say 16?

Being a little conservative, and it doesn't have much value to be a power of 2 here. We can look at the spec how bad a collision would really be. If it's hard enough to mine within the time span of the gossip message cache, it may not even matter much, and could be short. Also need to consider what happens to scoring if there would be a forged collision. And look at the gossip to say how much each byte here really means in terms of traffic (could be significant or not).

[djrtwo]

yeah, the seen_ttl is ~1 epoch. Def not worried on the 2^20 in that time frame and my intuition says not worried on the 2^16 collision

[protolambda]

You mean 2**(16*3) there? Wonder how short we could/should go.

[AgeManning]

For reference there is a config parameter which specifies the maximum number of message_ids in an IHAVE message. The default we have at the moment is 5000 (and a peer can at most send 10 per heartbeat, but a good peer should only send 1 to each peer it decides to).

So rough calculations give (including 1 byte protobuf varint for encoding) is like 5000*21 = 105kb per heartbeat per peer per topic. For a 16 byte message id its about 85kb.
This is assuming our history (which is 4.2 seconds) contains 5000 messages for a topic.

In total, I think across gossip we should expect around V/8 (aggregate messages) + V (subnets) messages per epoch in total. So lets say an extreme case of 4M vals, thats 4.5M messages per epoch (which is the time our duplicate cache remembers messages).

So I think the probability of at least one collision per epoch with message-id bit size x and expected messages k is (1-e^(-k(k-1)/(2*2^x)))

If this back of envelope calc is right:

• 8 bytes ~ 5.5e-5%
• 16 bytes ~ 3.0e-26%
• 20 bytes ~ 6.9e-34%

Disclaimer, this is just a super quick calc, some of my logic may be wrong here....

If we do get a collision, I imagine we would just drop collision messages. Which ever message comes first will be accepted and its counterpart dropped by the duplicate cache, which may be different for different peers.

Edit: @blacktemplar pointed out flaws in my math - edited to his suggestions

[blacktemplar]

Based on the calculations @AgeManning presented I would propose even using only 8 bytes. It results in a collision probability of 5.5e-5% which means in average one collision (in the whole network) every 22 years, I think that is acceptable. Also remember that this is the worst case.

The worst that can happen in case of a collision anyway is missing a block (but the probability for that is like 700,000 times smaller as a general collision) or in the most common case missing an attestation.

Edit: Just did some worst case calculations on traffic:

I only look at the attestation topics (aggregate + subnets) and assume 50 peers and the worst case that all peers are in all subnets. Then the average amount of messages in the cache would be:

(V + min(V/8, 16 * 64 * 32)) / 32 / 12 * 2.1.

In this calculation we can almost ignore the IHAVE size limit of 5000 per topic, since no topic gets close to 5000 except for the beacon_aggregate_and_proof topic which can reach slightly above 5000.

So this gives us with 50 peers and 8 byte ids a traffic (outgoing and ingoing) of

(V + min(V/8, 16 * 64 * 32)) / 32 / 12 * 2.1 * (8+1) * 50 / 0.7 = (V + min(V/8, 32768)) * 3.515625.

In numbers for different amounts of validators:

• 40k vals: 158 KB/s
• 400k vals: 1.52 MB/s
• 4kk vals: 14.2 MB/s

This is only the traffic for IHAVE messages. To compare that with the worst case traffic on the mesh assuming 8 peers per mesh we get a mesh traffic per second of

8 * (min(V/8, 16 * 64 * 32) * |SignedAggregateAndProof| + V * |Attestation|) / 32 / 12

Using the maximum sizes calculated in https://benjaminion.xyz/eth2-annotated-spec/phase0/beacon-chain/ we get

|SignedAggregateAndProof| = 8 + 480 + 96 + 96  = 680
|Attestation| = 480

which results in a mesh traffic of (V + min(V/8, 32768) * 680 / 480) * 10 bytes per second, in numbers again:

• 40k vals: 471 KB/s
• 400k vals: 4.46 MB/s
• 4kk vals: 40.5 MB/s

Note that those calculations average over the time horizon. In reality most messages will arrive at the beginning of each slot, so the traffic at the beginning of a slot may be significantly higher.

So I think 8 byte message ids result in a reasonable IHAVE traffic compared to mesh traffic. Note again that all the calculations above are the absolute (theoretical) worst case.

Edit: Edited for correct numbers considering the TARGET_AGGREGATORS_PER_COMMITTEE when max number of committees is reached as @djrtwo suggested.

Edit2: I just remembered that our message cache remembers messages for 4.2 seconds but we only advertise the messages received within the last 3 heartbeats, i.e. within the last 2.1 seconds, I adapted the calculations accordingly.

[djrtwo]

Awesome! Thanks @blacktemplar @AgeManning

Interesting -- the ratio of IHAVE to actual messages is pretty high even with the 8 byte message ID. And this is assuming max attestation size, which at lower validator counts, you don't fully hit that size.

I definitely agree going down to the 8 bytes here. Seeing a higher IHAVE cost or even at parity with the actual messages themselves is not desirable. Lets go ahead and get this into the PR

I think we might consider putting a cap (maybe dynamic) on the number of message IDs included in the IHAVEs to further reduce the overhead of the metadata gossip. For example for the attestation subnets, you could probably get away with max IHAVE ids per message being V // 32 // 64 // 4 -- that is, being 25% the committee size. If you have a reasonable number of peers on that topic, getting partial IHAVEs from many peers of them should retain a reasonable amount of responsiveness.

Thoughts on reducing max sizes here?
Is max IHAVE size easily (dynamically) tune-able? and can it be per topic?

Oh and an aside, the number of target aggregates is TARGET_AGGREGATORS_PER_COMMITTEE (16) per committee. Once you are at max committees per epoch, this doesn't scale up anymore

[blacktemplar]

I just updated my calculations based on your comment @djrtwo and based on a mistake on the number of heartbeats we propagate the same message.

I think its hard to calculate the effects of a restrictive IHAVE size limit maybe its easier to model it and check how the latencies change. I think the question is do we need a size limit or are we happy with the traffic as is (note that my calculations all assume that a node and all its peers is susbscribed to all subnets which is only interesting for a node that has many validators).

To answer your other questions:

Is max IHAVE size easily (dynamically) tune-able? and can it be per topic?

The current spec (and implementation) defines the IHAVE limits as a spam protection measure and not really as a traffic reduction tool. Currently only the same limit can be used for each topic and they are not configurable individually.

[djrtwo]

Nice. The ratio on meta-data messages to actual looks better.

I think its hard to calculate the effects of a restrictive IHAVE size limit maybe its easier to model it and check how the latencies change.

Yeah, agreed. The bandwidth looks okay for our purposes right now and in the context of needing to ship, but some more modeling and potential avenues for optimizing bandwidth might be warranted down the line