feat: deterministic message hashing #1233

fryorcraken · 2023-03-13T04:10:13Z

Ref #1208
After checking with @jm-clius, we should not change the gossipsub message id just yet. This change will have to be coordinated across the implementations.

However, we can provide the MUID calculation logic for now. This will be first use by nwaku to remove store duplicates.

fryorcraken · 2023-03-13T04:54:39Z

Can only be merged once #1230 introduces the meta field.

package.json

fryorcraken · 2023-03-13T23:44:36Z

Remove reviewers as PR is actually still draft

github-actions · 2023-03-14T00:31:45Z

size-limit report 📦

Path	Size	Loading time (3g)	Running time (snapdragon)	Total time
Waku core	121.45 KB (0%)	2.5 s (0%)	2.9 s (+23.83% 🔺)	5.4 s
Waku default setup	391 KB (0%)	7.9 s (0%)	10.7 s (+110.18% 🔺)	18.5 s
ECIES encryption	28.02 KB (0%)	561 ms (0%)	1.9 s (+64% 🔺)	2.5 s
Symmetric encryption	28.03 KB (0%)	561 ms (0%)	2.3 s (+83.28% 🔺)	2.9 s
DNS discovery	108.24 KB (0%)	2.2 s (0%)	3.3 s (-0.95% 🔽)	5.5 s
Privacy preserving protocols	118.32 KB (0%)	2.4 s (0%)	3.1 s (+12.56% 🔺)	5.4 s
Light protocols	120.11 KB (0%)	2.5 s (0%)	3.1 s (+12.89% 🔺)	5.5 s
History retrieval protocols	120.12 KB (0%)	2.5 s (0%)	3.1 s (+25% 🔺)	5.5 s
Deterministic Message Hashing	5.78 KB (+100% 🔺)	116 ms (+100% 🔺)	718 ms (+100% 🔺)	834 ms

fryorcraken · 2023-03-14T00:39:28Z

Now ready for review

fryorcraken · 2023-03-14T00:40:17Z

packages/message-hash/src/index.ts

+ * Deterministic Message Hashing as defined in
+ * [14/WAKU2-MESSAGE](https://rfc.vac.dev/spec/14/#deterministic-message-hashing)
+ */
+export function messageHash(


Interesting part: the deterministic message hash logic

fryorcraken · 2023-03-14T00:40:42Z

packages/core/src/lib/relay/message_id.ts

+import { toProtoMessage } from "../to_proto_message.js";
+
+const log = debug("waku:relay:message-id");
+export function gossipsubMessageId(


interesting part: the gossipsub message id calculation

danisharora099 · 2023-03-14T03:49:39Z

packages/core/src/lib/relay/message_id.ts

+export function gossipsubMessageId(
+  message: Message
+): Promise<Uint8Array> | Uint8Array {
+  const startTime = performance.now();


where does performance come from? can't find an import for this 👀

ah, TIL it's a native method as an alternative to Date.now() 💯

It's a JS API to handle performance. Which matters here as the methods need to execute within 100ms.

danisharora099 · 2023-03-14T03:58:24Z

packages/message-hash/src/index.ts

+  if (message.meta) {
+    bytesToHash = concat([
+      pubsubTopicBytes,
+      message.payload,
+      contentTopicBytes,
+      message.meta,
+    ]);
+  } else {
+    bytesToHash = concat([
+      pubsubTopicBytes,
+      message.payload,
+      contentTopicBytes,
+    ]);
+  }


is bytesToHash an apt naming?
as this variable does not hold the actual hash of the bytes but only the concatenated version - the hash is returned in the last line

It holds the bytes that needs to be hashed. The bytes to hash. What's apt?
Could probably just call it bytes

weboko · 2023-03-14T09:16:15Z

packages/message-hash/.prettierignore

+build
+bundle
+dist
+node_modules


Let's add CHANGELOG to ignore

fryorcraken · 2023-03-23T23:42:45Z

TODO: Check the API around the seen cache key and ensure MUID is used for it.

fryorcraken · 2023-04-24T10:41:32Z

Discussing with @jm-clius, we do not want to do the change on Relay at this stage. We will want to coordinate across implementations when we do it.

fryorcraken requested review from danisharora099, LNSD and weboko March 13, 2023 04:10

fryorcraken marked this pull request as draft March 13, 2023 04:54

weboko reviewed Mar 13, 2023

View reviewed changes

package.json Outdated Show resolved Hide resolved

fryorcraken force-pushed the feat/hash-message-hashing branch from 7c449b7 to a3ea420 Compare March 13, 2023 23:42

fryorcraken changed the title ~~chore: new empty message-hash package~~ feat: implement deterministic message hashing and use it for gossipsub message id Mar 13, 2023

fryorcraken changed the title ~~feat: implement deterministic message hashing and use it for gossipsub message id~~ feat!: implement deterministic message hashing and use it for gossipsub message id Mar 13, 2023

fryorcraken removed request for LNSD and danisharora099 March 13, 2023 23:44

fryorcraken force-pushed the feat/hash-message-hashing branch 2 times, most recently from 998e350 to 5f61914 Compare March 14, 2023 00:31

fryorcraken marked this pull request as ready for review March 14, 2023 00:39

fryorcraken requested review from weboko, alrevuelta, danisharora099 and LNSD March 14, 2023 00:39

fryorcraken commented Mar 14, 2023

View reviewed changes

danisharora099 approved these changes Mar 14, 2023

View reviewed changes

weboko reviewed Mar 14, 2023

View reviewed changes

packages/message-hash/.prettierignore

build

bundle

dist

node_modules

Copy link

Collaborator

weboko Mar 14, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add CHANGELOG to ignore

weboko approved these changes Mar 14, 2023

View reviewed changes

fryorcraken mentioned this pull request Mar 20, 2023

14/WAKU2-MESSAGE: Meta attribute and deterministic hashing #1208

Closed

7 tasks

fryorcraken force-pushed the feat/hash-message-hashing branch from 1f8030c to 65d6dc3 Compare March 20, 2023 04:44

chore: new empty message-hash package

76d1d4f

fryorcraken force-pushed the feat/hash-message-hashing branch from 65d6dc3 to 11b6450 Compare May 6, 2023 03:35

fryorcraken changed the title ~~feat!: implement deterministic message hashing and use it for gossipsub message id~~ feat: deterministic message hashing May 6, 2023

fryorcraken force-pushed the feat/hash-message-hashing branch from 11b6450 to 73043f5 Compare May 6, 2023 03:48

fryorcraken added 4 commits May 6, 2023 13:50

feat: implement deterministic message hash logic

fe57461

test: fix karma config

7198503

chore: improve var name

c781e49

chore: ignore changelog file as it's handled by release please

c8c2cbf

fryorcraken force-pushed the feat/hash-message-hashing branch from 73043f5 to c8c2cbf Compare May 6, 2023 03:50

fryorcraken merged commit 9eb3384 into master May 6, 2023

fryorcraken deleted the feat/hash-message-hashing branch May 6, 2023 03:59

weboko mentioned this pull request May 18, 2023

bug: inconsistent messages on eth-pm waku-org/examples.waku.org#199

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: deterministic message hashing #1233

feat: deterministic message hashing #1233

fryorcraken commented Mar 13, 2023 •

edited

Loading

fryorcraken commented Mar 13, 2023

fryorcraken commented Mar 13, 2023

github-actions bot commented Mar 14, 2023 •

edited

Loading

fryorcraken commented Mar 14, 2023

fryorcraken Mar 14, 2023

fryorcraken Mar 14, 2023

danisharora099 Mar 14, 2023

danisharora099 Mar 14, 2023

fryorcraken Mar 14, 2023

danisharora099 Mar 14, 2023

fryorcraken Mar 14, 2023

weboko Mar 14, 2023

fryorcraken commented Mar 23, 2023

fryorcraken commented Apr 24, 2023

feat: deterministic message hashing #1233

feat: deterministic message hashing #1233

Conversation

fryorcraken commented Mar 13, 2023 • edited Loading

fryorcraken commented Mar 13, 2023

fryorcraken commented Mar 13, 2023

github-actions bot commented Mar 14, 2023 • edited Loading

size-limit report 📦

fryorcraken commented Mar 14, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fryorcraken commented Mar 23, 2023

fryorcraken commented Apr 24, 2023

fryorcraken commented Mar 13, 2023 •

edited

Loading

github-actions bot commented Mar 14, 2023 •

edited

Loading