feat!(messagev2): tweak dag-cbor message schema

[rvagg]

For:

1. Efficiency: compacting the noisy structures into tuples representations and making top-level components of a message optional.
2. Migrations: providing a secondary mechanism to lean on for versioning if we want a gentler upgrade path than libp2p protocol versioning.

Closes: #351

In terms of what this looks like, an example of an original message is in #351 and with this change the same data would look like (as dag-json):

{
  "gs1": {
    "blk": [
      [
        { "/": { "bytes": "AVUSIA" } },
        { "/": { "bytes": "QgTLmh40xfCOmyCqdgkOcAILtWwMo9OvcpbNEFilESiQ/tIYSI8ITY355INftUrQRf/ZNuO/cmGwQmxRNSoJeBbtdEgruQhLSn7YrcUX8zceDgQ0tRFiXNGkF5IkPczc/ogJSw" } }
      ],
      [
        { "/": { "bytes": "AVUSIA" } },
        { "/": { "bytes": "xfPTKlWZ2kO4US4NsU11HtHWQJ2dy3rtBdbMbAmHxjTufEI28FF8INzGqZ6G5LrcAl5fXoG0WWEoUimfNvEQ6Qa6IW4ByDcDUq0ziUrrH+WNzNCC9FaSKBovz42VowZ4zaRVhw" } }
      ]
    ],
    "req": [
      [
        { "/": { "bytes": "GmIWTBeVSRyOznxIsnBumg" } },
        "n",
        101,
        { "/": "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi" },
        {"R":{":>":{"|":[{".":{}},{"a":{">":{"@":{}}}}]},"l":{"none":{}}}},
        {
          "AppleSauce/McGee": "yee haw"
        }
      ],
      [
        { "/": { "bytes": "vd/PRdnYTg27RolghnknVQ" } },
        "n",
        202,
        { "/": "bafyreibdoxfay27gf4ye3t5a7aa5h4z2azw7hhhz36qrbf5qleldj76qfy" },
        {"R":{":>":{"a":{">":{"@":{}}}},"l":{"none":{}}}},
        {}
      ]
    ],
    "rsp": [
      [
        { "/": { "bytes": "GmIWTBeVSRyOznxIsnBumg" } },
        34,
        [
          [
            { "/": "bafyreibdoxfay27gf4ye3t5a7aa5h4z2azw7hhhz36qrbf5qleldj76qfy" },
            "m"
          ]
        ],
        {}
      ],
      [
        { "/": { "bytes": "vd/PRdnYTg27RolghnknVQ" } },
        14,
        [],
        {
          "Hippity+Hoppity": {   "/": {   "bytes": "9V/48SUItj7yv+ynVXrpDfYxGl7BYxtKH6hDMQvZw6cQ6qzlob3XKtC/4El3HBHnVjOL2Thl5kXxreybnJnvQH+9T8aFnnkExa19yb0QpcwWlz1bKOwabdQ9n4L58Yw9A0GONQ"   }   }
        }
      ]
    ]
  }
}

In terms of bytes saved, using the randomish data in TestGraphsyncRoundTrip (i.e. it has some variability in output length and these are not using exactly the same data so consider it approximate):

• v1.0 -> v1.0 protobuf original: 213 bytes written by client, 25,361 bytes written by server
• v2.0 -> v2.0 prior to this change: 271 bytes written by requestor, 23,129 bytes written by server
• v2.0 -> v2.0 with this change: 239 bytes written by client, 22,076 bytes written by server

@mvdan @warpfork care to critique my schema?

Aside: it's interesting that we're even saving bytes written by server from protobuf to dag-cbor even without these changes. The majority of the data sent should be blocks and their CIDs and all these maps with string keys should drown out any saving CBOR gets from its compact int & length representation. Plus we have v1.0 doing int request IDs and v2 doing UUID (as bytes) request IDs so they're longer. Nothing in the protobuf spec is optional, although I believe repeated allows for zero occurrences so in effect the top-level items are optional at least. Perhaps it's all to do with moving the metadata into the core message rather than as an extension (so we save writing the string "graphsync/response-metadata" on each response). An interesting mystery that might be worth investigating at some point.

[hannahhoward]

So I have a few comments:

• I am pretty anxious about tuple representation for requests and responses. While the versioning helps, if we can move around fields without breaking versions, I much prefer that. Also seems like message size is already a reduction so... maybe that's fine?
• I feel less anxious about blocks being tuple. Blocks are blocks. They have a well defined standard and aren't likely to change.
• why do we need versioning in the message itself? Isn't the libp2p protocol version sufficient to give us this information?

[rvagg]

Adjusted schema as per feedback above and some discussion today via Zoom. The requests and responses are back to maps with keys because we don't expect them to repeat much and it's nice to have them descriptive. But I've also optionalised more of the fields in there so they could be left out entirely without any pain.

{
    "gs2": {
        "blk": [
            [
                { "/": { "bytes": "AVUSIA" } },
                { "/": { "bytes": "QgTLmh40xfCOmyCqdgkOcAILtWwMo9OvcpbNEFilESiQ/tIYSI8ITY355INftUrQRf/ZNuO/cmGwQmxRNSoJeBbtdEgruQhLSn7YrcUX8zceDgQ0tRFiXNGkF5IkPczc/ogJSw" } }
            ],
            [
                { "/": { "bytes": "AVUSIA" } },
                { "/": { "bytes": "xfPTKlWZ2kO4US4NsU11HtHWQJ2dy3rtBdbMbAmHxjTufEI28FF8INzGqZ6G5LrcAl5fXoG0WWEoUimfNvEQ6Qa6IW4ByDcDUq0ziUrrH+WNzNCC9FaSKBovz42VowZ4zaRVhw" } }
            ]
        ],
        "req": [
            {
                "ext": {
                    "AppleSauce/McGee": "yee haw"
                },
                "id": { "/": { "bytes": "k3Nu8gWHTfq53ypWnVFtwg" } },
                "pri": 101,
                "root": { "/": "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi" },
                "sel": {"R":{":>":{"|":[{".":{}},{"a":{">":{"@":{}}}}]},"l":{"none":{}}}},
                "type": "n"
            },
            {
                "id": { "/": { "bytes": "mdBpy6/wQyqT1LiaihfQWQ" } },
                "pri": 202,
                "root": { "/": "bafyreibdoxfay27gf4ye3t5a7aa5h4z2azw7hhhz36qrbf5qleldj76qfy" },
                "sel": {"R":{":>":{"a":{">":{"@":{}}}},"l":{"none":{}}}},
                "type": "n"
            }
        ],
        "rsp": [
            {
                "meta": [
                    [
                        { "/": "bafyreibdoxfay27gf4ye3t5a7aa5h4z2azw7hhhz36qrbf5qleldj76qfy" },
                        "m"
                    ]
                ],
                "reqid": { "/": { "bytes": "k3Nu8gWHTfq53ypWnVFtwg" } },
                "stat": 34
            },
            {
                "ext": {
                    "Hippity+Hoppity": { "/": { "bytes": "9V/48SUItj7yv+ynVXrpDfYxGl7BYxtKH6hDMQvZw6cQ6qzlob3XKtC/4El3HBHnVjOL2Thl5kXxreybnJnvQH+9T8aFnnkExa19yb0QpcwWlz1bKOwabdQ9n4L58Yw9A0GONQ" } }
                },
                "reqid": { "/": { "bytes": "mdBpy6/wQyqT1LiaihfQWQ" } },
                "stat": 14
            }
        ]
    }
}

In terms of bytes sent, the difference is negligible when averaged out. We're sending a few more bytes for the map keys but not enough of them and the blocks are still tuples, we're also saving on optional fields by skipping them entirely.

Unfortunately there's more pointers in here than I'd like, getting bindnode working with implicit values that don't need to be backed by pointers might be nice to help with that, but tricky (/cc @mvdan - e.g. optional but implicit 0 should mean I don't need a pointer field for that).

[hannahhoward]

This schema is looking good @rvagg