Fragmented bundle to chain transport

[kriskowal]

What is the Problem Being Solved?

The chain has a maximum transaction size and a maximum message size (within a transaction). Some bundles are larger than either of these sizes. We do not want to set an arbitrary limit on bundle sizes, and would rather incentivize code sharing and smaller bundles economically.

Description of the Design

With #4396, we will have support for a new chain transaction for sending bundles. We will modify or supplement this path to support submission of partial bundles or “bundle fragments”. The protocol for submitting a bundle to the chain will become:

1. Deployer unzips the bundle locally and sends the chain a message with a “bill of materials”, the hashes and of every file in the bundle.
2. The chain responds by emitting an event containing a list of the hashes of files it does not already have.
3. Deployer sends a sequence of messages containing base64 encoded, delimited content and corresponding content hashes.
4. Deployer finally sends a message requesting the installation of a bundle.

An open consideration is whether the final install message is redundant with the original “bill of materials” message. Also, the bill of materials could conceivably be the compartment-map.json without alteration.

Security Considerations

Test Plan

[kriskowal]

cc @warner

[dckc]

Some of our modules are of the form const bundle = {..., stuff: "lsejlisjef...lisje"} including a string constant over 1M in length. What happens when we try to install a bundle containing such a string contstant?

• "Doctor! Doctor! It hurts when I do that" that is: don't do that. This feature is limited to moderate sized modules.
• other?

See also: short-term version of this feature in #5002

[kriskowal]

@dckc That sounds like a "nestedEvaluate" bundle. This design would not support that bundle format, but it should be trivial to replace all "nestedEvaluate" bundles with "endoZipBase64" (it’s the default now).

An alternative design that would support "nestedEvaluate" would just send substrings of the bundle, and in fact be bundle format agnostic. That sounds nice, but wouldn’t do anything for code reuse since the fragments are not aligned at module boundaries.

[michaelfig]

Some of our modules are of the form const bundle = {..., stuff: "lsejlisjef...lisje"} including a string constant over 1M in length. What happens when we try to install a bundle containing such a string constant?

I'm working on sending those stringified bundles separately, so that may help mitigate the problem.

[dckc]

@dckc That sounds like a "nestedEvaluate" bundle. This design would not support that bundle format, but it should be trivial to replace all "nestedEvaluate" bundles with "endoZipBase64" (it’s the default now).

I don't see how the format makes any difference, but the bundles in question bear "moduleFormat":"endoZipBase64". For example:

export default {"moduleFormat":"endoZipBase64","endoZipBase64":"UEsDBAoAAAAAAAAAAADoIW

$ wc ../run-protocol/bundles/bundle-vaultFactory.js 
      0       3 1130256 ../run-protocol/bundles/bundle-vaultFactory.js