Create a bundle auditing tool (endo unbundle)

[kriskowal]

For the purposes of verifying the content and integrity of a bundle, as motivated by the need to audit governance proposals with Agoric chains, we need a tool that can verify the integrity of a bundle and extract it to the file system so an auditor can inspect the full working set of sources, without the distraction of adjacent files from the original source tree, notably in the absence of source maps.

Currently, we have @endo/check-package which provides the implementation of the integrity check. We also have most of the tools necessary for extraction between @endo/zip and JSON.parse. We just need a CLI veneer. That could be an endo subcommand.

This would be a substitute for the manual advice for validating a bundle:

• Verify that jq -r .moduleFormat bundle.json is endoZipBase64
• Extract the zip file jq -r .endoZipBase64 bundle.json | base64 -d > bundle.zip (or the governance proposal can just refer to a zip file, or even a tarball)
• unzip bundle.zip
• sha512sum compartmetn-map.json must match the proposed bundle identifier!
• In compartment-map.json, every module description must include a sha512 that matches the sha512sum of the corresponding .js.pre.json module file.
• You can extract the readable source from each of the precompiled modules with jq -r '.__syncModuleProgram__' compartment/module.pre.js.json

[dckc]

I prototyped a web-based tool for exploring bundles after they are put on chain:

• Bundle Explorer (prototype) shows modules, sizes from InstallBundle tx Agoric/agoric-sdk#8416

[dckc]

stretch goal: bundle diff... it would be nice if I went from marshal 0.6.1 to 0.6.2, it told me what the differences are, rather than saying "only in b2: 0.6.2"

[dckc]

More motivation:

• 2013-12: struggling to reproduce assets: Add Crabble proposal Agoric/agoric-3-proposals#41 (comment)

[dckc]

stretch goal: bundle diff...

I just discovered:

• diffoscope: in-depth comparison of files, archives, and directories

diffoscope tries to get to the bottom of what makes files or directories different. It will recursively unpack archives of many kinds and transform various binary formats into more human-readable form to compare them. It can compare two tarballs, ISO images, or PDF just as easily.

[kriskowal]

Here’s an “endo unbundle” implementation that will work on Mac.

function undo() {
  mkdir -p "$1" && jq -r .endoZipBase64 | base64 -d | tar xC "$1"
}

Usage:

undo bundle < bundle.json