Formulate a consistent canonicalization strategy

[znewman01]

There's been confusion and bugs caused by inconsistent canonicalization (e.g., #214, #247). This issue represents an attempt to catalogue issues and possible solutions. CC @asraa @toby-jn

In the offline signing flow (output something from one TUF repo, copy it to another box, sign it, copy it back, import the signature), when should we canonicalize?

There are two approaches that seem coherent:

1. Canonicalize at signing time. This allows us to pass around metadata in JSON format, modify it before signing, etc. It has the downside of requiring signing tools to understand canonicalization (though that could be mitigated by adding a tuf canonicalize command).
2. Canonicalize at output time. This lets users treat the payload as a black box. It means they cannot modify it (unless they convert from canonicalized form back to metadata form).

In the go-tuf library, when should we canonicalize?

• Should we store metadata on-disk canonicalized?
• Should we store metadata in-memory canonicalized?
• Should we have sign.Sign (which signs a payload with a single "Signer") canonicalize?
• Should we have repo.Sign (which grabs keys associated with a role, grabs metadata for a role, calls sign.Sign, and stores the signed metadata) canonicalize?
  • If we add a repo.SignPayload (which grabs keys associated with a role and calls sign.Sign with given metadata, then returns the signed metadata) as in feat: add payload and add-signature commands. #214, should that canonicalize?

I think there's a consensus that clients must canonicalize before verifying signatures, at verification time. There are design decisions to be made there, but let's call them out-of-scope for this issue.

I might be missing something; feel free to add.

[toby-jn]

The reason I started looking at the canonicalization originally was due to #246 (also discussed in theupdateframework/specification#92). The result of this is that canonicalization should only be performed when calculating the disgest. Therefore I think repo.Sign and sign.Sign should deal with metadata files, and the specific key implementations should just just receive arbitrary []byte (which would be canonicalized JSON) and sign them.

In order to fit payload and add-signature subcommands into this model, the code perhaps needs some refactoring: currently sign.Sign does the work of canoncalizing and then passing to the signer and then attaches the result to the metadata.. so should be a matter of splitting up this operation into these individual steps and then exposing them separately as subcommands.

[rdimitrov]

Closing since the code base changed and this is no longer relevant.

Thanks for raising this 👍