Handing random bitflips

[haydentherapper]

Copying over the conversation had on Slack in regards to https://groups.google.com/a/chromium.org/g/ct-policy/c/S17_j-WJ6dI, in which one of the CT logs had to be turned down to a bitflip.

The bitflip in the CT log occurred after extracting the to-be-signed (TBS) portion of the certificate but before computing the merkle leaf hash, an issue unique to CT logs. You can't modify the leaf hash to match since you invalidate the tree's consistency. You can't update the precertificate (outside of the merkle hash) to match the TBS, since that breaks the signature.

Note that if a bitflip occurs on the entry stored in Trillian, it may be recoverable. If it occurs after the merkle hash is computed, then the tree was constructed based on the correct entry, so we can simply edit the entry in the DB without touching the tree. However, if the entry was bitflipped before the merkle hash was computed, it would likely make the entry unusable.

We discussed exploring if it's possible to detect this efficiently and soon after the bitflip. Error correction might be possible if a new tree head hasn't been issued too.

If this did occur in Rekor and was not recoverable, we would create a new shard.

[haydentherapper]

cc @trishankatdatadog @znewman01 @SantiagoTorres