Verify integrity of get-entries response

[AGWA]

Since logs have accidentally returned the wrong entries from get-entries, it's important to verify the integrity of the response rather than relying on the validity of the log server's TLS certificate.

It's not that hard to verify entries using a Collapsed Merkle Tree. https://github.com/SSLMate/certspotter/blob/master/auditing.go contains a Go implementation with a GPLv3-compatible license of a Collapsed Merkle Tree which you could use.

You would need to add a column to the ct_log table to store a Collapsed Merkle Tree in JSON form (it is O(log n) hashes, so it's not big).

To download new entries from a log you:

• Unmarshal the Collapsed Merkle Tree into memory.
• Call get-sth as usual to get the latest STH.
• Use get-entries to download the entries between the current tree size and the latest STH's tree size, as usual.
• Add the Merkle Leaf Hash (SHA256(0x00 || data)) of every entry's leaf_input to the Collapsed Merkle Tree, using the Add function. Note that this has to be done serially, in order of the leaf index.
• When you're done downloading all the entries, ask the Collapsed Merkle Tree for the root hash (using the CalculateRoot function) and make sure it equals the latest STH's root hash.
• Marshal the Collapsed Merkle Tree back into the database, and update the log's tree size, as usual.

If the root hash doesn't match, you should log an error (ideally including both the previously-observed STH and the new STH so there's evidence if the log has misbehaved) and then try to fetch the entries again later.