[Persistence] Merkling a State Hash

[Olshansk]

Objective

Make the persistence "a real blockchain".

Expand on the existing SQL-based persistence implementation to create a verifiable authenticator (i.e. the state hash) which can be used to guarantee the integrity/immutability of the blockchain and verify various parts of the state (i.e. Merkle Proofs).

Origin Document

Pocket protocol persistence specification defines the persistence specification.

The current persistence interface can be found here with the shared interface available here.

This pull request to the persistence specification contains additional details and a spec coverage checklist that this will aim to implement.

Goals / Deliverables

• Research & Specification
  • Prepare a presentation explaining how Merkle Trees & Key-Value stores operate in blockchains
  • Update the persistence spec from v1.0.3 (the PR mentioned above) to v1.0.4
• Integration
  • Update the primary Persistence module interface
  • Hook up the KV Store, Merkle Tree and Persistence Module to function together
• Merkle Tree
  • Design an interface to interact with a Sparse Merkle Tree
  • Select an open-source and/or implement a Sparse Merkle Tree for Pocket based on an existing specification
• Key-Value Store
  • Design an interface to interact with a Key-Value store via the Merkle Tree
  • Select an open-source Key-Value store for Pocket based on an existing specification
• CLI
  • If needed, update the existing pocket client CLI to test, develop and drive the KV store (similar to Tendermint's [abci-cli]
• Testing
  • Test each new component / interface in isolation via:
    • Unit tests
    • Fuzzing tests
  • Determine if there is some sort of testing framework that can be implemented (i.e. DSL-like)
  • Determine the degree to which end-to-end tests can be implemented, or simply documented, and do so

General milestone checklist

• Update all the relevant CHANGELOGs
• Update all the relevant READMEs
• Update the source code tree explanation
• Add or update a state, sequence or flowchart diagram using mermaid
• Create followup milestones + issues
• Document small TODO along the way

Non-goals

• End-to-end benchmarking of the key-value store and database engine
• Functional state sync

Testing Methodology

• All tests: make test_all
• LocalNet: verify a LocalNet is still functioning correctly by following the instructions at docs/development/README.md
  Remove

---

Creator: @Olshansk
Co-Owners: ???

[Olshansk]

We're continuing to do research but a high level plan in terms of implementing this will be:

1. Pick a tree and KV and build a small interface on top of it. The underlying tech might change but the goal is to keep the interface small.
2. Add a block store.
3. Add loading a state from the block store.
4. Add the state trie for a single dataset/set (e.g. validators) and only compute the state hash using that one protobuf
5. Add all the other protocol actors
6. Identify all the unknown unknowns and learnings from the steps above and continue implementation

Note that steps (2) and (3) above will be able to unblock the following efforts:

• Enable state sync to be done in a parallel effort
• Enable infra work to start on this front

[Olshansk]

The research will be presented in this document once complete: https://docs.google.com/presentation/d/1dgXVKLf_uqnviCMC-VfVsmy64j8ihY3AkySZDMr3tyU/edit#slide=id.g13a056bedcb_0_22

[Olshansk]

This is a WIP but being tackled here: #152

[Olshansk]

This was an integration issue originally and therefore the scope is too big.

#147