[IBC] Implement ICS-24 - Tracking IBC store transitions in the networ…

…k state (#847) ## Description  ### Summary generated by Reviewpad on 12 Jul 23 20:45 UTC This pull request includes the following changes: - The file `main_test.go` was newly added to the repository and includes various changes such as package definition, imports, variable declaration, and function definitions related to IBC modules testing. - The file `proofs_ics23.go` includes changes in error handling for the functions `createMembershipProof` and `createNonMembershipProof`, replacing the error `coreTypes.ErrCreatingProof` with `coreTypes.ErrIBCCreatingProof` for more specific error messages. - The file `keys_ics03.go` was renamed from the `ibc/host` directory to the `ibc/path` directory. Additionally, the package declaration was changed from `host` to `path`. These changes seem related to the organization of the file within the codebase. - The `config.validator2.json` file includes changes in the `ibc` section, adding two new keys: `private_key` and `stores_dir`. The `enabled` key remains unchanged. The `private_key` key has been assigned a long string value, and the `stores_dir` key has been assigned the value "/var/ibc". - The file `internal/testutil/ibc/mock.go` includes renaming of the functions `BaseIbcMock` to `BaseIBCMock` and `IbcMockWithHost` to `IBCMockWithHost`. These changes update the function names to follow proper naming conventions. - The file includes the implementation of a store manager for IBC (Inter-Blockchain Communication), responsible for managing provable stores, caching, pruning, and restoring data. - The addition of a new function, `GetIBCStoreUpdates`, which retrieves the set of key-value pairs updated at the current height for the IBC store. - Changes in the state hashes used in the tests in the file `state_test.go`. - The file `runtime/manager_test.go` includes the addition of new fields `PrivateKey` and `StoresDir` to the `IBC` struct in the `NewManagerFromReaders` function. - Changes in the `config.validator4.json` file include modifications in the `ibc` section, adding two new properties: `private_key` and `stores_dir`. The `enabled` property remains unchanged. - The file `prefix.go` was renamed from the `ibc/host` package to the `ibc/path` package. The similarity index indicates that most of the content remains unchanged. - There was a change in error handling logic in a file involving the renaming of the error variable and addressing the error during the creation of a new public key. - The file `gov.go` includes changes in import statements, variable declarations, and function modifications related to new IBC store transaction fees. - The file `persistence/types/ibc.go` is a new file added to the codebase, containing functions and constants related to IBC. - The file `persistence/types/ibc.go` in the `persistence` package includes changes, introducing methods for setting and retrieving key-value pairs in the IBC store table. - The file `persistence/trees.go` includes changes related to the addition of a new tree and corresponding functions to handle updates for the tree. - The file `utils_test.go` includes changes in the casing of a function call from `ibcUtils.IbcMockWithHost` to `ibcUtils.IBCMockWithHost`. - The file `ibc/host/path_test.go` was renamed to `ibc/path/path_test.go` with no other changes in the file. - Changes in default variable values related to IBC functionality. - The file `db.go` includes a new function, `initialiseIBCTables`, responsible for initializing an IBC store table in a PostgreSQL database. - A new file, `ibc.feature.wip`, was added, containing integration and end-to-end tests for various scenarios related to IBC functionality. - Changes in the `PROTOCOL_STATE_HASH.md` file include additions and modifications related to the IBC state tree and the handling of IBCMessage objects. - The file `ibc/host.go` includes changes in comments, import statements, and the addition of fields and methods related to the host's interaction with the IBC store. - A new file, `convert.go`, was added in the `ibc/types` directory, containing functions and imports to work with IBC messages. - Changes in the `utils_test.go` file include modifications in the casing of a function call from `ibcUtils.IbcMockWithHost` to `ibcUtils.IBCMockWithHost`. - The file `ibc/host/path_test.go` was renamed to `ibc/path/path_test.go` with no other changes in the file. - Changes in default variable values related to IBC functionality. - A new file, `db.go`, was added, containing a function  ## Issue Fixes #840 ## Type of change Please mark the relevant option(s): - [x] New feature, functionality or library - [ ] Bug fix - [ ] Code health or cleanup - [ ] Major breaking change - [ ] Documentation - [ ] Other  ## List of changes - Add new `IbcMessage` protobuf with 2 types - `UpdateIbcStore` and `PruneIbcStore` - Track IBC store changes in the mempool - Update IBC state tree using these state transition messages ## Testing - [x] `make develop_test`; if any code changes were made - [x] `make test_e2e` on [k8s LocalNet](https://github.com/pokt-network/pocket/blob/main/build/localnet/README.md); if any code changes were made - [ ] `e2e-devnet-test` passes tests on [DevNet](https://pocketnetwork.notion.site/How-to-DevNet-ff1598f27efe44c09f34e2aa0051f0dd); if any code was changed - [x] [Docker Compose LocalNet](https://github.com/pokt-network/pocket/blob/main/docs/development/README.md); if any major functionality was changed or introduced - [x] [k8s LocalNet](https://github.com/pokt-network/pocket/blob/main/build/localnet/README.md); if any infrastructure or configuration changes were made ## Required Checklist - [x] I have performed a self-review of my own code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have added, or updated, [`godoc` format comments](https://go.dev/blog/godoc) on touched members (see: [tip.golang.org/doc/comment](https://tip.golang.org/doc/comment)) - [x] I have tested my changes using the available tooling - [ ] I have updated the corresponding CHANGELOG ### If Applicable Checklist - [x] I have updated the corresponding README(s); local and/or global - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have added, or updated, [mermaid.js](https://mermaid-js.github.io) diagrams in the corresponding README(s) - [ ] I have added, or updated, documentation and [mermaid.js](https://mermaid-js.github.io) diagrams in `shared/docs/*` if I updated `shared/*`README(s)
pokt-network · Jul 13, 2023 · 1742f81 · 1742f81
1 parent 021d90f
commit 1742f81
Show file tree

Hide file tree

Showing 52 changed files with 1,631 additions and 56 deletions.
diff --git a/Makefile b/Makefile
@@ -303,6 +303,9 @@ protogen_local: go_protoc-go-inject-tag ## Generate go structures for all of the
 	# P2P
 	$(PROTOC_SHARED) -I=./p2p/types/proto --go_out=./p2p/types ./p2p/types/proto/*.proto
 
+	# IBC
+	$(PROTOC_SHARED) -I=./ibc/types/proto --go_out=./ibc/types ./ibc/types/proto/*.proto
+
 	# echo "View generated proto files by running: make protogen_show"
 
 # CONSIDERATION: Some proto files contain unused gRPC services so we may need to add the following

diff --git a/build/config/config.validator1.json b/build/config/config.validator1.json
@@ -59,6 +59,8 @@
     "chains": ["0001"]
   },
   "ibc": {
-    "enabled": true
+    "enabled": true,
+    "private_key": "0ca1a40ddecdab4f5b04fa0bfed1d235beaa2b8082e7554425607516f0862075dfe357de55649e6d2ce889acf15eb77e94ab3c5756fe46d3c7538d37f27f115e",
+    "stores_dir": "/var/ibc"
   }
 }
diff --git a/build/config/config.validator2.json b/build/config/config.validator2.json
@@ -52,6 +52,8 @@
     "use_cors": false
   },
   "ibc": {
-    "enabled": true
+    "enabled": true,
+    "private_key": "0ca1a40ddecdab4f5b04fa0bfed1d235beaa2b8082e7554425607516f0862075dfe357de55649e6d2ce889acf15eb77e94ab3c5756fe46d3c7538d37f27f115e",
+    "stores_dir": "/var/ibc"
   }
 }
diff --git a/build/config/config.validator3.json b/build/config/config.validator3.json
@@ -52,6 +52,8 @@
     "use_cors": false
   },
   "ibc": {
-    "enabled": true
+    "enabled": true,
+    "private_key": "0ca1a40ddecdab4f5b04fa0bfed1d235beaa2b8082e7554425607516f0862075dfe357de55649e6d2ce889acf15eb77e94ab3c5756fe46d3c7538d37f27f115e",
+    "stores_dir": "/var/ibc"
   }
 }
diff --git a/build/config/config.validator4.json b/build/config/config.validator4.json
@@ -52,6 +52,8 @@
     "use_cors": false
   },
   "ibc": {
-    "enabled": true
+    "enabled": true,
+    "private_key": "0ca1a40ddecdab4f5b04fa0bfed1d235beaa2b8082e7554425607516f0862075dfe357de55649e6d2ce889acf15eb77e94ab3c5756fe46d3c7538d37f27f115e",
+    "stores_dir": "/var/ibc"
   }
 }
diff --git a/consensus/e2e_tests/utils_test.go b/consensus/e2e_tests/utils_test.go
@@ -122,7 +122,7 @@ func CreateTestConsensusPocketNode(
 	telemetryMock := baseTelemetryMock(t, eventsChannel)
 	loggerMock := baseLoggerMock(t, eventsChannel)
 	rpcMock := baseRpcMock(t, eventsChannel)
-	ibcMock := ibcUtils.IbcMockWithHost(t, eventsChannel)
+	ibcMock := ibcUtils.IBCMockWithHost(t, eventsChannel)
 
 	for _, module := range []modules.Module{
 		p2pMock,

diff --git a/ibc/docs/README.md b/ibc/docs/README.md
@@ -95,7 +95,9 @@ _Note_: Connections, Channels and Ports in IBC are not the same as networking co
 
 [ICS24][ics24] defines the IBC stores and these must be a part of the Pocket networks consensus state. As such the `ibcTree` is defined as one of the state trees used to generate the root hash. This tree contains the relevant information the hosts/relayers need to be able to use IBC, in accordance with ICS-24 and the other ICS components.
 
-TODO([#854](https://github.com/pokt-network/pocket/issues/854)): Add a local cache for changes to the state for use in the event of the node crashing.
+In order to interact with the IBC store's the host must create a `ProvableStore` instance which can make local changes to the state and propagate these through the network. This store maintains a local cache that can be backed up to disk and restored. In the event of a node failure, or local changes being unable to be propagated, the cache can be restored from the disk backup and the host can attempt to propagate the changes again.
+
+See: [store/provable_store.go](../store/provable_store.go) and [ics24.md](ics24.md) for more details on the specifics of the IBC store implementation for Pocket.
 
 ## Components
 

diff --git a/ibc/docs/ics24.md b/ibc/docs/ics24.md
@@ -1,32 +1,196 @@
 # ICS-24 Host Requirements <!-- omit in toc -->
 
 - [Overview](#overview)
+- [Host Configuration](#host-configuration)
 - [Implementation](#implementation)
+  - [Persistence](#persistence)
   - [Paths and Identifiers](#paths-and-identifiers)
   - [Timestamps](#timestamps)
+- [IBC State](#ibc-state)
+  - [IBC State Tree](#ibc-state-tree)
+  - [Data Retrieval](#data-retrieval)
+  - [IBC Messages](#ibc-messages)
+  - [IBC Message Handling](#ibc-message-handling)
+  - [Mempool](#mempool)
+  - [State Transition](#state-transition)
+- [Provable Stores](#provable-stores)
+  - [Caching](#caching)
 
 ## Overview
 
-[ICS-24][ics24] details the requirements of the host chain, in order for it to be compatible with IBC. A host is defined as a node on a chain that runs the IBC software. A host has the ability to create connections with counterparty chains, open channels, and ports as well as commit proofs to the consensus state of its own chain for the relayer to submit to another chain. The host is responsible to managing and creating clients and all other aspects of the IBC module.
+[ICS-24][ics24] details the requirements of the host chain, in order for it to be compatible with IBC. A host is defined as a node on a chain that runs the IBC software. A host has the ability to create connections with counterparty chains, open channels, expose ports, and commit proofs to the consensus state of its own chain for the relayer to submit to another chain. The host is responsible to managing and creating clients and all other aspects of the IBC module.
 
 As token transfers as defined in [ICS-20][ics20] work on a lock and mint pattern, any tokens sent from **chain A** to **chain B** will have a denomination unique to the connection/channel/port combination that the packet was sent over. This means that if a host where to shutdown a connection or channel without warning any tokens yet to be returned to the host chain would be lost. For this reason, only validator nodes are able to become hosts, as they provide the most reliability out of the different node types.
 
+## Host Configuration
+
+Only validators can be configured to be IBC hosts. If the IBC module, during its creation, detects the node is a validator (and the IBC `enabled` field in the config is `true`) it will automatically create a host.
+
+```json
+"ibc": {
+    "enabled": bool,
+    "private_key": string,
+    "stores_dir": string
+}
+```
+
+The `PrivateKey` field of the configuration is used to sign IBC store related messages and state transitions for inclusion in the block.
+
 ## Implementation
 
 **Note**: The ICS-24 implementation is still a work in progress and is not yet fully implemented.
 
 ICS-24 has numerous sub components that must be implemented in order for the host to be fully functional. These range from type definitions for identifiers, paths and stores as well as the methods to interact with them. Alongside these ICS-24 also defines the Event Logging system which is used to store the packet data and timeouts for the relayers to read, as only the `CommitmentProof` objects are committed to the chain state. In addition to these numerous other features are part of ICS-24 that are closely linked to other ICS components such as consensus state introspection and client state validation.
 
+### Persistence
+
+The IBC stores must be included in the networks consensus state as one of the many state trees. This is to ensure the IBC light clients verifying Pocket network's state can verify the inclusion or exclusion of IBC related information from the block headers.
+
+The following is a simplified sequence diagram of an IBC fungible token transfer. This requires **Chain A** to commit to its state the packet data related to the transfer, so that **Chain B** can verify the inclusion of this packet data with the light client of **Chain A** it runs.
+
+```mermaid
+sequenceDiagram
+  actor UA as User A
+  box Transparent Chain A
+    participant A1 as Validator A
+    participant A2 as IBC Host A
+    participant A3 as Light Client B
+  end
+  box Transparent Relayer
+    actor R1 as Relayer
+  end
+  box Transparent Chain B
+    participant B1 as Validator B
+    participant B2 as IBC Host B
+    participant B3 as Light Client A
+  end
+  actor UB as User B
+  R1->>R1: Watch(Chain A)
+  R1->>R1: Watch(Chain B)
+  UA->>+A1: Send 10$POKT to User B
+  A1->>A1: Lock(10$POKT)
+  A1->>+A2: Create(FungibleTokenPacketData)
+  A2->>-A1: Commit(FungibleTokenPacketData)
+  A1->>-A1: NewBlock()
+  R1->>R1: CheckNewBlockForIBCPackets()
+  R1->>+A2: QueryAndProve(FungibleTokenPacketData)
+  A2->>R1: FungibleTokenPacketData
+  A2->>-R1: Proof(FungibleTokenPacketData)
+  R1->>+B2: Validate(FungibleTokenPacketData, Proof(FungibleTokenPacketData))
+  B2->>+B3: Verify(Proof(FungibleTokenPacketData))
+  B3->>-B2: FoundInState(FungibleTokenPacketData)
+  B2->>-B1: Send 10$POKT to User B
+  B1->>B1: Mint(10$POKT)
+  B1->>UB: Receive 10$POKT from User A
+```
+
+As the IBC host will make changes to the IBC store locally, in response to functions being called by relayers, they require these changes to be propagated throughout the network (i.e. the mempool) and included in all other node's IBC stores so that during block production these changes are reflected in the state transition. This is done by utilizing the existing transaction workflow, adding the IBC store change messages to the mempool and then handling them as a new message type in block production/application logic.
+
+See: [IBC State](#ibc-state) below for more details on the IBC state transition process.
+
 ### Paths and Identifiers
 
-Paths are defined as bytestrings that are used to access the elements in the different stores. They are built with the function `ApplyPrefix()` which takes a store key as a prefix and a path string and will return the key to access an element in the specific store. The logic for paths can be found in [host/keys.go](../host/keys.go) and [host/prefix.go](../host/prefix.go)
+Paths are defined as bytestrings that are used to access the elements in the different stores. They are built with the function `ApplyPrefix()` which takes a store key as a prefix and a path string and will return the key to access an element in the specific store. The logic for paths can be found in [path/keys.go](../path/keys.go) and [path/prefix.go](../path/prefix.go)
 
-Identifiers are bytestrings constrained to specific characters and lengths depending on their usages. They are used to identify: channels, clients, connections and ports. Although the minimum length of the identifiers is much less we use a minimum length of 32 bytes and a maximum length that varies depending on the use case to randomly generate identifiers. This allows for an extremely low chance of collision between identifiers. Identifiers have no significance beyond their use to store different elements in the IBC stores and as such there is no need for non-random identifiers. The logic for identifiers can be found in [host/identifiers.go](../host/identifiers.go).
+Identifiers are bytestrings constrained to specific characters and lengths depending on their usages. They are used to identify: channels, clients, connections and ports. Although the minimum length of the identifiers is much less we use a minimum length of 32 bytes and a maximum length that varies depending on the use case to randomly generate identifiers. This allows for an extremely low chance of collision between identifiers. Identifiers have no significance beyond their use to store different elements in the IBC stores and as such there is no need for non-random identifiers. The logic for identifiers can be found in [path/identifiers.go](../path/identifiers.go).
 
 ### Timestamps
 
 The `GetTimestamp()` function returns the current unix timestamp of the host machine and is used to calculate timeout periods for packets
 
+## IBC State
+
+As mentioned [above](#persistence) the IBC store **MUST** be included in the consensus state of the network. As such the IBC store as defined in [ICS-24][ics24] has been implemented as a single IBC state tree.
+
+### IBC State Tree
+
+The IBC state tree is an `SMT` backed by a persistent `KVStore`, this is used for proof generation/verification. Data retrieval uses the `peristence` layer, see the [data retrieval](#data-retrieval) section below for more details.
+
+The root hash of the IBC state tree is included in the `rootTree` which computes the network's state hash for any given block. This allows verifiers to not only verify the inclusion/exclusion of any element in the IBC state tree itself but also that the IBC state tree was used to compute the network's state hash, by utilising the `CommitmentProof` object defined in [ICS-23][ics23].
+
+### Data Retrieval
+
+In order to query the IBC store the `persistence` layer is leveraged. All local changes to the IBC store are broadcasted as [IBC messages](#ibc-messages) and ultimately stored in each node's `peristence` layer. This allows for the efficient querying of the IBC store instead of having to query the IBC state tree directly.
+
+When attempting to generate a proof for a specific `key` in the IBC state tree the IBC host will import a local copy of the IBC state tree and use this to generate the proof. Otherwise all queries are handled by the `peristence` layer's underlying database.
+
+### IBC Messages
+
+Hosts maintain uncommitted changes in a local ephemeral IBC store while messages propagate through the mempool.
+
+These messages enable a variety of IBC related state changes such as creating light clients, opening connections, sending packets, etc... This is enabled by propagating `IBCMessage` types defined in [ibc/types/proto/messages.proto](../types/proto/messages.proto). This type acts as an enum representing two possible state transition events:
+
+- `UpdateIBCStore`: Updating the store with a key-value pair; adding a new or updating an existing element
+- `PruneIBCStore`: Pruning the store via its key; removal of an existing element
+
+_Note: In both types described above the `key` field **must** already be prefixed with the `CommitmentPrefix` and should be a valid path in the store._
+
+When changes are made locally they are not committed to the IBC store itself but are instead used to create an `IBCMessage` which is broadcasted to the network. This is akin to a simple send transaction that has been propagated throughout the mempool but has not been committed to the on-chain state.
+
+### IBC Message Handling
+
+Upon a node receiving an `IBCMessage` from the event bus it will use the `HandleMessage()` method of the `IBCModule` to add this message to the transactions mempool via the following steps:
+
+1. Wrap the `IBCMessage` within a `Transaction`
+2. Sign the `Transaction` using the `IBCModule`'s private key
+3. Broadcast the `Transaction` throughout the mempool
+
+```mermaid
+graph LR
+  subgraph Bus
+    A[Events]
+  end
+  subgraph I[IBC Host]
+    I1["HandleMessage(Message)"]
+  end
+  subgraph Handler
+    H1["ConvertIBCMessageToTransaction(IBCMessage)"]
+    subgraph Transaction
+      T1["coreTypes.Transaction{Msg: IBCMessage}"]
+    end
+    H2["SignTransaction(Transaction)"]
+  end
+  subgraph Mempool
+    M1["ValidateTransaction(Transaction)"]
+    M2["AddToMempool(Transaction)"]
+  end
+  Bus--Message-->I
+  I--IBCMessage-->Handler
+  H1--IBCMessage-->Transaction
+  Transaction--Transaction-->H2
+  Handler--Transaction-->Mempool
+  M1--Transaction-->M2
+```
+
+See: [ibc/module.go](../module.go) for the specific implementation details.
+
+### Mempool
+
+With the `IBCMessage` now propagated through the network's mempool, when it is reaped (by the block proposer) the message's validity will be handled by first determining the type of the `IBCMessage`:
+
+- `UpdateIBCStore`: The `key` and `value` fields are tracked by persistence and used to update the `ibc` store state tree
+- `PruneIBCStore`: The `key` field is tracked by persistence and marked for removal in the `ibc` store state tree
+
+### State Transition
+
+See: [PROTOCOL_STATE_HASH.md](../../persistence/docs/PROTOCOL_STATE_HASH.md#ibc-state-tree) for more information on how the persistence module uses the data it has tracked from the `IBCMessage` objects, in order to update the actual state trees and in turn the root hash.
+
+## Provable Stores
+
+The `ProvableStore` interface defined in [shared/modules/ibc_module.go](../../shared/modules/ibc_module.go) is implemented by the [`provableStore`](../store/provable_store.go) type and managed by the [`StoreManager`](../store/store_manager.go).
+
+The provable stores are each assigned a `prefix`. This represents the specific sub-store that they are able to access and interact with in the IBC state tree. When doing any operation `get`/`set`/`delete` the `prefix` is applied to the `key` provided to generate the `CommitmentPath` to the element in the IBC state tree.
+
+The provable stores do not directly interface with the IBC state tree but instead utliise the `peristence` layer to query the data locally. This allows for the efficient querying of the IBC store instead of having to query the IBC state tree directly. Any changes made by the `ProvableStore` instance are broadcasted to the network for inclusion in the next block, being stored in their mempools.
+
+### Caching
+
+Every local change made to the IBC store (`update`/`delete`) is stored in an in-memory cache. These caches are written to disk by the [`StoreManager`](../store/store_manager.go).
+
+In the event of a node failure, or local changes not being propagated correctly. Any changes stored in the cache can be "replayed" by the node and broadcasted to the network for inclusion in the next block.
+
+_TODO: Implement this functionality_
+
 [ics24]: https://github.com/cosmos/ibc/blob/main/spec/core/ics-024-host-requirements/README.md
 [ics20]: https://github.com/cosmos/ibc/blob/main/spec/app/ics-020-fungible-token-transfer/README.md
 [smt]: https://github.com/pokt-network/smt