Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Persistence] Base transaction indexer #151

Closed
8 tasks
andrewnguyen22 opened this issue Aug 5, 2022 · 2 comments · Fixed by #167
Closed
8 tasks

[Persistence] Base transaction indexer #151

andrewnguyen22 opened this issue Aug 5, 2022 · 2 comments · Fixed by #167
Assignees
Labels
core Core infrastructure - protocol related persistence Persistence specific changes

Comments

@andrewnguyen22
Copy link
Contributor

andrewnguyen22 commented Aug 5, 2022

Objective

  • CREATE an indexer for arbitrary transaction bytes
  • USE protobuf3 (already used throughout the project) for encoding and a filesystem db for storage
  • INDEX each transaction for <sha3>hash, height, sender, recipient (if applicable)
  • EXPOSE an interface for write and read
  • TEST create unit, fuzz, and integration tests

Example interface

type TxIndexer interface {  // TxIndexer interface defines methods to index and query transactions.

   // Index analyzes, indexes and stores a single transaction.
   Index(result *types.TxResult) error

   // GetByHash returns the transaction specified by hash or nil if the transaction is not indexed
   GetByHash(hash []byte) (*types.TxResult, error)

   // GetByHeight returns the transactions specified by height or nil if there are no transactions at that height
   GetByHeight(height int64) ([]*types.TxResult, error)

   // GetBySender returns the transactions signed by *sender* may be ordered descending/ascending
   GetBySender(sender address, descending bool) ([]*types.TxResult, error)

   // GetByRecipient returns the transactions *sent to address* may be ordered descending/ascending
   GetByRecipient(rec address, descending bool) ([]*types.TxResult, error)

   // DeleteFromHeight method added for pruning/rollback
   DeleteFromHeight(height int64) error
}

...
types/

type TxResult interface {
 Tx() Transaction        // the transaction object
 Height() int64          // height it was sent
 Index() int8            // which index it was within the block-transactions
 GetResultCode() int64   // 0 is no error, otherwise corresponds to error object code
 GetError() Error        // can be nil
 Signer() address        // get the address who signed
 Recipient() address     // can be nil
 MessageType() string    // corresponds to type of message (Ex. validator-stake, app-unjail, node-stake) etc.
}

type Transaction interface {
  // To be defined / for now it's just []byte
}

Origin Document

Thus transaction storage (indexing) is a fundamental component of any blockchain system. For M1, this is P0

Goals / Deliverables

  • Code complete implementation
  • README and CHANGELOG for the indexer

General issue checklist

  • Update the appropriate CHANGELOG
  • Update the README
  • If applicable, update the source code tree explanation
  • If applicable, add or update a state, sequence or flowchart diagram using mermaid
  • Update any relevant global documentation & references
  • Document small issues / TODOs along the way

Creator: @andrewnguyen22

@andrewnguyen22
Copy link
Contributor Author

Recommend to use a filesystem db for the implementation

Currently we are between Badger.db and Level.db

Please use one of these two options

@phthan0
Copy link
Contributor

phthan0 commented Aug 5, 2022 via email

@Olshansk Olshansk added the core Core infrastructure - protocol related label Aug 5, 2022
andrewnguyen22 pushed a commit that referenced this issue Aug 17, 2022
@Olshansk Olshansk changed the title [Indexer] Base transaction indexer [Persistence] Base transaction indexer Sep 5, 2022
andrewnguyen22 pushed a commit that referenced this issue Sep 20, 2022
## Description
Transaction storage (indexing) is a fundamental component of any blockchain system. For M1, this is P0

closes #151

## Type of change
Please mark the options that are relevant.

- [x] New feature (non-breaking change which adds functionality)

## How Has This Been Tested?
- [x] Unit tests

___REPLACE_ME_: Describe the tests and that you ran to verify your changes. If applicable, provide steps to reproduce. Bonus points for images and videos or gifs._

- [x] `make test_all`
- [x] [LocalNet](https://github.com/pokt-network/pocket/blob/main/docs/development/README.md)

## Checklist
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have tested my changes using the available tooling
andrewnguyen22 pushed a commit that referenced this issue Oct 21, 2022
## Description

Issue-#151 laid the foundation for a proper transaction indexer in isolation.

The integration of the transaction indexer into the M1 lifecycle is pending and required.

Integrate the transaction indexer into the lifecycle of M1

## Issue

Fixes #168 

## Type of change

Please mark the relevant option(s):

- [x] New feature, functionality or library
- [ ] Bug fix
- [ ] Code health or cleanup
- [ ] Major breaking change
- [ ] Documentation
- [ ] Other <!-- add details here if it a different type of change -->

## List of changes

<!-- List out all the changes made-->

- Added `TxIndexer` sub-package (previously in Utility Module)
- Added `TxIndexer` to both `PersistenceModule` and `PersistenceContext`
- Implemented `TransactionExists` and `StoreTransaction`
- Stores transactions along side blocks during `commit`
- Added current block `[]TxResult` to the module
- Moved TxIndexer package to persistence module
- Added new proto structure `DefaultTxResult`
- Integrated the `TxIndexer` into the lifecycle
  - Captured `TxResult` from each played transaction
  - Moved the storage of transactions to the Consensus Module
  - Returned the `TxResults` in the `ApplyBlock()` and `GetProposalTransactions()`
  - `AnteHandleMessage()` now returns `signer`
  - `HandleMessage()` now returns `messageType` and `recipient`
  - `ApplyTransaction()` returns `TxResult`
- Modified interface for Utility Module `ApplyBlock` and `GetProposalTransactions` to return `TxResults`
- Modified interface for Persistence Module `StoreTransaction` to store the `TxResult`
- Added shared interface `TxResult` under types.go

## Testing

- [x] `make develop_test`
- [x] [LocalNet](https://github.com/pokt-network/pocket/blob/main/docs/development/README.md) w/ all of the steps outlined in the `README`

## Required Checklist

- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have tested my changes using the available tooling
- [x] I have updated the corresponding CHANGELOG

### If Applicable Checklist
- [ ] I have updated the corresponding README(s); local and/or global
- [x] I have added tests that prove my fix is effective or that my feature works
- [ ] I have added, or updated, [mermaid.js](https://mermaid-js.github.io) diagrams in the corresponding README(s)
- [ ] I have added, or updated, documentation and [mermaid.js](https://mermaid-js.github.io) diagrams in `shared/docs/*` if I updated `shared/*`README(s)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Core infrastructure - protocol related persistence Persistence specific changes
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

3 participants