state proofs: Initial commit for stateproofs model #1002

Eric-Warehime · 2022-05-17T05:44:20Z

Summary

I'm opening this PR to integrate the indexer models w/ the new changes being worked on for state proofs. Much of the implementation code for that is located in a branch in go-algorand at the moment (https://github.com/algorand/go-algorand/tree/feature/stateproofs) so this PR will probably have to remain open until that is merged into go-algorand master.

codecov · 2022-05-17T05:52:55Z

Codecov Report

Merging #1002 (8079778) into develop (8c49d86) will increase coverage by 0.51%.
The diff coverage is 99.20%.

@@             Coverage Diff             @@
##           develop    #1002      +/-   ##
===========================================
+ Coverage    60.28%   60.80%   +0.51%     
===========================================
  Files           51       51              
  Lines         8359     8472     +113     
===========================================
+ Hits          5039     5151     +112     
  Misses        2862     2862              
- Partials       458      459       +1

Impacted Files	Coverage Δ
api/generated/common/routes.go	`46.51% <ø> (ø)`
api/generated/v2/routes.go	`17.76% <ø> (ø)`
idb/postgres/internal/encoding/types.go	`66.66% <ø> (ø)`
idb/txn_type_enum.go	`100.00% <ø> (ø)`
api/handlers.go	`75.15% <96.87%> (+0.41%)`	⬆️
api/converter_utils.go	`92.60% <100.00%> (+1.36%)`	⬆️
idb/postgres/internal/writer/writer.go	`87.22% <100.00%> (ø)`

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

api/indexer.oas2.json

winder · 2022-08-09T14:55:36Z

api/converter_utils.go

+	case protocol.StateProofTx:
+		partPath := make([]generated.GenericDigest, len(stxn.Txn.StateProof.PartProofs.Path))
+		for idx, part := range stxn.Txn.StateProof.PartProofs.Path {
+			partPath[idx] = generated.GenericDigest(part)


I believe you need to make a copy of part because the pointer is reused by the for loop.

Here's an example where we do something similar: https://github.com/algorand/indexer/pull/575/files

winder · 2022-08-09T16:51:46Z

idb/postgres/internal/writer/writer.go

@@ -3,6 +3,7 @@ package writer
 import (
 	"context"
 	"fmt"
+	"github.com/algorand/go-algorand/protocol"


nit: fix import grouping

idb/postgres/internal/writer/writer.go

winder · 2022-08-09T17:01:53Z

idb/txn_type_enum.go

@@ -19,6 +19,7 @@ const (
 	TypeEnumAssetTransfer
 	TypeEnumAssetFreeze
 	TypeEnumApplication
+	TypeStateProof


Let's stick with the TypeEnum prefix to keep consistent with the others, or update all of them and perhaps drop Type as well.

winder · 2022-08-09T17:03:13Z

misc/e2elive.py

@@ -57,7 +57,7 @@ def main():
        from botocore.config import Config
        from botocore import UNSIGNED
        s3 = boto3.client('s3', config=Config(signature_version=UNSIGNED))
-        tarname = 'net_done.tar.bz2'
+        tarname = 'feature-stateproofs.tar.bz2'


stateproofs is merged! We can change this:

Suggested change

tarname = 'feature-stateproofs.tar.bz2'

tarname = 'rel-nightly.tar.bz2'

id-ms

Overall looks good
thanks!
There is only one issue which I wasn't sure

api/indexer.oas2.json

id-ms · 2022-08-10T17:33:20Z

api/converter_utils.go

+				},
+			}
+		}
+		proof := generated.StateProof{


I think we should also copy PositionsToReveal slice

@Eric-Warehime You can use byteSlicePtr to create a copy

misc/e2elive.py

Co-authored-by: Will Winder <[email protected]>

* Bump version to 2.13.0-rc1 * Bump version to 2.13.0 * Documentation for data directory. (#1125) Co-authored-by: algobarb <[email protected]> * Don't lookup big foreign assets. (#1141) * Revert "Bump version to 2.13.0" This reverts commit 0a8af61. * Bump version to 2.13.0 * Fix import performance test runner. (#1133) * Start on round 1 since round 0 is now computed from the genesis file. * Wait for indexer processor to exit. * Better logging for metric collection errors. * Proper support for data directory. * New test script for future release automation. * Revert "Bump version to 2.13.0" This reverts commit 7915890. * Bump version to 2.13.0 * test fixes: Submodule updates (#1144) * Update go-algorand submodule * Fix test failure due to duplicate txns * Add new ledger interface method * Enhancement: remove import validator utility and obsolete ledger for evaluator (#1146) removing a bunch of code and make the random test pass with the new ledger for evaluator * Docs: Readme update (#1149) * Update README header * Testing: Use tempdir instead of /tmp for e2elive test (#1152) * Format misc/*.py with `black` (#1153) * apply black to e2elive.py as well (#1154) * Enhancement: More information about S3 keys searched for and Dockerfile that uses submodule instead of channel (#1151) * Eric's Dockerfile improvements * Update misc/e2elive.py * Bug-Fix: Implement BlockHdrCached + miscellany (#1162) * Enhancement: add max int64 checks (#1166) * state proofs: Indexer Support for State Proofs (#1002) Adds API support to the Indexer for State Proof Transactions and header fields. Co-authored-by: Will Winder <[email protected]> Co-authored-by: Will Winder <[email protected]> * Bump version to 2.14.0-rc1 * Stop Panics if no config is supplied (#1180) Give a default config if not supplied to stop panics. * Fix spec name collisions. (#1182) * Update go-algorand submodule to v3.9.1-beta (#1185) * Bump version to 2.14.0-rc2 * Disable deadlock detection (#1186) * Add support for new block header: TxnRoot SHA256 (#989) * Accept yaml and yml configuration files. (#1181) * Fix bug in reveals lookup (#1198) * Fix bug in reveals lookup (#1198) * Bump version to 2.14.0-rc3 * add state proof example with high reveal index - from betanet (#1199) * Devops: Bump go-algorand submodule to v3.9.2-beta (#1203) * Bump version to 2.14.0-rc4 * enhancement: Clarify REST query parameters for accounts search (#1201) * update description for /v2/accounts * cicd: add darwin arm64 support to release script (#1169) * Bump version to 2.14.0 * Downgrade mockery to prevent incorrect deprecation warning. (#1211) * Enhancement: update e2e test policy (#1197) *update e2e test policy * Fix release 2.14.0 (#1214) * Accept yaml and yml configuration files. (#1181) * Fix bug in reveals lookup (#1198) * add state proof example with high reveal index - from betanet (#1199) * enhancement: Clarify REST query parameters for accounts search (#1201) * update description for /v2/accounts * cicd: add darwin arm64 support to release script (#1169) * Downgrade mockery to prevent incorrect deprecation warning. (#1211) * Enhancement: update e2e test policy (#1197) *update e2e test policy * Update test expected value: transaction root sha256 Co-authored-by: AlgoStephenAkiki <[email protected]> Co-authored-by: Michael Diamant <[email protected]> Co-authored-by: algoidan <[email protected]> Co-authored-by: shiqizng <[email protected]> Co-authored-by: algolucky <[email protected]> Co-authored-by: Will Winder <[email protected]> Co-authored-by: DevOps Service <[email protected]> Co-authored-by: Will Winder <[email protected]> Co-authored-by: algobarb <[email protected]> Co-authored-by: Barbara Poon <[email protected]> Co-authored-by: Zeph Grunschlag <[email protected]> Co-authored-by: shiqizng <[email protected]> Co-authored-by: AlgoStephenAkiki <[email protected]> Co-authored-by: John Lee <[email protected]> Co-authored-by: Or Aharonee <[email protected]> Co-authored-by: Michael Diamant <[email protected]> Co-authored-by: algoidan <[email protected]> Co-authored-by: algolucky <[email protected]>

Eric-Warehime self-assigned this May 17, 2022

Eric-Warehime added the New Feature label May 17, 2022

Eric-Warehime changed the title ~~Initial commit for stateproofs model~~ state proofs: Initial commit for stateproofs model May 17, 2022

winder reviewed May 18, 2022

View reviewed changes

api/indexer.oas2.json Show resolved Hide resolved

Eric-Warehime requested review from Aharonee and id-ms May 18, 2022 14:20

Eric-Warehime requested a review from winder June 8, 2022 20:36

Eric-Warehime marked this pull request as ready for review June 8, 2022 20:36

Eric-Warehime force-pushed the state-proofs branch from 7e14083 to aedb5b1 Compare July 28, 2022 21:28

tzaffi mentioned this pull request Jul 29, 2022

Enhancement: More information about S3 keys searched for and Dockerfile that uses submodule instead of channel #1151

Merged

winder mentioned this pull request Aug 2, 2022

Add support for new block header: TxnRoot SHA256 #989

Merged

winder reviewed Aug 9, 2022

View reviewed changes

idb/postgres/internal/writer/writer.go Show resolved Hide resolved

winder reviewed Aug 9, 2022

View reviewed changes

Eric-Warehime added 13 commits August 10, 2022 09:30

Initial commit for stateproofs model

c8935fc

Update for state proof rename

0f344b1

Update args/types to match feature branch

b989b39

Fix StateProofID types in tests

4a19fa3

Update submodule

34cacff

go mod tidy

be13a08

Switch e2e test tar

c824d74

Add BlockHdrCached impl

2491b91

Fix BlkHeaderCached

8ec6f87

Update postgres writer to ignore signature on stpf txns

ae33a55

Add state proof txn type

938465c

Update submodule

0fae52a

Update submodule

f804ca7

Eric-Warehime added 8 commits August 10, 2022 09:32

Add state proof txn api test

4d70ce9

Update model, add converter

4c5c48f

Make state proof responses deterministic

e3ac7c7

Fix range loop pointer bugs

b9e7a96

Add block header stpf changes

819ab1f

Pr comment changes

9f29afb

Update submodule

a1cffbc

Revise VotersTotalWeight to OnlineTotalWeight

c409cb9

Eric-Warehime force-pushed the state-proofs branch from dda2c18 to c409cb9 Compare August 10, 2022 16:36

Fix lint error

90b47a2

Eric-Warehime requested a review from winder August 10, 2022 17:08

id-ms reviewed Aug 10, 2022

View reviewed changes

PR comment updates

81a1bd1

Eric-Warehime requested a review from id-ms August 10, 2022 18:16

winder reviewed Aug 10, 2022

View reviewed changes

misc/e2elive.py Outdated Show resolved Hide resolved

Update misc/e2elive.py

8079778

Co-authored-by: Will Winder <[email protected]>

Eric-Warehime requested a review from winder August 10, 2022 19:07

winder approved these changes Aug 10, 2022

View reviewed changes

Eric-Warehime merged commit dc8f994 into algorand:develop Aug 10, 2022

Eric-Warehime deleted the state-proofs branch August 10, 2022 19:42

This was referenced Aug 10, 2022

FOR REVIEW ONLY: indexer 2.14.0 into master #1170

Merged

FOR REVIEW ONLY: indexer 2.14.0 into develop #1171

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

state proofs: Initial commit for stateproofs model #1002

state proofs: Initial commit for stateproofs model #1002

Eric-Warehime commented May 17, 2022

codecov bot commented May 17, 2022 •

edited

Loading

winder Aug 9, 2022

winder Aug 9, 2022

winder Aug 9, 2022

winder Aug 9, 2022

winder Aug 9, 2022

id-ms left a comment

id-ms Aug 10, 2022

winder Aug 10, 2022

	tarname = 'feature-stateproofs.tar.bz2'
	tarname = 'rel-nightly.tar.bz2'

state proofs: Initial commit for stateproofs model #1002

state proofs: Initial commit for stateproofs model #1002

Conversation

Eric-Warehime commented May 17, 2022

Summary

codecov bot commented May 17, 2022 • edited Loading

Codecov Report

winder Aug 9, 2022

Choose a reason for hiding this comment

winder Aug 9, 2022

Choose a reason for hiding this comment

winder Aug 9, 2022

Choose a reason for hiding this comment

winder Aug 9, 2022

Choose a reason for hiding this comment

winder Aug 9, 2022

Choose a reason for hiding this comment

id-ms left a comment

Choose a reason for hiding this comment

id-ms Aug 10, 2022

Choose a reason for hiding this comment

winder Aug 10, 2022

Choose a reason for hiding this comment

codecov bot commented May 17, 2022 •

edited

Loading