enhancement: refactor postgres integration tests #1384

shiqizng · 2022-12-19T17:53:22Z

Summary

refactor postgres_integration_test.go. replace proc() calls with hard-coded state deltas.

Test Plan

regression testing.

* Local Ledger (#1011) * integrate block processor * Local Ledger Deployment (#1013) * add simple local ledger migration * add deleted opts * fast catchup (#1023) * add fast catchup * Localledger merge (#1036) * return empty lists from fetchApplications and fetchAppLocalStates (#1010) * Update model to converge with algod (#1005) * New Feature: Adds Data Directory Support (#1012) - Updates the go-algorand submodule hash to point to rel/beta - Moves the cpu profiling file, pid file and indexer configuration file to be options of only the daemon sub-command - Changes os.Exit() to be a panic with a special handler. This is so that defer's are handled instead of being ignored. - Detects auto-loading configuration files in the data directory and issues errors if equivalent command line arguments are supplied. - Updates the README with instructions on how to use the auto-loading configuration files and the data directory. * Update mockery version Co-authored-by: erer1243 <[email protected]> Co-authored-by: AlgoStephenAkiki <[email protected]> * recovery scenario (#1024) * handle ledger recovery scenario * refactor create genesis block (#1026) * refactor create genesis block * Adds Local Ledger Readme (#1035) * Adds Local Ledger Readme Resolves #4109 Starts Readme docs * Update docs/LocalLedger.md Co-authored-by: Will Winder <[email protected]> * Update docs/LocalLedger.md Co-authored-by: Will Winder <[email protected]> * Update docs/LocalLedger.md Co-authored-by: Will Winder <[email protected]> * Removed troubleshooting section Co-authored-by: Will Winder <[email protected]> * update ledger file path and migration (#1042) * LocalLedger Refactoring + Catchpoint Service (#1049) Part 1 cleanup genesis file access. put node catchup into a function that can be swapped out with the catchup service. pass the indexer logger into the block processor. move open ledger into a util function, and move the initial state util function into a new ledger util file. add initial catchupservice implementation. move ledger init from daemon.go to constructor. Merge multiple read genesis functions. Part 2 Merge local_ledger migration package into blockprocessor. Rename Migration to Initialize Use logger in catchup service catchup Part 3 Update submodule and use NewWrappedLogger. Make util.CreateInitState private * build: merge develop into localledger/integration (#1062) * Ledger init status (#1058) * Generate an error if the catchpoint is not valid for initialization. (#1075) * Use main logger in handler and fetcher. (#1077) * Switch from fullNode catchup to catchpoint catchup service. (#1076) * Refactor daemon, add more tests (#1039) Refactors daemon cmd into separate, testable pieces. * Merge develop into localledger/integration (#1083) * Misc Local Ledger cleanup (#1086) * Update processor/blockprocessor/initialize.go Co-authored-by: Zeph Grunschlag <[email protected]> * commit * fix function call args * RFC-0001: Rfc 0001 impl (#1069) Adds an Exporter interface and a noop exporter implementation with factory methods for construction * Fix test errors * Add/fix tests * Add postgresql_exporter tests * Update config loading * Change BlockExportData to pointers * Move and rename ExportData * Add Empty func to BlockData * Add comment Co-authored-by: shiqizng <[email protected]> Co-authored-by: [email protected] <[email protected]> Co-authored-by: erer1243 <[email protected]> Co-authored-by: AlgoStephenAkiki <[email protected]> Co-authored-by: Will Winder <[email protected]> Co-authored-by: Zeph Grunschlag <[email protected]>

* Updates for changes to our plugin design based on feedback in RFCs * Merges in submodule update for test failure fixes

* Bump version to 2.13.0-rc1 * Bump version to 2.13.0 * Documentation for data directory. (#1125) Co-authored-by: algobarb <[email protected]> * Don't lookup big foreign assets. (#1141) * Revert "Bump version to 2.13.0" This reverts commit 0a8af61. * Bump version to 2.13.0 * Fix import performance test runner. (#1133) * Start on round 1 since round 0 is now computed from the genesis file. * Wait for indexer processor to exit. * Better logging for metric collection errors. * Proper support for data directory. * New test script for future release automation. * Revert "Bump version to 2.13.0" This reverts commit 7915890. * Bump version to 2.13.0 * test fixes: Submodule updates (#1144) * Update go-algorand submodule * Fix test failure due to duplicate txns * Add new ledger interface method * Enhancement: remove import validator utility and obsolete ledger for evaluator (#1146) removing a bunch of code and make the random test pass with the new ledger for evaluator * Docs: Readme update (#1149) * Update README header * Testing: Use tempdir instead of /tmp for e2elive test (#1152) * Format misc/*.py with `black` (#1153) * apply black to e2elive.py as well (#1154) * Enhancement: More information about S3 keys searched for and Dockerfile that uses submodule instead of channel (#1151) * Eric's Dockerfile improvements * Update misc/e2elive.py Co-authored-by: DevOps Service <[email protected]> Co-authored-by: Will Winder <[email protected]> Co-authored-by: algobarb <[email protected]> Co-authored-by: Barbara Poon <[email protected]> Co-authored-by: Zeph Grunschlag <[email protected]>

* Update READEME * Add initial Conduit.md * Add example exporter, exporter README.md

* importer interface | algod importer implementation * added new tests | modified importer interface * exported mockAlgodServer from test * updated algodImporter plugin * Update importers/algod/algod_importer.go Co-authored-by: Eric Warehime <[email protected]> Co-authored-by: Eric Warehime <[email protected]>

* Added Processor Added processor functionality * pr updates * Pr * PR comments * Reduce proto checks * Linting * Pr * PR comments * PR comments

* Use config to control noop exporter round start

* Initial files * Initial Conduit Binary Resolves #1165 Creates an initial framework for conduit binary production. Introduces a data directory flag as well as init and shutdown (basic) functionality. * linting * PR comments * init via importer * Passing test * e2e test passing * Modify makefile so that algorand-indexer is built * Pr changes * Log level changes * Modify bindflagset for conduit * Makefile changes * Makefile * PR updates

* Add waits/retries to algod importer

* conduit pipeline run loop implementation

* Adds unit tests Resolves #1192 Adds unit tests to the config directory * Extra assert * PR comments * Wait for round increase * Added tests * context fix

Merge indexer/develop

* Adds cpu profile and pid flag Resolves #1172 Adds the options for cpu profiling and pid flags as well as unit tests * Update conduit/pipeline.go Co-authored-by: shiqizng <[email protected]> * linting * Added pid file function * rebase Co-authored-by: shiqizng <[email protected]>

* Convert indexer daemon to conduit

* adding file exporter

* Bump version to 2.13.0-rc1 * Bump version to 2.13.0 * Documentation for data directory. (#1125) Co-authored-by: algobarb <[email protected]> * Don't lookup big foreign assets. (#1141) * Revert "Bump version to 2.13.0" This reverts commit 0a8af61. * Bump version to 2.13.0 * Fix import performance test runner. (#1133) * Start on round 1 since round 0 is now computed from the genesis file. * Wait for indexer processor to exit. * Better logging for metric collection errors. * Proper support for data directory. * New test script for future release automation. * Revert "Bump version to 2.13.0" This reverts commit 7915890. * Bump version to 2.13.0 * test fixes: Submodule updates (#1144) * Update go-algorand submodule * Fix test failure due to duplicate txns * Add new ledger interface method * Enhancement: remove import validator utility and obsolete ledger for evaluator (#1146) removing a bunch of code and make the random test pass with the new ledger for evaluator * Docs: Readme update (#1149) * Update README header * Testing: Use tempdir instead of /tmp for e2elive test (#1152) * Format misc/*.py with `black` (#1153) * apply black to e2elive.py as well (#1154) * Enhancement: More information about S3 keys searched for and Dockerfile that uses submodule instead of channel (#1151) * Eric's Dockerfile improvements * Update misc/e2elive.py * Bug-Fix: Implement BlockHdrCached + miscellany (#1162) * Enhancement: add max int64 checks (#1166) * state proofs: Indexer Support for State Proofs (#1002) Adds API support to the Indexer for State Proof Transactions and header fields. Co-authored-by: Will Winder <[email protected]> Co-authored-by: Will Winder <[email protected]> * Bump version to 2.14.0-rc1 * Stop Panics if no config is supplied (#1180) Give a default config if not supplied to stop panics. * Fix spec name collisions. (#1182) * Update go-algorand submodule to v3.9.1-beta (#1185) * Bump version to 2.14.0-rc2 * Disable deadlock detection (#1186) * Add support for new block header: TxnRoot SHA256 (#989) * Accept yaml and yml configuration files. (#1181) * Fix bug in reveals lookup (#1198) * Fix bug in reveals lookup (#1198) * Bump version to 2.14.0-rc3 * add state proof example with high reveal index - from betanet (#1199) * Devops: Bump go-algorand submodule to v3.9.2-beta (#1203) * Bump version to 2.14.0-rc4 * enhancement: Clarify REST query parameters for accounts search (#1201) * update description for /v2/accounts * cicd: add darwin arm64 support to release script (#1169) * Bump version to 2.14.0 * Downgrade mockery to prevent incorrect deprecation warning. (#1211) * Enhancement: update e2e test policy (#1197) *update e2e test policy * Fix release 2.14.0 (#1214) * Accept yaml and yml configuration files. (#1181) * Fix bug in reveals lookup (#1198) * add state proof example with high reveal index - from betanet (#1199) * enhancement: Clarify REST query parameters for accounts search (#1201) * update description for /v2/accounts * cicd: add darwin arm64 support to release script (#1169) * Downgrade mockery to prevent incorrect deprecation warning. (#1211) * Enhancement: update e2e test policy (#1197) *update e2e test policy * Update test expected value: transaction root sha256 Co-authored-by: AlgoStephenAkiki <[email protected]> Co-authored-by: Michael Diamant <[email protected]> Co-authored-by: algoidan <[email protected]> Co-authored-by: shiqizng <[email protected]> Co-authored-by: algolucky <[email protected]> Co-authored-by: Will Winder <[email protected]> Co-authored-by: DevOps Service <[email protected]> Co-authored-by: Will Winder <[email protected]> Co-authored-by: algobarb <[email protected]> Co-authored-by: Barbara Poon <[email protected]> Co-authored-by: Zeph Grunschlag <[email protected]> Co-authored-by: shiqizng <[email protected]> Co-authored-by: AlgoStephenAkiki <[email protected]> Co-authored-by: John Lee <[email protected]> Co-authored-by: Or Aharonee <[email protected]> Co-authored-by: Michael Diamant <[email protected]> Co-authored-by: algoidan <[email protected]> Co-authored-by: algolucky <[email protected]>

* Initial Filter Processor Resolves #1100 Adds dynamic filter processor and initial tests * Nits * Update processors/filterprocessor/filter_processor.go Co-authored-by: Eric Warehime <[email protected]> * Update processors/filterprocessor/filter_processor_test.go Co-authored-by: Eric Warehime <[email protected]> * Update processors/filterprocessor/filter_processor_test.go Co-authored-by: Eric Warehime <[email protected]> * PR comments * Pr comments * PR updates * PR updates * Pr updates * PR updates * PR updates * PR updates * Changed some to any and const to exact * Test failure fix * Added more unit tests * Config comments Co-authored-by: Eric Warehime <[email protected]>

* Add metrics reporting to conduit pipeline

* Changed tag format * Updated generation code * Added filtering of numerical values * PR comments * imports

* Add panic recovers Resolves #263 Adds recovery functions in main loop of conduit as well as in a go-routine in the start function of the pipeline * helper function

Resolves #1245 Explicitly spell out empty data directory as well as condensed error messages when running conduit

winder · 2023-01-13T13:45:18Z

types/types.go

+// LegercoreValidatedBlock for serialization
+type LegercoreValidatedBlock struct {
+	Blk   bookkeeping.Block
+	Delta ledgercore.StateDelta
+}


So we'll run these vb files through a "convert" function to update tests when we change the interfaces from go-algorand types?

I was thinking that vb could be deserialized to types.ValidatedBlock

winder · 2023-01-13T13:48:05Z

idb/postgres/postgres_integration_test.go

-	createAsset := test.MakeAssetConfigTxn(0, total, uint64(6), false, "mcn", "my coin", "http://antarctica.com", test.AccountD)
-	optInA1 := test.MakeAssetOptInTxn(assetid, test.AccountA)
-	optInA2 := test.MakeAssetOptInTxn(assetid, test.AccountA)
-	fundA := test.MakeAssetTransferTxn(assetid, amt, test.AccountD, test.AccountA, basics.Address{})
-	optInB := test.MakeAssetOptInTxn(assetid, test.AccountB)
-	optInC := test.MakeAssetOptInTxn(assetid, test.AccountC)
-	closeA := test.MakeAssetTransferTxn(assetid, 1000, test.AccountA, test.AccountB, test.AccountC)
-	payMain := test.MakeAssetTransferTxn(assetid, amt, test.AccountD, test.AccountA, basics.Address{})


I'm curious what others think here.

Should we keep this code commented out to serve as documentation for whats inside the validated blocks?

Yeah I think it's fine to keep these as comments for now. Will they be restored once we fully migrate?

I don't think we'll restore these until we have a way to create the state deltas for tests.

winder · 2023-01-13T13:55:01Z

idb/postgres/postgres_integration_test.go

+	block := ledgercore.MakeValidatedBlock(vb.Blk, vb.Delta)

 	//////////
 	// When // We commit the round accounting to the database.
 	//////////
-	err = proc(&rpcs.EncodedBlockCert{Block: block})
+	err = db.AddBlock(&block)


nit: consider moving these lines next to each other since they'll need to be updated together during the interface migration.

winder

Looks great. Just had some minor comments.

winder

LGTM

Eric-Warehime and others added 30 commits July 21, 2022 08:18

Add all PluginTypes (#1136)

e2c50a0

Conduit: Rfc updates (#1139)

2c945b4

* Updates for changes to our plugin design based on feedback in RFCs * Merges in submodule update for test failure fixes

docs: Plugin docs (#1150)

f2b422f

* Update READEME * Add initial Conduit.md * Add example exporter, exporter README.md

Create RFC for processor (#1140)

3d2720c

* Added Processor Added processor functionality * pr updates * Pr * PR comments * Reduce proto checks * Linting * Pr * PR comments * PR comments

Use config to control noop exporter round start (#1176)

34986db

* Use config to control noop exporter round start

Set pgsql exporter round from db on init (#1177)

959c576

Add waits/retries to algod importer (#1178)

a0b6ecc

* Add waits/retries to algod importer

conduit: pipeline run loop implementation (#1183)

9cc3d36

* conduit pipeline run loop implementation

Adds unit tests (#1202)

adbae7e

* Adds unit tests Resolves #1192 Adds unit tests to the config directory * Extra assert * PR comments * Wait for round increase * Added tests * context fix

succeed close calls when init fails (#1204)

bf8b441

silence usage data on error (#1205)

13aaba6

conduit: Conduit merge upstream (#1206)

d0046cb

Merge indexer/develop

conduit: Convert daemon to conduit pipeline (#1208)

88fca5c

* Convert indexer daemon to conduit

file exporter (#1213)

e8fa891

* adding file exporter

Conduit: e2e tests (#1221)

f93bd51

Add wait-for-block to block-gen (#1229)

756be9b

Conduit: prometheus metrics (#1230)

7cf43e8

* Add metrics reporting to conduit pipeline

Add filter_processor to all imports for processors (#1233)

a2e7bdc

File Exporter: save metadata more often (#1232)

ce54ff3

Conduit: init subcommand (#1235)

c809d35

Changed tag format (#1238)

3bc8f52

* Changed tag format * Updated generation code * Added filtering of numerical values * PR comments * imports

Add panic recovers (#1251)

a6b7473

* Add panic recovers Resolves #263 Adds recovery functions in main loop of conduit as well as in a go-routine in the start function of the pipeline * helper function

Condensed Duplicate logger output (#1256)

eaab3a1

Resolves #1245 Explicitly spell out empty data directory as well as condensed error messages when running conduit

shiqizng added Enhancement New feature or request Team Lamprey labels Jan 11, 2023

shiqizng changed the title ~~[WIP] refactor postgres integration tests~~ [WIP] enhancement: refactor postgres integration tests Jan 11, 2023

shiqizng added 12 commits January 11, 2023 15:23

update tests

a37235c

remove misc files

95b4473

revert proc change

df41cb3

update test

65f2461

refactor

b600220

make lint

41cde55

update tests

6bd98a0

update tests

8a36240

updates

4826612

reorder imports

f042071

delete file

a2c0116

more updates

c863459

shiqizng changed the title ~~[WIP] enhancement: refactor postgres integration tests~~ enhancement: refactor postgres integration tests Jan 12, 2023

shiqizng requested review from winder, Eric-Warehime, AlgoStephenAkiki and mciccarello January 12, 2023 21:31

winder reviewed Jan 13, 2023

View reviewed changes

shiqizng added 2 commits January 13, 2023 09:55

update test descriptions

ab41ef5

address pr comments

07a66f9

shiqizng requested a review from winder January 17, 2023 16:13

winder approved these changes Jan 18, 2023

View reviewed changes

shiqizng merged commit d6ae8c3 into develop Jan 18, 2023

shiqizng deleted the shiqi/inttests branch January 18, 2023 14:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enhancement: refactor postgres integration tests #1384

enhancement: refactor postgres integration tests #1384

shiqizng commented Dec 19, 2022

winder Jan 13, 2023

shiqizng Jan 13, 2023

winder Jan 13, 2023 •

edited

Loading

Eric-Warehime Jan 13, 2023

shiqizng Jan 13, 2023

winder Jan 13, 2023

winder left a comment

winder left a comment

enhancement: refactor postgres integration tests #1384

enhancement: refactor postgres integration tests #1384

Conversation

shiqizng commented Dec 19, 2022

Summary

Test Plan

winder Jan 13, 2023

Choose a reason for hiding this comment

shiqizng Jan 13, 2023

Choose a reason for hiding this comment

winder Jan 13, 2023 • edited Loading

Choose a reason for hiding this comment

Eric-Warehime Jan 13, 2023

Choose a reason for hiding this comment

shiqizng Jan 13, 2023

Choose a reason for hiding this comment

winder Jan 13, 2023

Choose a reason for hiding this comment

winder left a comment

Choose a reason for hiding this comment

winder left a comment

Choose a reason for hiding this comment

winder Jan 13, 2023 •

edited

Loading