Create a Lily node by embedding lotus into visor to observe tipset apply and revert events #408

frrist · 2021-03-03T22:48:52Z

What

The PR imports and embeds the lotus daemon into Visro; currently, this node is called Lily.
Changes required to lotus live here: filecoin-project/lotus#5693

Usage

make (may need to run git submodule update --init --recursive first)
./visor lily daemon --import-snapshot minimal_finality_stateroots_517061_2021-02-20_11-00-00.car
make dockerup to create a database
./visor lily watch start will instruct lily to start watching and processing the Filecoin blockchain and indexing it to the database.

-- will allow lotus binary to be built in following commit

- allows a lotus node to be instanciated and for visor to observe tipsets applied and reverted by the lotus node.

frrist · 2021-03-08T22:29:42Z

node/impl/lily.go

+
+	obs := observer.NewIndexingTipSetObserver(indexer, tsCache)
+
+	if err := m.Events.Observe(obs); err != nil {


I think we can avoid this dependency in lotus if we go back to using the chain notify API call that is already present in the impl.FullNodeAPI.

iand · 2021-03-09T16:44:24Z

node/api/apistruct/struct.go

@@ -0,0 +1,26 @@
+package apistruct


Does this need to be a separate package?

It does not -- I had everything broken into packages like this to make it easier to mimic lotus's layout. I have sense created a "lily" lens and moved the LilyNode API and struct into it.

iand · 2021-03-09T16:45:10Z

node/cli/daemon.go

@@ -0,0 +1,282 @@
+package cli


Can we move this to our commands package?

iand · 2021-03-09T16:47:06Z

node/cli/daemon.go

+	return opts
+}
+
+func ImportChain(ctx context.Context, r repo.Repo, fname string, snapshot bool) (err error) {


Is this just a copy of what is in Lotus? We talked about exporting it in Lotus but copying it here means one less change on the Lotus side which is good.

We should find a better package for this function, although it's not urgent to do that

Yes, a copy from lotus; I agree copying this code isn't ideal but I would prefer to punt on extraction in lotus since this copy is low risk.

iand · 2021-03-09T16:50:01Z

node/main.go

+		Flags: []cli.Flag{
+			&cli.StringFlag{
+				Name:    "repo",
+				EnvVars: []string{"SENTINEL_LOTUS_PATH"},


SENTINEL_REPO perhaps? Let's start being consistent so a flag called xxx has an env variable called SENTINEL_XXX

iand · 2021-03-09T16:59:15Z

node/api/apistruct/struct.go

+
+var log = logging.Logger("lily-api")
+
+type LilyNodeStruct struct {


Since this is the start of the new JSON-RPC API that visor will expose to allow the daemon to be controlled we should name it more concretely, e.g. just API.

Would be nice if we could stop using the word node in other places where we mean lens.Interface and we should try to stop using it here too.

Dropped the "node" word and moved this all to the lens package

iand · 2021-03-09T16:59:53Z

node/api/apistruct/struct.go

+var log = logging.Logger("lily-api")
+
+type LilyNodeStruct struct {
+	lotusstruct.FullNodeStruct


We don't need all of this. What is the minimum we need?

We need the following API's from lotus:

common.CommonAPI for the RPC and authentication/permissions.

full.ChainAPI for chain event notification and chain inspection.

full.StateAPI for actor state extraction and inspection.

I have redeclared the LilyAPI to require only these.

frrist · 2021-03-09T23:29:07Z

commands/util/rpc.go

@@ -0,0 +1,148 @@
+package util


mostly copy-paste from lotus to get the injection working.

frrist · 2021-03-09T23:30:11Z

main.go

+				Name:        "lens",
+				EnvVars:     []string{"VISOR_LENS"},
+				Value:       "lotus",
+				Destination: &commands.VisorCmdFlags.Lens,


Destination looks like a cleaner way (imo) to pass values to subcommands, e.g. lily watch start

Yeah maybe. I'm not 100% sold on it but willing to try it out.

codecov-io · 2021-03-10T23:15:04Z

Codecov Report

Merging #408 (79231aa) into lily (8f367a3) will decrease coverage by 0.2%.
The diff coverage is 0.0%.

@@           Coverage Diff           @@
##            lily    #408     +/-   ##
=======================================
- Coverage   43.9%   43.6%   -0.3%     
=======================================
  Files         28      29      +1     
  Lines       2246    2261     +15     
=======================================
  Hits         986     986             
- Misses      1076    1091     +15     
  Partials     184     184

iand

I'm happy to merge this to lily and use it as a basis for the next steps.

iand · 2021-03-11T14:22:32Z

chain/indexing.go

+	return nil
+}
+
+var _ events.TipSetObserver = (*IndexingTipSetObserver)(nil)


This line is the only reason for importing the events package. I don't think we need it since we can't compile without it satisfying the interface anyway

iand · 2021-03-11T14:26:29Z

chain/indexing.go

+	"github.com/filecoin-project/lotus/chain/types"
+)
+
+func NewIndexingTipSetObserver(obs TipSetObserver, cache *TipSetCache) *IndexingTipSetObserver {


As you mentioned before, we could simply revert to using ChainNotify and then this type is redundant with Watcher. For now let's keep it and work on refactoring to a more optimized lens interface that could work by calling ChainNotiify or could hook into events directly.

iand · 2021-03-11T14:28:04Z

commands/lily/watch.go

+	After:  destroy,
+	Flags: []cli.Flag{
+		&cli.IntFlag{
+			Name:        "indexhead-confidence",


We'll refine the names of the params later

iand · 2021-03-11T14:30:27Z

main.go

+				Name:        "lens",
+				EnvVars:     []string{"VISOR_LENS"},
+				Value:       "lotus",
+				Destination: &commands.VisorCmdFlags.Lens,


Yeah maybe. I'm not 100% sold on it but willing to try it out.

…ply and revert events (#408)

@iand

…ply and revert events (#408) chore: reuse watcher in lily watch command (#422) Merge master into lily (#425) * fix: ignore false positive gosec failure in wait package (#421) * fix: embed genesis block in executable (#420) * feat: add benchmarking of vectors (#423) * feat: add benchmarking of vectors * feat: add benchmarking of vectors * Use nicer test and benchmark names fix(linter): make deps before linting in CI (#437) merge master into lily (#432) * fix(indexer): only close TIpSetIndexer in walk and watch * chore: increase linter timeout in ci scheduler job control + watch, walk, and job cli (#414) merge master into lily (#442) * fix: ignore false positive gosec failure in wait package (#421) * fix: embed genesis block in executable (#420) * feat: add benchmarking of vectors (#423) * feat: add benchmarking of vectors * feat: add benchmarking of vectors * Use nicer test and benchmark names * Chore/sqlotus dag prefetch option (#430) * Disable preloader by default, toggle with LOTUS_CHAINSTORE_PRELOAD_RECENTS * Switch to a cli string as suggested by @iand * Remove one more inefficiency for the R/O lens filecoin-project/go-bs-postgres-chainnotated@b8b5d500e368d2ed * Forgotten option/helptext spec * Add support for static Get() prefetch * close TipSetObs in walker and watcher * fix(indexer): only close TIpSetIndexer in walk and watch - subsequent calls to close will hang forever * chore: increase linter timout in ci * chore: update for Lotus 1.5.3 (#439) * chore: update for Lotus 1.5.3 * fixup! chore: update for Lotus 1.5.3 * Update statediff version Co-authored-by: frrist <[email protected]> Co-authored-by: Ian Davis <[email protected]> Co-authored-by: Peter Rabbitson <[email protected]> chore: remove confidence argument from lily walk command (#444) chore: reorganise visor commands (#445) feat: add storage configuration to daemon (#446) * feat: add storage configuration to daemon * Start wiring up storage to config * Finish wiring up storage config * check error return when ensuring config * Close config file on error feat: Add --api parameter to daemon client commands (#450) * feat: Add --api parameter to daemon client commands * Make api default value a valid multiaddr * Add api token parameter Merge master into lily (#463) * fix: ignore false positive gosec failure in wait package (#421) * fix: embed genesis block in executable (#420) * feat: add benchmarking of vectors (#423) * feat: add benchmarking of vectors * feat: add benchmarking of vectors * Use nicer test and benchmark names * Chore/sqlotus dag prefetch option (#430) * Disable preloader by default, toggle with LOTUS_CHAINSTORE_PRELOAD_RECENTS * Switch to a cli string as suggested by @iand * Remove one more inefficiency for the R/O lens filecoin-project/go-bs-postgres-chainnotated@b8b5d500e368d2ed * Forgotten option/helptext spec * Add support for static Get() prefetch * close TipSetObs in walker and watcher * fix(indexer): only close TIpSetIndexer in walk and watch - subsequent calls to close will hang forever * chore: increase linter timout in ci * chore: update for Lotus 1.5.3 (#439) * chore: update for Lotus 1.5.3 * fixup! chore: update for Lotus 1.5.3 * Update statediff version Co-authored-by: frrist <[email protected]> * Release notes for v0.5.5 (#440) * chore: update lotus to 1.6.0 (#443) * chore: upgrade lotus dependency to 1.6.0-rc1 (#447) * chore: update lotus to 1.6.0 (#455) * Release notes for v0.5.6 (#456) * fix: ensure persistence semaphore channel is drained on close (#457) * fix: ensure persistence semaphore channel is drained on close * Add hidden window flag for testing * Reduce actor happy path logging * Make sure actor errors reported are appended to correct list * Release notes for v0.5.7 (#458) * Update to latest lotus commit * Add missing support/tools/go.sum * Fix merge bugs in makefile * Fix merge bugs in makefile Co-authored-by: Peter Rabbitson <[email protected]> Co-authored-by: frrist <[email protected]> feat: clean up command line flags for single-run jobs (#464) * feat: clean up command line flags for single-run jobs * Fix dockerfile Co-authored-by: frrist <[email protected]> polish(lily): add job name flag (#467) - closes #452 polish(job): print job type, params, and errors in list (#468) - closes #451 fix: ensure event channel is initialised before watch starts (#471) chore: merge master into lily branch (#472) * fix: ignore false positive gosec failure in wait package (#421) * fix: embed genesis block in executable (#420) * feat: add benchmarking of vectors (#423) * feat: add benchmarking of vectors * feat: add benchmarking of vectors * Use nicer test and benchmark names * Chore/sqlotus dag prefetch option (#430) * Disable preloader by default, toggle with LOTUS_CHAINSTORE_PRELOAD_RECENTS * Switch to a cli string as suggested by @iand * Remove one more inefficiency for the R/O lens filecoin-project/go-bs-postgres-chainnotated@b8b5d500e368d2ed * Forgotten option/helptext spec * Add support for static Get() prefetch * close TipSetObs in walker and watcher * fix(indexer): only close TIpSetIndexer in walk and watch - subsequent calls to close will hang forever * chore: increase linter timout in ci * chore: update for Lotus 1.5.3 (#439) * chore: update for Lotus 1.5.3 * fixup! chore: update for Lotus 1.5.3 * Update statediff version Co-authored-by: frrist <[email protected]> * Release notes for v0.5.5 (#440) * chore: update lotus to 1.6.0 (#443) * chore: upgrade lotus dependency to 1.6.0-rc1 (#447) * chore: update lotus to 1.6.0 (#455) * Release notes for v0.5.6 (#456) * fix: ensure persistence semaphore channel is drained on close (#457) * fix: ensure persistence semaphore channel is drained on close * Add hidden window flag for testing * Reduce actor happy path logging * Make sure actor errors reported are appended to correct list * Release notes for v0.5.7 (#458) * fix(tasks): use correct ActorInfo for actor state extraction (#449) * fix(tasks): use corret actor info for actor state extraction - fixes #273 * fix(lens): remove OptimizedStateGetActorWithFallback - fixes #405 * fix(market-task): fix market actor task and gen vector - fixes #465 * polish(vector): support msapproval model validation * test(vector): generate new test vectors * ci: increase instance size due to test vectors * Add MinerSectorInfo for extended sectors (#448) Fixes #441. Extended sectors have new deadlines but this is not recorded anywhere. * Fix makefile conflicts Co-authored-by: Peter Rabbitson <[email protected]> Co-authored-by: frrist <[email protected]> Co-authored-by: Hector Sanjuan <[email protected]>

@iand

* Create a Lily node by embedding lotus into visor to observe tipset apply and revert events (#408) * chore: reuse watcher in lily watch command (#422) * Merge master into lily (#425) * fix: ignore false positive gosec failure in wait package (#421) * fix: embed genesis block in executable (#420) * feat: add benchmarking of vectors (#423) * feat: add benchmarking of vectors * feat: add benchmarking of vectors * Use nicer test and benchmark names * fix(linter): make deps before linting in CI (#437) * merge master into lily (#432) * fix(indexer): only close TIpSetIndexer in walk and watch * chore: increase linter timeout in ci * scheduler job control + watch, walk, and job cli (#414) * merge master into lily (#442) * fix: ignore false positive gosec failure in wait package (#421) * fix: embed genesis block in executable (#420) * feat: add benchmarking of vectors (#423) * feat: add benchmarking of vectors * feat: add benchmarking of vectors * Use nicer test and benchmark names * Chore/sqlotus dag prefetch option (#430) * Disable preloader by default, toggle with LOTUS_CHAINSTORE_PRELOAD_RECENTS * Switch to a cli string as suggested by @iand * Remove one more inefficiency for the R/O lens filecoin-project/go-bs-postgres-chainnotated@b8b5d500e368d2ed * Forgotten option/helptext spec * Add support for static Get() prefetch * close TipSetObs in walker and watcher * fix(indexer): only close TIpSetIndexer in walk and watch - subsequent calls to close will hang forever * chore: increase linter timout in ci * chore: update for Lotus 1.5.3 (#439) * chore: update for Lotus 1.5.3 * fixup! chore: update for Lotus 1.5.3 * Update statediff version Co-authored-by: frrist <[email protected]> Co-authored-by: Ian Davis <[email protected]> Co-authored-by: Peter Rabbitson <[email protected]> * chore: remove confidence argument from lily walk command (#444) * chore: reorganise visor commands (#445) * feat: add storage configuration to daemon (#446) * feat: add storage configuration to daemon * Start wiring up storage to config * Finish wiring up storage config * check error return when ensuring config * Close config file on error * feat: Add --api parameter to daemon client commands (#450) * feat: Add --api parameter to daemon client commands * Make api default value a valid multiaddr * Add api token parameter * Merge master into lily (#463) * fix: ignore false positive gosec failure in wait package (#421) * fix: embed genesis block in executable (#420) * feat: add benchmarking of vectors (#423) * feat: add benchmarking of vectors * feat: add benchmarking of vectors * Use nicer test and benchmark names * Chore/sqlotus dag prefetch option (#430) * Disable preloader by default, toggle with LOTUS_CHAINSTORE_PRELOAD_RECENTS * Switch to a cli string as suggested by @iand * Remove one more inefficiency for the R/O lens filecoin-project/go-bs-postgres-chainnotated@b8b5d500e368d2ed * Forgotten option/helptext spec * Add support for static Get() prefetch * close TipSetObs in walker and watcher * fix(indexer): only close TIpSetIndexer in walk and watch - subsequent calls to close will hang forever * chore: increase linter timout in ci * chore: update for Lotus 1.5.3 (#439) * chore: update for Lotus 1.5.3 * fixup! chore: update for Lotus 1.5.3 * Update statediff version Co-authored-by: frrist <[email protected]> * Release notes for v0.5.5 (#440) * chore: update lotus to 1.6.0 (#443) * chore: upgrade lotus dependency to 1.6.0-rc1 (#447) * chore: update lotus to 1.6.0 (#455) * Release notes for v0.5.6 (#456) * fix: ensure persistence semaphore channel is drained on close (#457) * fix: ensure persistence semaphore channel is drained on close * Add hidden window flag for testing * Reduce actor happy path logging * Make sure actor errors reported are appended to correct list * Release notes for v0.5.7 (#458) * Update to latest lotus commit * Add missing support/tools/go.sum * Fix merge bugs in makefile * Fix merge bugs in makefile Co-authored-by: Peter Rabbitson <[email protected]> Co-authored-by: frrist <[email protected]> * feat: clean up command line flags for single-run jobs (#464) * feat: clean up command line flags for single-run jobs * Fix dockerfile Co-authored-by: frrist <[email protected]> * polish(lily): add job name flag (#467) - closes #452 * polish(job): print job type, params, and errors in list (#468) - closes #451 * fix: ensure event channel is initialised before watch starts (#471) * chore: merge master into lily branch (#472) * fix: ignore false positive gosec failure in wait package (#421) * fix: embed genesis block in executable (#420) * feat: add benchmarking of vectors (#423) * feat: add benchmarking of vectors * feat: add benchmarking of vectors * Use nicer test and benchmark names * Chore/sqlotus dag prefetch option (#430) * Disable preloader by default, toggle with LOTUS_CHAINSTORE_PRELOAD_RECENTS * Switch to a cli string as suggested by @iand * Remove one more inefficiency for the R/O lens filecoin-project/go-bs-postgres-chainnotated@b8b5d500e368d2ed * Forgotten option/helptext spec * Add support for static Get() prefetch * close TipSetObs in walker and watcher * fix(indexer): only close TIpSetIndexer in walk and watch - subsequent calls to close will hang forever * chore: increase linter timout in ci * chore: update for Lotus 1.5.3 (#439) * chore: update for Lotus 1.5.3 * fixup! chore: update for Lotus 1.5.3 * Update statediff version Co-authored-by: frrist <[email protected]> * Release notes for v0.5.5 (#440) * chore: update lotus to 1.6.0 (#443) * chore: upgrade lotus dependency to 1.6.0-rc1 (#447) * chore: update lotus to 1.6.0 (#455) * Release notes for v0.5.6 (#456) * fix: ensure persistence semaphore channel is drained on close (#457) * fix: ensure persistence semaphore channel is drained on close * Add hidden window flag for testing * Reduce actor happy path logging * Make sure actor errors reported are appended to correct list * Release notes for v0.5.7 (#458) * fix(tasks): use correct ActorInfo for actor state extraction (#449) * fix(tasks): use corret actor info for actor state extraction - fixes #273 * fix(lens): remove OptimizedStateGetActorWithFallback - fixes #405 * fix(market-task): fix market actor task and gen vector - fixes #465 * polish(vector): support msapproval model validation * test(vector): generate new test vectors * ci: increase instance size due to test vectors * Add MinerSectorInfo for extended sectors (#448) Fixes #441. Extended sectors have new deadlines but this is not recorded anywhere. * Fix makefile conflicts Co-authored-by: Peter Rabbitson <[email protected]> Co-authored-by: frrist <[email protected]> Co-authored-by: Hector Sanjuan <[email protected]> Co-authored-by: Ian Davis <[email protected]> Co-authored-by: Peter Rabbitson <[email protected]> Co-authored-by: Hector Sanjuan <[email protected]>

frrist mentioned this pull request Mar 4, 2021

Implement Event observer and Settings for 3rd party dep injection filecoin-project/lotus#5693

Merged

frrist force-pushed the frrist/visor-lotus branch from 52a025b to fb11208 Compare March 4, 2021 19:16

frrist self-assigned this Mar 4, 2021

frrist marked this pull request as draft March 4, 2021 19:17

frrist changed the base branch from master to lily March 8, 2021 20:25

dep(ffi): replace ffi stub with the real deal

c161a52

-- will allow lotus binary to be built in following commit

frrist force-pushed the frrist/visor-lotus branch from aa4d49f to ba1e22f Compare March 8, 2021 21:58

frrist changed the title ~~WIP: make a lotus+visor node binary~~ Create a Lily node by embed lotus into visor to observe tipset apply and revert events Mar 8, 2021

frrist changed the title ~~Create a Lily node by embed lotus into visor to observe tipset apply and revert events~~ Create a Lily node by embedding lotus into visor to observe tipset apply and revert events Mar 8, 2021

frrist requested review from placer14 and iand and removed request for placer14 March 8, 2021 22:04

feat(lily): embed lotus daemon in visor called lily

656b012

- allows a lotus node to be instanciated and for visor to observe tipsets applied and reverted by the lotus node.

frrist force-pushed the frrist/visor-lotus branch from ba1e22f to 656b012 Compare March 8, 2021 22:07

frrist commented Mar 8, 2021

View reviewed changes

iand reviewed Mar 9, 2021

View reviewed changes

polish: move lily api to lens, reduce lotus api consumed

199dd2a

frrist marked this pull request as ready for review March 9, 2021 20:30

move everything under lily root command on visor

fc2ad2d

frrist force-pushed the frrist/visor-lotus branch from 54395bb to fc2ad2d Compare March 9, 2021 22:09

frrist added 2 commits March 9, 2021 15:22

polish(cil): add full config support

adc430e

remove some old crap

19106ff

frrist commented Mar 9, 2021

View reviewed changes

commands/util/rpc.go

@@ -0,0 +1,148 @@

package util

Copy link

Member Author

frrist Mar 9, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mostly copy-paste from lotus to get the injection working.

frrist commented Mar 9, 2021

View reviewed changes

iand self-requested a review March 10, 2021 15:00

frrist added 5 commits March 10, 2021 14:35

I think circleci is caching too much

bb5f17f

mo god tidy

a6bd8fb

make file was just wrong, borrow from lotus

b254bf1

sudo build my code circleci!

9d65ac0

prepare first, then do circleci stuffs

79231aa

iand approved these changes Mar 11, 2021

View reviewed changes

frrist merged this pull request into lily Mar 11, 2021

frrist deleted the frrist/visor-lotus branch March 11, 2021 18:59

frrist added a commit that referenced this pull request Mar 11, 2021

Create a Lily node by embedding lotus into visor to observe tipset ap…

a0e1aa0

…ply and revert events (#408)

frrist added a commit that referenced this pull request Mar 22, 2021

Create a Lily node by embedding lotus into visor to observe tipset ap…

5585c0f

…ply and revert events (#408)

iand mentioned this pull request Apr 20, 2021

feat: create a long running daemon mode for visor (lily) #473

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a Lily node by embedding lotus into visor to observe tipset apply and revert events #408

Create a Lily node by embedding lotus into visor to observe tipset apply and revert events #408

frrist commented Mar 3, 2021 •

edited

Loading

frrist Mar 8, 2021

iand Mar 9, 2021

frrist Mar 9, 2021

iand Mar 9, 2021

frrist Mar 9, 2021

iand Mar 9, 2021

iand Mar 9, 2021

frrist Mar 9, 2021

iand Mar 9, 2021

iand Mar 9, 2021

frrist Mar 9, 2021

iand Mar 9, 2021

frrist Mar 9, 2021

frrist Mar 9, 2021

frrist Mar 9, 2021

iand Mar 11, 2021

codecov-io commented Mar 10, 2021

iand left a comment

iand Mar 11, 2021

iand Mar 11, 2021

iand Mar 11, 2021

iand Mar 11, 2021


		obs := observer.NewIndexingTipSetObserver(indexer, tsCache)

		if err := m.Events.Observe(obs); err != nil {


		var log = logging.Logger("lily-api")

		type LilyNodeStruct struct {

Create a Lily node by embedding lotus into visor to observe tipset apply and revert events #408

Create a Lily node by embedding lotus into visor to observe tipset apply and revert events #408

Conversation

frrist commented Mar 3, 2021 • edited Loading

What

Usage

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Mar 10, 2021

Codecov Report

iand left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frrist commented Mar 3, 2021 •

edited

Loading