Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add new option for walk config #1212

Merged
merged 4 commits into from
May 26, 2023
Merged

Conversation

Terryhung
Copy link
Collaborator

@Terryhung Terryhung commented May 24, 2023

This pull request can resolve the issue: #1210.

Usage

By default, when the stopOnFatalError flag is set to false, the process will continue from start to end without halting, regardless of any errors encountered.

To turn on the flag: --stop-on-error=true

./lily job run --storage=CSV --tasks="fevm_block_header"  --stop-on-error=true walk --from=566400 --to=566405

@Terryhung Terryhung changed the title Add new option for walk config feat: add new option for walk config May 24, 2023
@Terryhung Terryhung marked this pull request as ready for review May 24, 2023 07:59
Copy link
Contributor

@birdychang birdychang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just some nit picks.

@@ -102,7 +105,14 @@ func (c *Walker) WalkChain(ctx context.Context, node lens.API, ts *types.TipSet)
c.report.UpdateCurrentHeight(int64(ts.Height()))
if success, err := c.obs.TipSet(ctx, ts, indexer.WithIndexerType(indexer.Walk), indexer.WithTasks(c.tasks)); err != nil {
span.RecordError(err)
return fmt.Errorf("notify tipset: %w", err)
log.Errorf("notify tipset: %w", err)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's improve the log message.

// return an error only if the "stopOnFatalError" flag is set to true.
if c.stopOnFatalError {
return err
}
} else if !success {
log.Errorw("walk incomplete", "height", ts.Height(), "tipset", ts.Key().String(), "reporter", c.name)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above so it looks consistent.

@@ -17,28 +17,30 @@ import (

var log = logging.Logger("lily/chain/walk")

func NewWalker(obs indexer.Indexer, node lens.API, name string, tasks []string, minHeight, maxHeight int64, r *schedule.Reporter) *Walker {
func NewWalker(obs indexer.Indexer, node lens.API, name string, tasks []string, minHeight, maxHeight int64, r *schedule.Reporter, stopOnFatalError bool) *Walker {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just call it stopOnError so less verbose everywhere?

var StopOnFatalError = &cli.BoolFlag{
Name: "stop-on-fatal-error",
var StopOnError = &cli.BoolFlag{
Name: "stop-on-error",
Usage: "Stop the job if it get error.",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stop the job if it encounters an error.

@Terryhung Terryhung merged commit 9da9561 into master May 26, 2023
@Terryhung Terryhung deleted the terryhung/change-walk-pipeline branch May 26, 2023 02:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants