Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(chainwatch): Make processor failures independent of each other #3254

Merged
merged 3 commits into from
Aug 24, 2020

Conversation

placer14
Copy link
Contributor

Chainwatch processor works in batches, collecting the deltas for each batch and shipping them off to sub-processing handlers which work through persisting the deltas to the DB. When any part of sub-processing fails, the errgroup cancels the whole batch. Instead, let's make each sub-process handler persist best-effort since they don't have schema dependencies between one another and log the failure. This should allow us to capture much more of our data despite internal state inconsistencies arising and making the whole thing bail.

Also, reduce the batch size. Smaller batches means a less efficient catchup when chainwatch lags, but the trade off is smaller gaps in data on failures.

@placer14 placer14 added the kind/enhancement Kind: Enhancement label Aug 23, 2020
@placer14 placer14 self-assigned this Aug 23, 2020
@magik6k magik6k merged commit ecc3ff4 into master Aug 24, 2020
@magik6k magik6k deleted the fix/chainwatch/independent-processor-subtasks branch August 24, 2020 20:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement Kind: Enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants