Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

snapshots: reduce merge limit of blocks to 100K #8614

Merged
merged 28 commits into from
Nov 1, 2023
Merged

Conversation

AskAlexSharov
Copy link
Collaborator

@AskAlexSharov AskAlexSharov commented Oct 30, 2023

Reason:

  • produce and seed snapshots earlier on chain tip. reduce depnedency on "good peers with history" at p2p-network.
    Some networks have no much archive peers, also ConsensusLayer clients are not-good(not-incentivised) at serving history.
  • avoiding having too much files:
    more files(shards) - means "more metadata", "more lookups for non-indexed queries", "more dictionaries", "more bittorrent connections", ...
    less files - means small files will be removed after merge (no peers for this files).

ToDo:
[x] Recent 500K - merge up to 100K
[x] Older than 500K - merge up to 500K
[x] Start seeding 100k files
[x] Stop seeding 100k files after merge (right before delete)

In next PR:
[] Old version of Erigon must be able download recent hashes. To achieve it - at first start erigon will download preverified hashes .toml from s3 - if it's newer that what we have (build-in) - use it.

@AskAlexSharov AskAlexSharov changed the title [wip] snapshots: reduce merge limit of blocks to 100K snapshots: reduce merge limit of blocks to 100K Nov 1, 2023
@AskAlexSharov AskAlexSharov merged commit 329d18e into devel Nov 1, 2023
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant