Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sharded TxnLookup index: non-existing hash (or old hash) may random-touch all files #9486

Closed
AskAlexSharov opened this issue Feb 22, 2024 · 0 comments

Comments

@AskAlexSharov
Copy link
Collaborator

AskAlexSharov commented Feb 22, 2024

Seems txn lookup by hash start slowly suffer from too many block snapshots - because we don't have cover-index (also reduced snap size to 100K).
Each lookup of old txn causing - random touch of all files tx.seg files (by false-positives of recsplit index).
Bor-mainnet has 286 tx.seg files (+ 286 tx.idx files).

Vmtouch output:
Uploading vmtouch_erigon.txt…

Reporter: https://discord.com/channels/687972960811745322/789381905958699018/1209824609206149140
cpu

Solutions

  • Make false-positives cheaper: change format of tx.seg from list_of(tx_hash_1st_byte+txn_rlp) to list_of(tx_hash_1st_byte)+list_of(txn_rlp) - means all first bytes will have data-locality and all tx_lookup operations will come to that part of file. First byte of hash we have to reduce false-positives of recsplit.

  • Different sharding key of tx.seg: block_num is not set in stone.

  • Different indices: btree (too much keys), bloom (too much keys).

AskAlexSharov added a commit that referenced this issue Mar 1, 2024
for #9486
it will apply only to new .idx files. 
it doesn't require user to re-gen .idx files (backward compatible), but
if you need: delete `snapshots/*.idx` and run `erigon snapshots index`
mriccobene pushed a commit to mriccobene/erigon that referenced this issue Mar 13, 2024
for erigontech#9486
it will apply only to new .idx files. 
it doesn't require user to re-gen .idx files (backward compatible), but
if you need: delete `snapshots/*.idx` and run `erigon snapshots index`
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant