chore(trie): store only deleted keys in TrieWalker
#9226
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rkrasiuk
added
C-debt
Rjected
approved these changes
Jul 1, 2024
allnil
added a commit
to weaveVM/wvm-reth
that referenced
this pull request
Jul 15, 2024
* fix: skip failed new payload submission (paradigmxyz#9003) * test: disable dns discovery (paradigmxyz#9004) * chore: remove proptest arbitrary from codec derive and tests (paradigmxyz#8968) * chore(deps): rm unused dev deps (paradigmxyz#9005) * chore(deps): rm provider dep (paradigmxyz#9006) * chore: move ratelimit type to tokio util (paradigmxyz#9007) * chore: move different chain hardfork sets to `reth-ethereum-forks` (paradigmxyz#8984) * chore: remove `AllGenesisFormats` (paradigmxyz#9013) * chore: rename net-common to banlist (paradigmxyz#9016) * chore: remove `serde` from `ChainSpec` (paradigmxyz#9017) * chore: remove unused type (paradigmxyz#9019) * chore: rm serde for network builder (paradigmxyz#9020) * chore: rm default serde feature in reth-dns (paradigmxyz#9021) * chore(op): add link to op labs bedrock datadir download (paradigmxyz#9014) * chore(deps): replace fnv with fx (paradigmxyz#9024) * chore(deps): rm reth-rpc-types dep from reth-network (paradigmxyz#9023) * chore: remove some more usages of BytesMut (paradigmxyz#9025) * chore(deps): weekly `cargo update` (paradigmxyz#9036) Co-authored-by: github-merge-queue <[email protected]> * Change the wrong 'Child' and 'Auxiliary' usage (paradigmxyz#9033) * refactor(rpc): add builder pattern for `EthHandlers` (paradigmxyz#9035) * feat: add `AnyNodeTypes` type (paradigmxyz#9034) * chore: release 1.0.0 (paradigmxyz#9045) * feat(rpc): remove ipc future and now using ServerHandle and StopHandle from jsonrpsee (paradigmxyz#9044) * feat(node): derive `Clone` for `FullNode` (paradigmxyz#9046) * feat: integrate Node traits into LaunchContextWith (paradigmxyz#8993) * ci: use reth-prod.png for release notes (paradigmxyz#9047) * chore: tweak profiles, rename debug-fast to profiling (paradigmxyz#9051) * feat(examples): remote exex (paradigmxyz#8890) * fix: Change Arc<KzgSettings> to EnvKzgSettings (paradigmxyz#9054) * fix(op): configure discv5 properly in op builder (paradigmxyz#9058) * chore: move sync test to separate github action (paradigmxyz#9061) * feat: add base mainnet 10k block sync test to CI (paradigmxyz#9062) * fix: move base sync test comment (paradigmxyz#9066) * docs(examples): add Remote ExEx to README.md (paradigmxyz#9067) * feat: use a binary for sync tests (paradigmxyz#9071) * feat: add AnyNode type (paradigmxyz#9056) Co-authored-by: Matthias Seitz <[email protected]> * chore: add empty commands crate (paradigmxyz#9039) * fix: check the correct block in op-sync (paradigmxyz#9073) * fix(ci): inherit profiling in bench profile (paradigmxyz#9072) * clippy: rm outdated clippy allow (paradigmxyz#9070) * chore(trie): `TrieOp::as_update` (paradigmxyz#9076) * chore: simplify OptimismGenesisInfo extraction (paradigmxyz#9031) Co-authored-by: Matthias Seitz <[email protected]> * fix(ci): use correct profile for iai benches (paradigmxyz#9081) * chore(hive): update failed tests comments (paradigmxyz#9080) * Using associated trait bound for db error (paradigmxyz#8951) * readme: rm wip note * feat(examples): remove Remote ExEx (paradigmxyz#9085) * feat: initial cli abstraction (paradigmxyz#9082) * perf(trie): hold direct reference to post state storage in the cursor (paradigmxyz#9077) * chore(trie): add helpers to return trie keys as variants (paradigmxyz#9075) * test: include unexpected event in panic (paradigmxyz#9087) * chore(trie): hold direct reference to hashed accounts in cursor (paradigmxyz#9078) * fix: do not drop sub protocol messages during EthStream Handshake (paradigmxyz#9086) * ExEx Discv5 (paradigmxyz#8873) Co-authored-by: Matthias Seitz <[email protected]> * chore: add optimism cli crate (paradigmxyz#9096) * feat: reth stage unwind --offline (paradigmxyz#9097) Co-authored-by: Georgios Konstantopoulos <[email protected]> Co-authored-by: Oliver <[email protected]> * Add a metric for blob transactions nonce gaps (paradigmxyz#9106) * feat(cli): fail on invalid config (paradigmxyz#9107) * chore: import from static files crate directly (paradigmxyz#9111) * chore(deps): bump ratatui 0.27 (paradigmxyz#9110) * test: fix flaky connect (paradigmxyz#9113) * chore: rm beta checks (paradigmxyz#9116) * feat(ci): update GPG key in release workflow (paradigmxyz#9121) * refactor: move `DbTool` type to `db-common` (paradigmxyz#9119) * feat(cli): `reth prune` (paradigmxyz#9055) * refactor: move node-core/engine to standalone crate (paradigmxyz#9120) Co-authored-by: Matthias Seitz <[email protected]> * Subprotocol example (paradigmxyz#8991) Co-authored-by: owanikin <[email protected]> Co-authored-by: Matthias Seitz <[email protected]> * feat: add temporary docker tag action (paradigmxyz#9126) * fix: remove temp docker tag action (paradigmxyz#9128) * feat: add base fee metrics (paradigmxyz#9129) * feat: add parser functionality to `RethCli` (paradigmxyz#9127) * refactor: extract configuration types to `reth-network-types` (paradigmxyz#9136) * feat(trie): forward-only in-memory cursor (paradigmxyz#9079) * chore: remove empty ban_list.rs file (paradigmxyz#9133) * chore: rm utils.rs from cli crate (paradigmxyz#9132) * chore: move engine-primitives to engine folder (paradigmxyz#9130) * fix: use 8 byte SHA in getClientVersionV1 (paradigmxyz#9137) * fix(docs): Fix links node builder docs (paradigmxyz#9140) * chore(rpc): `reth-eth-api` crate (paradigmxyz#8887) Co-authored-by: Matthias Seitz <[email protected]> * chore(rpc): move impl of eth api server out of `reth-rpc-eth-api` (paradigmxyz#9135) Co-authored-by: Matthias Seitz <[email protected]> * chore(rpc): add me to RPC codeowners (paradigmxyz#9144) * refactor: clean-up discv5 configuration (paradigmxyz#9143) * chore: remove unused methods from `EvmEnvProviders` (paradigmxyz#9148) * chore(static_files): fix hacky type inference (paradigmxyz#9150) * feat: integrate CLI runner in CLI trait (paradigmxyz#9146) * feat: use new `ChainHardforks` type on `ChainSpec` (paradigmxyz#9065) * refactor(node-core/matrics): refactor the register version metrics function (paradigmxyz#9149) * chore: rename `TrieCursorFactory::storage_tries_cursor` to `TrieCursorFactory::storage_trie_cursor` (paradigmxyz#9145) * chore: move `revm_spec` methods from `reth-primitives` to chain specific crates (paradigmxyz#9152) * refactor(net): move node record constants to network-peers crate (paradigmxyz#9161) * chore: fix clippy (paradigmxyz#9163) * feat(trie): in-memory trie node overlay (paradigmxyz#8199) Co-authored-by: Roman Krasiuk <[email protected]> * chore(storage, provider): rename bundle state with receipts (paradigmxyz#9160) * dev: update `NodeExitFuture` (paradigmxyz#9153) Co-authored-by: Matthias Seitz <[email protected]> * chore: fix wrong function name (paradigmxyz#9164) * refactor: reduce number of args for `post_block_balance_increments` (paradigmxyz#9154) * fix: derive arbitrary for tests (paradigmxyz#9167) * fix(net/peer): remove the duplicated disconnect logic (paradigmxyz#9162) Signed-off-by: jsvisa <[email protected]> * feat(exex): backfill executor (paradigmxyz#9123) * chore: rm redundant clone (paradigmxyz#9174) * chore: remove `tx_env_with_recovered` from rpc crates (paradigmxyz#9158) * chore(trie): remove database-related types from trie keys (paradigmxyz#9175) * chore(ecies): expose ECIESCodec for fuzzing (paradigmxyz#9182) * chore: update audit doc to v2 (paradigmxyz#9177) * chore: rm leftover mods (paradigmxyz#9188) * feat(net/peer): add peer with udp socket (paradigmxyz#9156) Signed-off-by: jsvisa <[email protected]> * chore: replace raw usage of revm evm builder with EvmConfig usage (paradigmxyz#9190) * chore: finally move node-core to node folder (paradigmxyz#9191) * chore(deps): remove igd-next (paradigmxyz#9200) * chore(deps): weekly `cargo update` (paradigmxyz#9199) Co-authored-by: github-merge-queue <[email protected]> * chore(trie): rename in-memory trie cursors (paradigmxyz#9203) * refactor(net): some refactor in eth requests (paradigmxyz#9205) * refactor(revm): simplify `fill_tx_env` (paradigmxyz#9206) * docs: fix the links to code in discv4 docs (paradigmxyz#9204) * feat(net/peer): set rpc added peer as static (paradigmxyz#9201) Signed-off-by: jsvisa <[email protected]> * refactor: small refactoring (paradigmxyz#9208) * refactor: some simplifications around revm database (paradigmxyz#9212) * refactor(chainspec): simplify `genesis_header` using default pattern (paradigmxyz#9198) * fix: ambiguous deposit mint value in arbitrary (paradigmxyz#9216) * feat: new engine API handler (paradigmxyz#8559) Co-authored-by: Roman Krasiuk <[email protected]> Co-authored-by: Dan Cline <[email protected]> Co-authored-by: Federico Gimenez <[email protected]> * chore: remove unused `MIN_LENGTH_EIPXXXX_ENCODED` (paradigmxyz#9211) Co-authored-by: Matthias Seitz <[email protected]> * chore(trie): clean up trie update operation matching (paradigmxyz#9202) * chore: add builder with rng secret key fn (paradigmxyz#9218) * chore: remove usage of `tx_env_with_recovered` (paradigmxyz#9222) * chore: remove unused `static-file` code (paradigmxyz#9178) * chore: rename pipeline references to backfill sync (paradigmxyz#9223) * chore(execution): verify cumulative gas used before receipts root (paradigmxyz#9224) * docs(book): remote ExEx chapter (paradigmxyz#8992) Co-authored-by: Matthias Seitz <[email protected]> * chore(trie): store only deleted keys in `TrieWalker` (paradigmxyz#9226) * feat: implement write method on persistence task (paradigmxyz#9225) * chore: extract db commands (paradigmxyz#9217) Co-authored-by: Matthias Seitz <[email protected]> * feat(clippy): add `iter_without_into_iter` (paradigmxyz#9195) Co-authored-by: Matthias Seitz <[email protected]> Co-authored-by: Roman Krasiuk <[email protected]> * fix: typo in book intro (paradigmxyz#9228) * fix(rpc/admin): missing enode/enr in admin_peers endpoint (paradigmxyz#9043) Signed-off-by: jsvisa <[email protected]> * chore(trie): return nibbles from `TrieCursor::current` (paradigmxyz#9227) * chore: disable discovery for --dev (paradigmxyz#9229) * feat: add pruning related persistence API (paradigmxyz#9232) * chore: move `fill_block_env` to `ConfigureEvmEnv` (paradigmxyz#9238) * chore: add send_action method to persistence task (paradigmxyz#9233) * chore: use format_gas and format_gas_throughput for gas logs (paradigmxyz#9247) * chore: remove prune_modes from BlockWriter (paradigmxyz#9231) * chore: fix pruner exex height docs, add run docs (paradigmxyz#9250) * feat: add ChainspecParser trait (paradigmxyz#9259) Co-authored-by: Matthias Seitz <[email protected]> * refactor(evm): set prune modes optionally for the batch executor (paradigmxyz#9176) * feat: add pruner to persistence task (paradigmxyz#9251) Co-authored-by: Matthias Seitz <[email protected]> Co-authored-by: Federico Gimenez <[email protected]> * feat: add non feature gated noop block reader (paradigmxyz#9261) * refactor: move write_peers_to_file to NetworkManager impl (paradigmxyz#9134) Co-authored-by: Matthias Seitz <[email protected]> * chore: simplify p2p subcommand (paradigmxyz#9265) * trie: revamp trie updates (paradigmxyz#9239) * feat: moved optimism commands to create and remove from bin (paradigmxyz#9242) Co-authored-by: Matthias Seitz <[email protected]> * chore: move `pre_block_beacon_root_contract_call` to evm crates (paradigmxyz#9244) * feat: add ethereum engine chain orchestrator (paradigmxyz#9241) * feat: add empty ethereum cli crate (paradigmxyz#9268) * test: add unit tests for `save_receipts` (paradigmxyz#9255) * chore(rpc): `EthApi` builder (paradigmxyz#9041) Co-authored-by: Matthias Seitz <[email protected]> * feat: add resolve blocking for TrustedNode (paradigmxyz#9258) * test(transaction-pool): add unit tests for `BestTransactionsWithFees` `next` (paradigmxyz#9274) * clippy: rm some `type_complexity` (paradigmxyz#9276) Co-authored-by: Matthias Seitz <[email protected]> * test: rm useless unit tests for `calc_next_block_base_fee` (paradigmxyz#9280) * fix: always evaluate build_profile_name at compile time (paradigmxyz#9278) * feat(rpc): enable historical proofs (paradigmxyz#9273) * ci(hive): build `reth` externally (paradigmxyz#9281) * chore: Expose `TrieUpdates` inner fields (paradigmxyz#9277) * chore: use direct link to threshold docs (paradigmxyz#9284) * chore(rpc): rm dup getters `EthApi` (paradigmxyz#9283) * chore: move `withdrawal_requests_contract_call` to `reth-evm` (paradigmxyz#9272) * fix(cli): don't init datadir if it doesn't exist in db command (paradigmxyz#9264) Co-authored-by: Matthias Seitz <[email protected]> * chore: rename eth engine module orchestrator -> service (paradigmxyz#9288) * chore(trie): revamp inner in-memory trie cursor representation (paradigmxyz#9287) * perf: resolve trusted nodes concurrently (paradigmxyz#9291) * perf: spawn eth proof on IO pool (paradigmxyz#9293) * feat: add empty optimism rpc crate (paradigmxyz#9295) * ci: re-enable hive tests (paradigmxyz#9240) * feat: feature gate tokio::net lookup (paradigmxyz#9289) * chore: remove unused async (paradigmxyz#9299) * feat(rpc/ots): add rpc erigon_getHeaderByNumber (paradigmxyz#9300) Signed-off-by: jsvisa <[email protected]> * chore(cli): move utils to `reth-cli-utils` crate (paradigmxyz#9297) * fix: remove useless arbitrary feature (paradigmxyz#9307) * fix(rpc/ots): set block_number as u64 instead of NumberOrTag (paradigmxyz#9302) Signed-off-by: jsvisa <[email protected]> * feat: extract proof generation into `StateProofProvider` (paradigmxyz#9303) * chore(trie): return mutable prefix sets from `HashedPostState::construct_prefix_sets` (paradigmxyz#9306) * Fix: fix the issue of not being able to specify bootnode through command parameters (paradigmxyz#9237) * feat(trie): allow setting hashed cursor factory on `Proof` (paradigmxyz#9304) * feat: backfill job single block iterator (paradigmxyz#9245) * perf: resolve trusted peers (paradigmxyz#9301) * fix: holesky genesis hash (paradigmxyz#9318) * feat(trie): allow supplying prefix sets to `Proof` (paradigmxyz#9317) * feat(trie): `HashedPostState::account_proof` (paradigmxyz#9319) * github-workflows: delete the direction of dead(deleted) code (paradigmxyz#9316) * fix: no_std build (paradigmxyz#9313) * use op-alloy genesis types for genesis parsing (paradigmxyz#9292) Co-authored-by: Matthias Seitz <[email protected]> * chore(evm): turn associated `ConfigureEvm` fns into methods (paradigmxyz#9322) * qol: purge goerli (paradigmxyz#9310) * Remove fullprovider trait restriction in backfill impls (paradigmxyz#9326) * feat(trie): pass state reference to `StateProofProvider::proof` (paradigmxyz#9308) * feat: op eth api scaffolding (paradigmxyz#9324) * feat: implement `HistoricalStateProviderRef::proof` (paradigmxyz#9327) * feat: log throughput in execution stage (paradigmxyz#9253) Co-authored-by: Matthias Seitz <[email protected]> * chore: use `*_GENESIS_HASH` constants on ethereum chainspecs (paradigmxyz#9328) * chore(ci): improve `hive` workflow (paradigmxyz#9320) * chore: move `reth test-vectors` to `cli/commands` with feature (paradigmxyz#9329) * chore: disable `test-utils` for `stages-api` on `stages` (paradigmxyz#9331) * fix: format_gas show two decimal places (paradigmxyz#9336) * chore(deps): trim tokio features in eth-wire (paradigmxyz#9343) * move header.rs to eth-wire-types (paradigmxyz#9345) * chore: move featureless commands from `bin` to `reth-cli-commands` (paradigmxyz#9333) * chore: remove `test-utils`, `arbitrary` and `proptest` from built binary (paradigmxyz#9332) * chore: use usize in internal functions (paradigmxyz#9337) * chore: rm unused optimism feature (paradigmxyz#9342) * test: make eth-wire tests compile with --all-features (paradigmxyz#9340) * chore(meta): fix link in issue template config (paradigmxyz#9349) * chore(deps): rm discv4 dep from eth-wire (paradigmxyz#9344) * chore: dont enable serde by default for eth-wire (paradigmxyz#9339) * chore(meta): remove security link from issue template config (paradigmxyz#9350) * chore: make eyre optional in reth-db (paradigmxyz#9351) * chore: remove cfg'ed use serde (paradigmxyz#9352) * fix: encode block as is in debug_getRawBlock (paradigmxyz#9353) * chore: remove unused private stream type (paradigmxyz#9357) * chore(deps): weekly `cargo update` (paradigmxyz#9354) Co-authored-by: github-merge-queue <[email protected]> Co-authored-by: Matthias Seitz <[email protected]> * test(tx-pool): add unit tests for `BestTransactions` `add_new_transactions` (paradigmxyz#9355) * feat: add entrypoint and main loop to EngineApiTreeHandlerImpl (paradigmxyz#9334) * feat: adds eth request panels to grafana (paradigmxyz#9026) * feat: Add required trait impls for OpEthApi (paradigmxyz#9341) * feat: eip-7251 (paradigmxyz#9335) * replacing network_handle with peer_info trait object (paradigmxyz#9367) * book: add troubleshooting commands to check disk and memory health and performance (paradigmxyz#9364) Co-authored-by: Alexey Shekhirin <[email protected]> * chore(deps): bump alloy 0.1.4 (paradigmxyz#9368) * feat(pruner): log stats as an ordered list of segments (paradigmxyz#9370) * feat: move mev rpc types to alloy (paradigmxyz#9108) Co-authored-by: Matthias Seitz <[email protected]> * feat(tree): validate state root (paradigmxyz#9369) * docs: typos (paradigmxyz#9379) * perf(pruner): delete history indices by changeset keys (paradigmxyz#9312) Co-authored-by: joshieDo <[email protected]> Co-authored-by: Emilia Hane <[email protected]> * clippy: rm useless clippy statement (paradigmxyz#9380) * feat(tree): pre-validate fcu (paradigmxyz#9371) * Integrate permits for getproof (paradigmxyz#9363) * feat: add support for payload bodies (paradigmxyz#9378) * chore: move `stage` command to `reth-cli-commands` (paradigmxyz#9384) * feat(rpc/ots): implement ots_getContractCreator (paradigmxyz#9236) Signed-off-by: jsvisa <[email protected]> Co-authored-by: Alexey Shekhirin <[email protected]> Co-authored-by: Matthias Seitz <[email protected]> * refactor(rpc): remove intermediate types from rpc start up process (paradigmxyz#9180) Co-authored-by: Matthias Seitz <[email protected]> * chore: fix clippy warnings for needless_borrows_for_generic_args (paradigmxyz#9387) * feat(rpc/ots): implement ots_traceTransaction RPC (paradigmxyz#9246) Signed-off-by: jsvisa <[email protected]> * chore: update private-testnet.md (paradigmxyz#9389) * feat(rpc/ots): implement ots_getTransactionBySenderAndNonce (paradigmxyz#9263) Signed-off-by: jsvisa <[email protected]> Co-authored-by: Emilia Hane <[email protected]> Co-authored-by: Matthias Seitz <[email protected]> * chore: release 1.0.1 (paradigmxyz#9388) * fix: support additional eth call bundle args (paradigmxyz#9383) * chore(deps): bump revm 11 (paradigmxyz#9391) * chore(deps): rm reth-codecs dep (paradigmxyz#9390) * chore: fmt, clean, refactor --------- Signed-off-by: jsvisa <[email protected]> Co-authored-by: Matthias Seitz <[email protected]> Co-authored-by: Dan Cline <[email protected]> Co-authored-by: joshieDo <[email protected]> Co-authored-by: Emilia Hane <[email protected]> Co-authored-by: DaniPopes <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-merge-queue <[email protected]> Co-authored-by: leniram159 <[email protected]> Co-authored-by: Thomas Coratger <[email protected]> Co-authored-by: Aurélien <[email protected]> Co-authored-by: Tien Nguyen <[email protected]> Co-authored-by: Federico Gimenez <[email protected]> Co-authored-by: Alexey Shekhirin <[email protected]> Co-authored-by: Omid Chenane <[email protected]> Co-authored-by: Roman Roibu <[email protected]> Co-authored-by: Roman Krasiuk <[email protected]> Co-authored-by: Vid Kersic <[email protected]> Co-authored-by: Aditya Pandey <[email protected]> Co-authored-by: Georgios Konstantopoulos <[email protected]> Co-authored-by: Luca Provini <[email protected]> Co-authored-by: Oliver <[email protected]> Co-authored-by: owanikin <[email protected]> Co-authored-by: Arsenii Kulikov <[email protected]> Co-authored-by: Michael Sproul <[email protected]> Co-authored-by: Kien Trinh <[email protected]> Co-authored-by: Darshan Kathiriya <[email protected]> Co-authored-by: greged93 <[email protected]> Co-authored-by: Huber <[email protected]> Co-authored-by: Delweng <[email protected]> Co-authored-by: sboou <[email protected]> Co-authored-by: fdsuuo <[email protected]> Co-authored-by: Ninja <[email protected]> Co-authored-by: Luca Provini <[email protected]> Co-authored-by: Paul Wackerow <[email protected]> Co-authored-by: Querty <[email protected]> Co-authored-by: clabby <[email protected]> Co-authored-by: yutianwu <[email protected]> Co-authored-by: 令狐一冲 <[email protected]> Co-authored-by: Krishang <[email protected]> Co-authored-by: John <[email protected]> Co-authored-by: nk_ysg <[email protected]> Co-authored-by: kostekIV <[email protected]> Co-authored-by: Park Smith <[email protected]> Co-authored-by: Qiwei Yang <[email protected]> Co-authored-by: d3or <[email protected]> Co-authored-by: SHio <[email protected]> Co-authored-by: daobaniw <[email protected]> Co-authored-by: jn <[email protected]> Co-authored-by: Sean Matt <[email protected]> Co-authored-by: Barnabas Busa <[email protected]> Co-authored-by: Emilia Hane <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
We only ever track the trie keys we need to delete in
TrieWalker. Get rid ofTrieUpdatesin favor ofHashSet<TrieKey>.