This repository has been archived by the owner on Nov 15, 2023. It is now read-only.
Update implementors guide with sanitize_* & enter
#4294
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
emostov
changed the title
sanitize_*sanitize_* & enter
paritytech-processbot bot
pushed a commit
that referenced
this pull request
Nov 16, 2021
* move things around, add filter methods�� * validator keys, modify availability bitfields according to disputes * simplify, keep the filter -> sanitize generic for both usecases * minor * assure tests still work, reduce changeset * integration * start entropy passing * fixins * compile, 1 failing test * filter with coverage * fixins * Update runtime/parachains/src/paras_inherent.rs Co-authored-by: Robert Habermeier <[email protected]> * slip of the pen * improve test cases * misc * fix * fixins * test avoid extra into() calls in assert_noop! * chores * ff * test fixup superfluous into call * chore: pfmt * improve apply_block_weight_limit to try to maximize the number of sufficiently backed blocks and add extra bitfields in a round-robin fashion * new code treats the lack of backed candidates as ok * Use vrf based entropy * fixup vrf random * add warn * slip of the pen * fixup * assure ordering * rethink apply_weights * mock * use a closure as predicate check * extract and use DisputedBitfield * chore: simplify * remove stray dbg * chore: fmt * address feedback * fix test, halfway there * stage1 * dbg stuff * make group selection align * fix session index * fix wrongly returned candidates * cleanup * chore fmt * fix ensure check * make good case test work * more tests for bitfields * create sanitize_backed_candidates * fixup tests * update guide * add check referenced in the guide * improve weights code * fmt * fixins * Update roadmap/implementers-guide/src/runtime/inclusion.md Co-authored-by: Zeke Mostov <[email protected]> * compiling + address review * add comments * fix weight calc * address review comments and test failure * fix * fix: condition * Fix random_sel function * Fix overlength block check * Zeke + Ladi commit for disputes filtering + integration test builder + runtime benchmarks + integration tests * Add benchmarks for code upgrades * Code upgrade bench; Feature gate TestWeightInfo * Try and make CI happier * Feature gate enter test to not(benchmarks) * Make sure no unused imports/fn * refactor, re-use, the beginning * Fix issue with frame benchmarking dep compilation * More precise feature gating for some derives * integrate piece-wise * foo * fixins * chore fmt * fixins * rename const generic * Update runtime/parachains/src/paras_inherent.rs Co-authored-by: Zeke Mostov <[email protected]> * Fix compilation * limit to test * remove unused spam slots * spellcheck * remove a tick, fix a typo * Add Code upgrade weights * comment improvements + >= Co-authored-by: Zeke Mostov <[email protected]> * remove another tick * Update runtime/parachains/src/paras_inherent/benchmarking.rs Co-authored-by: Zeke Mostov <[email protected]> * saturating fixins + some spaces * fix * benchmarking - preliminary results * Add training wheels * Refactor some early exit logic for enter * Gracefully handle filtering bitfields & candidates (#4280) This updates the logic for sanitize_bitfields and sanitize_backed_candidates to never error when there is an issue, but instead to simply skip the problematic items. * Refactor inherent data weight limiting logic (#4287) * Apply suggestions from code review * Update runtime/parachains/src/builder.rs Co-authored-by: Zeke Mostov <[email protected]> * Update runtime/parachains/src/builder.rs * Update runtime/parachains/src/paras_inherent.rs * final pass * Run cargo +nightly-2021-10-29 fmt * Update implementors guide with `sanitize_*` & `enter` (#4294) * Make spell check happier * Make wasm runtimes compile with benchmarks enabled (#4303) * comment stuff out, use old toml * Seems to be working? * Remove feature gating from builder * Remove commented out stuff * Remove generic from digest * Update weight files for runtime Co-authored-by: Robert Habermeier <[email protected]> Co-authored-by: Zeke Mostov <[email protected]> Co-authored-by: Lldenaurois <[email protected]> Co-authored-by: Zeke Mostov <[email protected]> Co-authored-by: Bastian Köcher <[email protected]>
drahnr
added a commit
that referenced
this pull request
Nov 26, 2021
* move things around, add filter methods�� * validator keys, modify availability bitfields according to disputes * simplify, keep the filter -> sanitize generic for both usecases * minor * assure tests still work, reduce changeset * integration * start entropy passing * fixins * compile, 1 failing test * filter with coverage * fixins * Update runtime/parachains/src/paras_inherent.rs Co-authored-by: Robert Habermeier <[email protected]> * slip of the pen * improve test cases * misc * fix * fixins * test avoid extra into() calls in assert_noop! * chores * ff * test fixup superfluous into call * chore: pfmt * improve apply_block_weight_limit to try to maximize the number of sufficiently backed blocks and add extra bitfields in a round-robin fashion * new code treats the lack of backed candidates as ok * Use vrf based entropy * fixup vrf random * add warn * slip of the pen * fixup * assure ordering * rethink apply_weights * mock * use a closure as predicate check * extract and use DisputedBitfield * chore: simplify * remove stray dbg * chore: fmt * address feedback * fix test, halfway there * stage1 * dbg stuff * make group selection align * fix session index * fix wrongly returned candidates * cleanup * chore fmt * fix ensure check * make good case test work * more tests for bitfields * create sanitize_backed_candidates * fixup tests * update guide * add check referenced in the guide * improve weights code * fmt * fixins * Update roadmap/implementers-guide/src/runtime/inclusion.md Co-authored-by: Zeke Mostov <[email protected]> * compiling + address review * add comments * fix weight calc * address review comments and test failure * fix * fix: condition * Fix random_sel function * Fix overlength block check * Zeke + Ladi commit for disputes filtering + integration test builder + runtime benchmarks + integration tests * Add benchmarks for code upgrades * Code upgrade bench; Feature gate TestWeightInfo * Try and make CI happier * Feature gate enter test to not(benchmarks) * Make sure no unused imports/fn * refactor, re-use, the beginning * Fix issue with frame benchmarking dep compilation * More precise feature gating for some derives * integrate piece-wise * foo * fixins * chore fmt * fixins * rename const generic * Update runtime/parachains/src/paras_inherent.rs Co-authored-by: Zeke Mostov <[email protected]> * Fix compilation * limit to test * remove unused spam slots * spellcheck * remove a tick, fix a typo * Add Code upgrade weights * comment improvements + >= Co-authored-by: Zeke Mostov <[email protected]> * remove another tick * Update runtime/parachains/src/paras_inherent/benchmarking.rs Co-authored-by: Zeke Mostov <[email protected]> * saturating fixins + some spaces * fix * benchmarking - preliminary results * Add training wheels * Refactor some early exit logic for enter * Gracefully handle filtering bitfields & candidates (#4280) This updates the logic for sanitize_bitfields and sanitize_backed_candidates to never error when there is an issue, but instead to simply skip the problematic items. * Refactor inherent data weight limiting logic (#4287) * Apply suggestions from code review * Update runtime/parachains/src/builder.rs Co-authored-by: Zeke Mostov <[email protected]> * Update runtime/parachains/src/builder.rs * Update runtime/parachains/src/paras_inherent.rs * final pass * Run cargo +nightly-2021-10-29 fmt * Update implementors guide with `sanitize_*` & `enter` (#4294) * Make spell check happier * Make wasm runtimes compile with benchmarks enabled (#4303) * comment stuff out, use old toml * Seems to be working? * Remove feature gating from builder * Remove commented out stuff * Remove generic from digest * Update weight files for runtime Co-authored-by: Robert Habermeier <[email protected]> Co-authored-by: Zeke Mostov <[email protected]> Co-authored-by: Lldenaurois <[email protected]> Co-authored-by: Zeke Mostov <[email protected]> Co-authored-by: Bastian Köcher <[email protected]>
drahnr
added a commit
that referenced
this pull request
Nov 26, 2021
* move things around, add filter methods�� * validator keys, modify availability bitfields according to disputes * simplify, keep the filter -> sanitize generic for both usecases * minor * assure tests still work, reduce changeset * integration * start entropy passing * fixins * compile, 1 failing test * filter with coverage * fixins * Update runtime/parachains/src/paras_inherent.rs Co-authored-by: Robert Habermeier <[email protected]> * slip of the pen * improve test cases * misc * fix * fixins * test avoid extra into() calls in assert_noop! * chores * ff * test fixup superfluous into call * chore: pfmt * improve apply_block_weight_limit to try to maximize the number of sufficiently backed blocks and add extra bitfields in a round-robin fashion * new code treats the lack of backed candidates as ok * Use vrf based entropy * fixup vrf random * add warn * slip of the pen * fixup * assure ordering * rethink apply_weights * mock * use a closure as predicate check * extract and use DisputedBitfield * chore: simplify * remove stray dbg * chore: fmt * address feedback * fix test, halfway there * stage1 * dbg stuff * make group selection align * fix session index * fix wrongly returned candidates * cleanup * chore fmt * fix ensure check * make good case test work * more tests for bitfields * create sanitize_backed_candidates * fixup tests * update guide * add check referenced in the guide * improve weights code * fmt * fixins * Update roadmap/implementers-guide/src/runtime/inclusion.md Co-authored-by: Zeke Mostov <[email protected]> * compiling + address review * add comments * fix weight calc * address review comments and test failure * fix * fix: condition * Fix random_sel function * Fix overlength block check * Zeke + Ladi commit for disputes filtering + integration test builder + runtime benchmarks + integration tests * Add benchmarks for code upgrades * Code upgrade bench; Feature gate TestWeightInfo * Try and make CI happier * Feature gate enter test to not(benchmarks) * Make sure no unused imports/fn * refactor, re-use, the beginning * Fix issue with frame benchmarking dep compilation * More precise feature gating for some derives * integrate piece-wise * foo * fixins * chore fmt * fixins * rename const generic * Update runtime/parachains/src/paras_inherent.rs Co-authored-by: Zeke Mostov <[email protected]> * Fix compilation * limit to test * remove unused spam slots * spellcheck * remove a tick, fix a typo * Add Code upgrade weights * comment improvements + >= Co-authored-by: Zeke Mostov <[email protected]> * remove another tick * Update runtime/parachains/src/paras_inherent/benchmarking.rs Co-authored-by: Zeke Mostov <[email protected]> * saturating fixins + some spaces * fix * benchmarking - preliminary results * Add training wheels * Refactor some early exit logic for enter * Gracefully handle filtering bitfields & candidates (#4280) This updates the logic for sanitize_bitfields and sanitize_backed_candidates to never error when there is an issue, but instead to simply skip the problematic items. * Refactor inherent data weight limiting logic (#4287) * Apply suggestions from code review * Update runtime/parachains/src/builder.rs Co-authored-by: Zeke Mostov <[email protected]> * Update runtime/parachains/src/builder.rs * Update runtime/parachains/src/paras_inherent.rs * final pass * Run cargo +nightly-2021-10-29 fmt * Update implementors guide with `sanitize_*` & `enter` (#4294) * Make spell check happier * Make wasm runtimes compile with benchmarks enabled (#4303) * comment stuff out, use old toml * Seems to be working? * Remove feature gating from builder * Remove commented out stuff * Remove generic from digest * Update weight files for runtime Co-authored-by: Robert Habermeier <[email protected]> Co-authored-by: Zeke Mostov <[email protected]> Co-authored-by: Lldenaurois <[email protected]> Co-authored-by: Zeke Mostov <[email protected]> Co-authored-by: Bastian Köcher <[email protected]>
ordian
pushed a commit
that referenced
this pull request
Nov 26, 2021
* move things around, add filter methods�� * validator keys, modify availability bitfields according to disputes * simplify, keep the filter -> sanitize generic for both usecases * minor * assure tests still work, reduce changeset * integration * start entropy passing * fixins * compile, 1 failing test * filter with coverage * fixins * Update runtime/parachains/src/paras_inherent.rs Co-authored-by: Robert Habermeier <[email protected]> * slip of the pen * improve test cases * misc * fix * fixins * test avoid extra into() calls in assert_noop! * chores * ff * test fixup superfluous into call * chore: pfmt * improve apply_block_weight_limit to try to maximize the number of sufficiently backed blocks and add extra bitfields in a round-robin fashion * new code treats the lack of backed candidates as ok * Use vrf based entropy * fixup vrf random * add warn * slip of the pen * fixup * assure ordering * rethink apply_weights * mock * use a closure as predicate check * extract and use DisputedBitfield * chore: simplify * remove stray dbg * chore: fmt * address feedback * fix test, halfway there * stage1 * dbg stuff * make group selection align * fix session index * fix wrongly returned candidates * cleanup * chore fmt * fix ensure check * make good case test work * more tests for bitfields * create sanitize_backed_candidates * fixup tests * update guide * add check referenced in the guide * improve weights code * fmt * fixins * Update roadmap/implementers-guide/src/runtime/inclusion.md Co-authored-by: Zeke Mostov <[email protected]> * compiling + address review * add comments * fix weight calc * address review comments and test failure * fix * fix: condition * Fix random_sel function * Fix overlength block check * Zeke + Ladi commit for disputes filtering + integration test builder + runtime benchmarks + integration tests * Add benchmarks for code upgrades * Code upgrade bench; Feature gate TestWeightInfo * Try and make CI happier * Feature gate enter test to not(benchmarks) * Make sure no unused imports/fn * refactor, re-use, the beginning * Fix issue with frame benchmarking dep compilation * More precise feature gating for some derives * integrate piece-wise * foo * fixins * chore fmt * fixins * rename const generic * Update runtime/parachains/src/paras_inherent.rs Co-authored-by: Zeke Mostov <[email protected]> * Fix compilation * limit to test * remove unused spam slots * spellcheck * remove a tick, fix a typo * Add Code upgrade weights * comment improvements + >= Co-authored-by: Zeke Mostov <[email protected]> * remove another tick * Update runtime/parachains/src/paras_inherent/benchmarking.rs Co-authored-by: Zeke Mostov <[email protected]> * saturating fixins + some spaces * fix * benchmarking - preliminary results * Add training wheels * Refactor some early exit logic for enter * Gracefully handle filtering bitfields & candidates (#4280) This updates the logic for sanitize_bitfields and sanitize_backed_candidates to never error when there is an issue, but instead to simply skip the problematic items. * Refactor inherent data weight limiting logic (#4287) * Apply suggestions from code review * Update runtime/parachains/src/builder.rs Co-authored-by: Zeke Mostov <[email protected]> * Update runtime/parachains/src/builder.rs * Update runtime/parachains/src/paras_inherent.rs * final pass * Run cargo +nightly-2021-10-29 fmt * Update implementors guide with `sanitize_*` & `enter` (#4294) * Make spell check happier * Make wasm runtimes compile with benchmarks enabled (#4303) * comment stuff out, use old toml * Seems to be working? * Remove feature gating from builder * Remove commented out stuff * Remove generic from digest * Update weight files for runtime Co-authored-by: Robert Habermeier <[email protected]> Co-authored-by: Zeke Mostov <[email protected]> Co-authored-by: Lldenaurois <[email protected]> Co-authored-by: Zeke Mostov <[email protected]> Co-authored-by: Bastian Köcher <[email protected]>
ordian
pushed a commit
that referenced
this pull request
Nov 29, 2021
* move paras inherent filtering to runtime (#4028) * move things around, add filter methods�� * validator keys, modify availability bitfields according to disputes * simplify, keep the filter -> sanitize generic for both usecases * minor * assure tests still work, reduce changeset * integration * start entropy passing * fixins * compile, 1 failing test * filter with coverage * fixins * Update runtime/parachains/src/paras_inherent.rs Co-authored-by: Robert Habermeier <[email protected]> * slip of the pen * improve test cases * misc * fix * fixins * test avoid extra into() calls in assert_noop! * chores * ff * test fixup superfluous into call * chore: pfmt * improve apply_block_weight_limit to try to maximize the number of sufficiently backed blocks and add extra bitfields in a round-robin fashion * new code treats the lack of backed candidates as ok * Use vrf based entropy * fixup vrf random * add warn * slip of the pen * fixup * assure ordering * rethink apply_weights * mock * use a closure as predicate check * extract and use DisputedBitfield * chore: simplify * remove stray dbg * chore: fmt * address feedback * fix test, halfway there * stage1 * dbg stuff * make group selection align * fix session index * fix wrongly returned candidates * cleanup * chore fmt * fix ensure check * make good case test work * more tests for bitfields * create sanitize_backed_candidates * fixup tests * update guide * add check referenced in the guide * improve weights code * fmt * fixins * Update roadmap/implementers-guide/src/runtime/inclusion.md Co-authored-by: Zeke Mostov <[email protected]> * compiling + address review * add comments * fix weight calc * address review comments and test failure * fix * fix: condition * Fix random_sel function * Fix overlength block check * Zeke + Ladi commit for disputes filtering + integration test builder + runtime benchmarks + integration tests * Add benchmarks for code upgrades * Code upgrade bench; Feature gate TestWeightInfo * Try and make CI happier * Feature gate enter test to not(benchmarks) * Make sure no unused imports/fn * refactor, re-use, the beginning * Fix issue with frame benchmarking dep compilation * More precise feature gating for some derives * integrate piece-wise * foo * fixins * chore fmt * fixins * rename const generic * Update runtime/parachains/src/paras_inherent.rs Co-authored-by: Zeke Mostov <[email protected]> * Fix compilation * limit to test * remove unused spam slots * spellcheck * remove a tick, fix a typo * Add Code upgrade weights * comment improvements + >= Co-authored-by: Zeke Mostov <[email protected]> * remove another tick * Update runtime/parachains/src/paras_inherent/benchmarking.rs Co-authored-by: Zeke Mostov <[email protected]> * saturating fixins + some spaces * fix * benchmarking - preliminary results * Add training wheels * Refactor some early exit logic for enter * Gracefully handle filtering bitfields & candidates (#4280) This updates the logic for sanitize_bitfields and sanitize_backed_candidates to never error when there is an issue, but instead to simply skip the problematic items. * Refactor inherent data weight limiting logic (#4287) * Apply suggestions from code review * Update runtime/parachains/src/builder.rs Co-authored-by: Zeke Mostov <[email protected]> * Update runtime/parachains/src/builder.rs * Update runtime/parachains/src/paras_inherent.rs * final pass * Run cargo +nightly-2021-10-29 fmt * Update implementors guide with `sanitize_*` & `enter` (#4294) * Make spell check happier * Make wasm runtimes compile with benchmarks enabled (#4303) * comment stuff out, use old toml * Seems to be working? * Remove feature gating from builder * Remove commented out stuff * Remove generic from digest * Update weight files for runtime Co-authored-by: Robert Habermeier <[email protected]> Co-authored-by: Zeke Mostov <[email protected]> Co-authored-by: Lldenaurois <[email protected]> Co-authored-by: Zeke Mostov <[email protected]> Co-authored-by: Bastian Köcher <[email protected]> * prefer code upgrades in inherent filtering (#4334) * impl prefered items Closes #4330 * do not stop attempting to select, just because one did not fit * doc * prefered -> preferred * missing usage of the preferred indices * sigh * shuffle is not available for chacha * remove duplicate weight addition * ref vs no ref * add additional assurances to `create_inherent` (#4349) * minor: move checks into separate fn * add additional validity checks * simplify shuffling * Closes potential OOB weight * improve docs * fooo * remove obsolete comment * move filtering into the rollback-transaction Technically this is not necessary but avoids future footguns. * move check up and avoid duplicate checks * refactor: make sure backed candidates are sane, even more * doc wording Co-authored-by: Zeke Mostov <[email protected]> * refactor: avoid const generics for sake of wasm size `true` -> `FullCheck::Skip`, `false` -> `FullCheck::Yes`. * chore: unify `CandidateCheckContext` instance names * refactor: introduce `IndexedRetain` for `Vec<T>` * chore: make tests prefix free * doc: re-introduce removed comment * refactor: remove another const generic to save some wasm size Co-authored-by: Zeke Mostov <[email protected]> * Inherent filtering follow up (#4305) * Add feature more feature gating for benchmarking + tests * New line * cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=kusama-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/kusama/src/weights/runtime_parachains_paras_inherent.rs * cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=polkadot-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/polkadot/src/weights/runtime_parachains_paras_inherent.rs * Do not assume we use max validators per core * Use kusama weights for rococo (hopefully temp) * cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_paras_inherent.rs * Add more validity votes when neccesary * Some fixes for the last commit * Restore westend weights * cargo run --quiet --release --features=runtime-benchmarks -- benchmark --chain=westend-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --header=./file_header.txt --output=./runtime/westend/src/weights/runtime_parachains_paras_inherent.rs * Revert bad westend weights write * Make sure to update val idx before skipping * Fix validity vote range to max at group size' * Temp setup for rococo * cargo run --quiet --release --features runtime-benchmarks -- benchmark --chain=rococo-dev --steps=50 --repeat=20 --pallet=runtime_parachains::paras_inherent --extrinsic=* --execution=wasm --wasm-execution=compiled --heap-pages=4096 --output=./runtime/rococo/src/weights/runtime_parachains_paras_inherent.rs --header=./file_header.txt * Augment generated Rococo weights * Make it compile * Revert range for enter_backed_candidates_variable * Delete runtime/kusama/src/weights/runtime_paras_paras_inherent.rs Co-authored-by: Parity Bot <[email protected]> * prepare worker: Catch unexpected unwinds (#4304) * prepare worker: Catch unexpected unwinds * Use more specific wording for unknown panic payload * Treat non-deterministic prep errors as internal errors (#4364) Closes #4293 This PR changes the way how we treat a certain subset of PVF preparation errors. Specifically, now only the deterministic errors are treated as invalid candidates. That is, the errors that are easily attributable to either the the PVF contents or the wasmtime code, but not e.g. I/O errors that could be triggered by the OS (insufficient memory, disk failure, too much load, etc). The latter are treated as internal errors and thus do not trigger the disputes. Co-authored-by: Robert Habermeier <[email protected]> Co-authored-by: Zeke Mostov <[email protected]> Co-authored-by: Lldenaurois <[email protected]> Co-authored-by: Zeke Mostov <[email protected]> Co-authored-by: Bastian Köcher <[email protected]> Co-authored-by: Parity Bot <[email protected]> Co-authored-by: Sergei Shulepov <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
target pr: #4028
target branch: bernhard-inherent-filtering