[WIP] PoV friendly, multi-block Election Provider Multi-Phase pallet

[gpestana]

highly WIP, opening draft PR for early feedback.

This PR implements a PoV friendly, multi-block EPM to be used by the staking parachain. It is split into multiple sub-pallets for better logic and storage encapsulation and better readability. The pallet split consists of a main pallet and several other sub-pallets that implement the logic for different sub-systems, namely:

• main pallet:
  • implements the trait ElectionProvider
  • fn elect(remaining_pages) basically fetches the queued page from the pallet::verifier, which keeps the valid solutions in its storage.
  • manages current election Phase in on_initialize. The current phase signals other sub-pallets what to do.
  • stores and manages the (paged) target and voter snapshots.
    • note: the staking pallet needs to return/interpret a paged fetching of the snapshot data for both voters and targes.
• signed pallet:
  • implements the trait SolutionDataProvider, which provides a paged solution for the current best score (the pallet::verifier is the consumer, when trying to fetch the best queued solution to verify).
  • keeps track of the best solution commitments from submitters.
  • exposes Call::register for submitters to submit a commitment score for their solution.
  • exposes callable Call::submit_page for submitters to submit a page of their solution.
  • upon the verifier pallet finalizing the paged solution verification, it handles the submission deposit/rewards based on the reported VerificationResult (from pallet::signed).
• verifier pallet:
  • implements the trait Verifier: verifies one solution page on-call. The inputs for the verification are provided in-place.
  • implements the trait AsyncVerifier: fetches pages from the implementor of SolutionDataProvider (implemented by pallet::signed) and verifies the paged solution.
  • on_initialize, it checks if the verification is ongoing and proceeds with it
  • it has it's own VerificationStatus which signals the current state of the verification
  • for each successfully verified page, add it to the QueuedSolution storage.
  • at the end of verifying a solution, it reports the results of the verification back to the implementor of trait SolutionDataProvider (pallet::signed)
• unsigned pallet:
  • on_initialize checks if on UnsignedPhase and no queued solution; compute a solution with offchain Miner.
  • implements the off-chain unsigned (paged) miner.
  • implements the inherent call that processes unsigned submissions.

---

Todo/discussion

• E2E multi-page election with staking and EPM-MB pallet integration.
• refactor the current on_initialize across all pallets to make explicit calls depending on the current phase, rather than relying on the pallet's on_initialize and current phase to decide what to do at a given block (TBD).
• remove the Emergency phase and instead just keep trying the election in case of failure.
• refactor current SignedValidation phase to have enough blockspace to verify all queued signed submissions, for security purposes (ie. at least max_num_queued_submissions * T::Pages blocks allocated to signed verification, return early if a submission is valid and accepted).
• implement the paged ingestion of the election results in the staking pallet. How to convert from multiple BoundedSupports to Exposures in the staking pallet in a nice way (add integration tests).
  • idea: if each page contains up to N targets and a validator may appear only in one page, we can process the pages in serie in the staking side, keeping track of the state of Exposures across the pages.
• allow the validator to replace the current submission if their submission has better score than the accepted queued submission.
• mutations to both the target and voter lists need to "freeze" while the snapshot is being generated (now multi-block). what's the best approach?

Closes: #2199
Related to: #491
Inspiration from https://github.com/paritytech/substrate/tree/kiz-multi-block-election

[Ank4n]

Just realised it might not work since the value is not a top level Vec. Probably in hindsight this could have been structured differently but for now I guess its okay to decode and append. Someday me may want to migrate it to make it more efficient.

[paritytech-cicd-pr]

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: test-linux-stable 2/3
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/5062907