Future of FRAME pallets

[xlc]

Currently FRAME offers a set of generalized & reusable pallets but the size is still small and we don't really have much external contributors submitting pallets to FRAME. A few reasons that I can think of are:

• Making PR and getting it merged to Substrate can be hard. Core devs are busy and don't always have time to review the change.
• We don't have any predictable release cycle for Substrate. This creates a big uncertainty on when the newly contributed changes will be available to downstream projects.
• Pallets in Substrate should be generalized, but that come with cost of making code harder to write.
• The pallets in Substrate cannot use anything from Polkadot or Cumulus. i.e. No XCM related pallet.

To workaround some of the above issues, we at Acala created ORML aims to provide a set of reusable pallets that covers more use cases such as multi token and XCM utility. It is currently used by at least 7 live parachains we are start receiving community contributed pallets into ORML. However, this is still not ideal.

Problems (and rants) >>> Click to expand <<<

ORML as a separate repo, it can be hard for external team to include it. In the end, it is another 3rd party deps that needs to be carefully managed to ensure the version aligns. Specially that Substrate is not published to crates.io and therefore people cannot use cargo to manage the versions. They need to manually pick the right branch or there will be compiler error.

While I believe most of core devs for Substrate are aware of ORML, the development are completely separated and we end up with a lot of similar but incompatible pallets. A few examples are:

• vesting
  • Vesting Module open-web3-stack/open-runtime-module-library#76
  • Refactor the balances module #4649
• NFT
  • Add nft module open-web3-stack/open-runtime-module-library#296
  • build-script-utils: allow reading the git commit hash from env var #10845
• tokens
  • Tokens Module open-web3-stack/open-runtime-module-library#3
  • Features needed for reserve-backed stablecoins #7152

And we need to remove features from ORML because they have been implemented in Substrate. Occasionally that's due to our own contribution as we still want the features to be maintained in Substrate as much as possible.

• Use AtLeast32Bit & AtLeast32BitUnsigned open-web3-stack/open-runtime-module-library#222
• Remove Fixed128 open-web3-stack/open-runtime-module-library#145
• Remove schedule-update and replace it with pallet-scheduler from Substrate open-web3-stack/open-runtime-module-library#236
• Replace our StorageMapIterator to the one from Substrate open-web3-stack/open-runtime-module-library#571
• remove build-script-utils open-web3-stack/open-runtime-module-library#695

On top of that, there are a lot of almost required pallet for parachains that are not provided by Substrate/Polkadot/Cumulus and end up been multiple incompatible implementations between all the parachains. I only checked a handful of parachains so this is not a complete list.

• Ability to dynamically pause transaction via governance

  • Acala: https://github.com/AcalaNetwork/Acala/blob/master/modules/transaction-pause/src/lib.rs
  • Bifrost: https://github.com/bifrost-finance/bifrost/blob/develop/pallets/call-switchgear/src/lib.rs
  • Manta: https://github.com/Manta-Network/Manta/tree/manta/pallets/tx-pause
  • Moonbeam: https://github.com/PureStake/moonbeam/blob/master/pallets/maintenance-mode/src/lib.rs
  • Parallel: https://github.com/parallel-finance/parallel/blob/master/pallets/emergency-shutdown/src/lib.rs

• Asset registry to manage token metadata and XCM MultiLocation

  • ORML, contributed by Interlay, reviewed by Acala: https://github.com/open-web3-stack/open-runtime-module-library/blob/master/asset-registry/src/lib.rs
  • Acala: https://github.com/AcalaNetwork/Acala/blob/master/modules/asset-registry/src/lib.rs
  • Bifrost: https://github.com/bifrost-finance/bifrost/blob/develop/pallets/asset-registry/src/lib.rs
  • HydraDX: https://github.com/galacticcouncil/warehouse/blob/main/asset-registry/src/lib.rs
  • Manta: https://github.com/Manta-Network/Manta/blob/manta/pallets/asset-manager/src/lib.rs
  • Moonbeam: https://github.com/PureStake/moonbeam/blob/master/pallets/asset-manager/src/lib.rs
  • Parallel: https://github.com/parallel-finance/parallel/blob/master/pallets/asset-registry/src/lib.rs

• Ability to pay tx fee using different tokens

  • Substrate: https://github.com/paritytech/substrate/blob/master/frame/transaction-payment/asset-tx-payment/src/lib.rs
  • Acala: https://github.com/AcalaNetwork/Acala/blob/master/modules/transaction-payment/src/lib.rs
  • Bifrost: https://github.com/bifrost-finance/bifrost/blob/develop/pallets/flexible-fee/src/lib.rs
  • HydraDX: https://github.com/galacticcouncil/warehouse/blob/main/transaction-multi-payment/src/lib.rs

And we have different flavor of collector-selection, xcm utility etc. Everyone are referencing each others and making their own pallets but with their own twist to fit their own requirements. This means wallets, blockchain explorers, any multi chain tools are very hard to be compatible with all the parachains. We have PSPs but I would leave it out for now as it isn't really helping.

All the Substrate developers needs to unite together to change the current situation so that we can focus on our own core business logic, instead spend all the time to implement logics required to run a L1 such as token handling, governance, XCM, etc.

I believe most of the Substrate developers are aware most the issues I raised here, but the fact we are still here is mainly due the the issues are not easy to solve. But the issues aren't going to solve itself so I am making this a discussion post, trying to identify where we want to be, and laying down a path towards there. Please help contribute your ideas to refine the tasks and help contribute followup issues and PRs to make this happen. But of course we will also need full support from Parity team as it requires restructure of the Substrate repo as well. Please signaling your support by upvote issue discussion.

[xlc]

Where we want to be

• Substrate and all the pallets are published to crates.io
  • This will allow us to actually semvar and mix & match different pallets without worrying too much about versioning
  • Substrate is stable enough that most of the pallets is compatible with multiple versions and therefore we should need to maintain a pallet unless there is a breaking change
• Semvar for all the pallets to ensure compatibility
  • Ideally we want automatic semvar by for example checking the metadata of a pallet to detect breaking changes. But a manual process + careful review can achieve similar result.
• Move most of the FRAME pallets out of Substrate
  • We still need to figure out how exactly we want to structure pallets, but Substrate repo is definitely not the right place
  • With improved stability, we should be able to move FRAME out without too much considerations. The companion flow works good enough to ensure compatibility between multiple repos so we should be good.
  • The core pallets such as system, timestamp can stay in Substrate.
  • To be discussed We may have a monorepo or a repo per pallet or something in between
• Easy to contribute as a community developer
  • We want to allow the pallet owners to manage their pallets (i.e. approve & merge PR) and that can be from community so PRs are not always blocked by core devs
  • Substrate already have a good processbot so it shouldn't be too hard to support code owners from community
• Parachain teams and freelancer developers actually want to contribute
  • We want to encourage community to build & contribute generalized composable pallets
  • The tip bot and treasury funding can provide monetary incentive
  • Similar to the ORML security bounty, the treasury should also fund security bounty and auditing
    • This helps auditing that auditors only need to audit the shared pallet once, or skip the audit knowing it is already audited.
    • This means parachain teams will want to have their pallet contributed here so it get a free audit & security bounty
• Anything else?

[jak-pan]

I agree with all of the above. I'd go for lower effort things like crates.io and semver enforcing (I know you have some process but doing this would help many other projects).

I Also agree with moving most of the basic (non-necessary for running chain) pallets out of the substrate and maintaining it separately it could be still under Parity but it would make the process of doing something less cumbersome and I think much faster (and cheaper) in terms of ci/cd. It might look like a big change but I don't think it needs to be that big from point of view of Parity.

Everything else above and under this post that @xlc posted we support. It might have not been the time before but I think it is about time that we start better coordinating and building common pallets together. It's great to experiment with design and use-cases and some teams might still need different functionality, but I think assets / tokens pallet is a really good example of where we should work more in sync.

P.S. We even have own version of stopping trading in certain pools in the making (not txs yet but also want to do it)...

[xlc]

How do we get there

• Figure out what's preventing publishing Substrate & FRAME to crates.io and create issues to track them and have them fixed
• Decide the repo structure for the new home for FRAME
  • The easy way is just a monorepo for all the pallets
    • That can cause some issues with code in Polkadot/Cumulus (i.e. XCM & parachain related code)
    • We could also move those out from Polkadot/Cumulus
  • Or one repo per pallet
    • That can be hard to manage when we need to make changes that affect multiple pallets
  • Or somewhere in between
    • e.g. a Core collection, XCM collection, Assets collection
    • The problem is it can be hard to define the boundary. Where do we put pallet that manages both XCM & Assets?
• Figure out what pallets that we want to build together and create issues to track them
  • All the pallets that I listed in my rant should be included here
• Create process to propose pallet, discuss design, and then implement them after it is been reviewed.
• Anything else?

[akru]

Hello @xlc, thank you for amazing proposal. Let me share my opinion, just as one of substrate developer for some years.

Shortly, I understand your point, but can’t support move core pallets out of substrate. Let’s go deep into details.

This is my personal opinion, let’s go step by step for the problems list:

• Making PR and getting it merged to Substrate can be hard. Core devs are busy and don't always have time to review the change.

That’s absolutely right, merging PR into substrate is hard, but it makes difficult to introduce dangerous/breaking change from other side. Fix all vulnerable code going pretty quick, thanks substrate devs.

• We don't have any predictable release cycle for Substrate. This creates a big uncertainty on when the newly contributed changes will be available to downstream projects.

Yes, yes and yes! Substrate should have predictable release cycle someday. I believe anyone from substrate community dream about it. But move frame out unfortunately don’t solve it.

• The pallets in Substrate cannot use anything from Polkadot or Cumulus. i.e. No XCM related pallet.

I’m sure that substrate core pallets should don’t known anything about XCM and other high level abstractions, it makes it non-core then.

Summarize, in my point of view, right now we have fine tuned and fully compatible minimal frame that guarantied to be well audited and stable. I’m not sure we can achieve it for dedicated repo without additional work for substrate devs.

From other side, really, community driven frame, where PR is easy to approve and merge, is what we need in growing ecosystem. I’m totally agree that we need more XCM pallets, more integration and complex use cases, but let’s don’t broke the foundation we builds for years and finally have now.

[xlc]

That’s absolutely right, merging PR into substrate is hard, but it makes difficult to introduce dangerous/breaking change from other side. Fix all vulnerable code going pretty quick, thanks substrate devs.

We already have processbot and companion checks to ensure we won't have unexpected breaking changes across multiple repos. With code owners and right process I believe we can ensure the quality of pallets in the new repo.

Yes, yes and yes! Substrate should have predictable release cycle someday. I believe anyone from substrate community dream about it. But move frame out unfortunately don’t solve it.

One of the reason that we can't have frequent release on Substrate is that occasionally the master will be in unreleasable state, could be due to a known bug, or a breaking change, or something else. The bigger the repo, the more likely one of the blocker issue will happen. If we move the pallets out of the repo, then we can release the pallets library independently of Substrate core.

We definitely want to ensure we still have a solid foundation, but I do believe we can still maintain it and restructure the repo at same time. If you take a look of the all pallets we currently have in Substrate, a lot of them are actually not really a core pallet, for whatever definitions that "core" is.

[hoonsubin]

Hello @xlc, this is a great discussion and I would like to throw my thoughts in here too in hopes of finding a solution.

First off, I completely agree with your observation on the current state of Substrate pallets. I have a lot of complaints about how things are managed with duplicate implementations. Although PSP can more or less help on the client side, this won't do much on the runtime development side.
Now, there is something that I will have to bring up though, and that is the purpose of Substrate itself. Substrate is meant to be a blockchain framework for building public, private, standalone blockchains, and parachains. This means that not all pallets belong in the main Substrate repo. Because of this, I agree with @akru that the main Substrate project shouldn't care about XCM. I would go far as to suggest that we could define a "core" pallet as something that can be used on any blockchain regardless of the context (for all private, public blockchains and parachains). For example, the balance pallet and sudo pallet can be defined as core pallets, while everything in Cumulus is not since it is only relevant to parachains. For other use case pallets like uniques and lottery, we could keep them in a separate repo that is completely maintained by the community since it has nothing to do with the core Polkadot ecosystem or Parity.

Regardless, my recommendation is that we create a new GitHub organization named Substrate, and move everything related to Substrate and its pallets under that organization (migrate ORML, move frontier, cumulus, etc). We can perhaps split the main repo too into substrate-core and substrate-frames if that can help maintain the projects.

I believe that the main problem we have right now is the community's view on the purpose of Substrate as a framework, and the fractured nature of how pallets are being kept by multiple organizations.

[bernardoaraujor]

@hoonsubin can you elaborate more on the reasons you believe PSPs can't help on bringing standardization to the runtime?

clearly PSPs have been underutilized and haven't provided much value to the community so far, but given the right structure and direction, IMO they could play a pivotal role in helping the community achieve a shared vision for standards in the entire substrate stack

a few examples:

• OpenBrush has made some meaningful contributions in the context of Wasm Smart Contracts, which is technically a subset of the runtime (PSP-22, PSP-34, PSP-35)
• Efinity has a pending proposal for a paratoken standard, which could be seen as a runtime equivalent to ERC1155 (PSP-32)

although the whole discussion proposed by @xlc revolves more around practical issues in regards to repository management and development workflows, it would be great to foster a shared vision where FRAME and ORML are standardized together and don't need to exist as separate entities

[hoonsubin]

@bernardoaraujor For smart contracts, token representations, and ecosystem tools, I think PSP is doing a great job. I see a lot of potential in this area.

However, I think the runtime development is a bit different because the whole point of Substrate is that anyone can easily create their own blockchain with custom logic that can be extended from the existing one or rely on the network's inherent attributes (ex: the behavior of the token is defined by multiple pallets). And because we had a lot of Substrate projects way before XCM was usable between parachains, and those chains needed to focus on usability over compatibility, there was no incentive for complying with PSP.
I think pallets in Unique Network are a great example where it extends the existing fungible and non-fungible token pallets with custom behaviors like NFT interactivity.

Smart contracts only work on the contract sandbox, and therefore, we can make assumptions about what is possible, what is limited, and how to make it compatible just by creating a common interface. But for runtime pallets, I think it's difficult to make any guarantees based on its interface, and there are no incentives for projects to comply with PSP if XCM is not a high priority for them.

Now, this is just my opinion on the matter, and I won't say this is entirely correct.

given the right structure and direction, IMO they could play a pivotal role in helping the community achieve a shared vision for standards in the entire substrate stack

I definitely agree with this, and if we create PSPs that are specifically dealing with cross-consensus transactions of assets like ICS20 and for other things like extrinsic calls, then I think my point above won't be an issue since regardless of how the chain implements the runtime, the transaction between the chain won't be a problem.

[ETeissonniere]

I would tend to agree with the general opinion here which is that we need a better way to manage, version, release, and contribute to pallets. A good example of how this is done in the ETH sphere would be OpenZeppelin. It is community-owned and managed (to some extent) and now widely used by the ecosystem. We could start something similar by moving most non-core pallets out of FRAME, ORML, or equivalents and push them to a unique repo under the Polkadot/Substrate/Parity umbrella (pick your favorite).

With that being said, we probably do not want to only emphasize the single repo/org solution and forget all the valid reasons for teams to manage pallets on their own side. In such a case shouldn't we also discuss a better way to discover and standardize pallets? This could be as simple as tagging conventions on github and crates.io, along with a simple frontend to discover those. Or even a simple index of all the pallets in the ecosystem.

[hoonsubin]

+1 for having a standard pallet registry

[xlc]

tagging conventions on crates.io will be great. we will have a default UI with crates.io and it will be very easy to build custom Ui on top of it

[stiiifff]

We have something upcoming on that front.