wip: Remove subxt Config associated types by relying on enriched metadata (v16)

[lexnv]

This PR exposes the custom types present in the enriched metadata (possibly the next V16).

This stands as a PoC to pave the way towards the V16 and check the viability of exposing chain specific types to reduce the amount of configuration subxt users need to handle.

This aims to make subxt more robust and ideally allow users to communicate to any chain without relying on preset code configs. At the same time, this reduces the burden on developers to add said preset code configs and instead relies on the metadata to expose the appropriate types.

This metadata PoC exposes 7 out of 8 configuration parameters needed by subxt to communicate with chains.

• At the moment, 4/7 configs from the enriched metadata can be plugged in subxt directly
• The remaining 3 configs Hasher Header and Signature need extra functionality on the subxt::codegen side
  • Mainly to implement the underlying functionality
  • The plan atm is to match the type's name to deduce and implement the required code (for example, the type Hasher = Blake2Hasher can be deduced to be blake2 and therefore the said functionality introduced at compile-time via codegen)
• ExtrinsicParams needs further investigation since the metadata cannot safely export code functionality

Custom Types Exposed

This is what the codegen will produce after inspecting the metadata:

    pub mod custom_types {
        type AccountId = ::subxt::ext::subxt_core::utils::AccountId32;
        type Address = ::subxt::ext::subxt_core::utils::MultiAddress<
            ::subxt::ext::subxt_core::utils::AccountId32,
            ::core::primitive::u32,
        >;
        type AssetId = ::core::primitive::u32;
        type CallEnumTy = runtime_types::kitchensink_runtime::RuntimeCall;
        type Hash = ::subxt::ext::subxt_core::utils::H256;
        type Hashing = runtime_types::sp_runtime::traits::BlakeTwo256;
        type Header = runtime_types::sp_runtime::generic::header::Header<::core::primitive::u32>;
        type Signature = runtime_types::sp_runtime::MultiSignature;
    }

Metadata Config Example

     Running `metadata_config`
Balance transfer success: Transfer { from: AccountId32([212, 53, 147, 199, 21, 253, 211, 28, 97, 20, 26, 189, 4, 169, 159, 214, 130, 44, 133, 88, 133, 76, 205, 227, 154, 86, 132, 231, 165, 109, 162, 125]), to: AccountId32([142, 175, 4, 21, 22, 135, 115, 99, 38, 201, 254, 161, 126, 37, 252, 82, 135, 97, 54, 147, 201, 18, 144, 156, 178, 38, 170, 71, 148, 242, 106, 72]), amount: 10000 }

[jsdw]

The thought I had for this sort of thing was originally going to be more like:

type Hasher = DynamicHasher

And then this hasher type (DynamicHasher here) is given the metadata when it tries to hash something, and can inspect the name of the hashing function and do the correct hashing logic itself.

In either case though, you need it to line up with the type Hash I think, which would be interesting/messy to do. It might be that we can infer and generate the Hash type based on what we see that the Hasher is though, rather than trying to extract it from the metadata. So maybe you'd end up with:

// DynamicHasher would spit out hashes like this, and these would encode
// to their inner bytes only (ignoring the variant byte):
enum DynamicHash {
    // an enum that supports the common hash types we know about
    BlakeTwo256([u8; 32]),
    TwoX128([u8; 16])
}


// ...
type Hash = DynamicHash;
type Hasher = DynamicHasher;

But we could theoretically do it all with codegen instead and extend the Subxt macro so that we can provide the necessary config types or override existing ones, but by default it will codegen relevant types like Hasher and Hash based on the metadata!

This would have the advantage of being faster/more correct and the disadvantage of needing more different Config impls for different chains rather than potentially being able to rely on one Config impl for multiple chains (or customising it for one chain).

Something to ponder :)

[lexnv]

Updated the PoC to use build on top of the latest PRs:

• Metadata V16 (unstable): Enrich metadata with associated types of config traits  polkadot-sdk#5274
• v16: Add unstable metadata v16 frame-metadata#82
• v16: Expose the unstable metadata v16 polkadot-sdk#5732

I'll still need to think about the dynamic hashers/types, wanted so far to check that everything is exposed from the substrate side

[jsdw]

I haven't looked at this yet but just wanted to leave this thought form yesterdays call before I forget:

It might be nice to have both:

1. Statically generated code for some chain config based on what the metadata is telling us (where possible; in some cases we won't be able to provide this eg unknown hasher type used or whatever, and I'm not certain if we can provide everything with certainty, so might need to be some generic that you extend with a coupel of other params to make a full Config impl)
2. Have the SubstrateConfig in subxt aim to work with as many chains as possible, ie make it use DynamicHasher etc and then at runtime use the metadata to look up eg which hashing function to use or whatever. Again, this won't always work and may fail at runtime when eg the metadata says some hashing funciton that we don't know about.

This combo would allow for things to "just work" in the ismple case using SubstrateConfig but also allow for a more optimal hard coded Config to exist for a specific chain where more perf is desirable.

I think at the moment, if we just did one of those, I'd vote for (2) ie make SubstrateConfig automatically work with as many chains as possible, and then add the codegen as a later step if we decide it's worth adding to optimise things a bit.