v16: Expose the unstable metadata v16 #5732

lexnv · 2024-09-16T16:08:53Z

Builds on top of: #5274

cc @paritytech/subxt-team

Signed-off-by: Alexandru Vasile <[email protected]>

types Signed-off-by: Alexandru Vasile <[email protected]>

Signed-off-by: Alexandru Vasile <[email protected]>

…associated-types

Signed-off-by: Alexandru Vasile <[email protected]>

…associated-types Signed-off-by: Alexandru Vasile <[email protected]>

Signed-off-by: Alexandru Vasile <[email protected]>

…associated-types-enable-unstable Signed-off-by: Alexandru Vasile <[email protected]>

Signed-off-by: Alexandru Vasile <[email protected]>

…associated-types-enable-unstable

substrate/frame/metadata-hash-extension/Cargo.toml

Co-authored-by: Niklas Adolfsson <[email protected]>

niklasad1 · 2024-10-30T16:09:18Z

prdoc/pr_5732.prdoc

+
+crates:
+  - name: sp-metadata-ir
+    bump: patch


Shouldn't this be minor i.e, you add new functionality in backward compatible manner?

niklasad1 · 2024-10-30T16:17:54Z

substrate/primitives/metadata-ir/src/v14.rs

@@ -151,7 +151,7 @@ impl From<ExtrinsicMetadataIR> for ExtrinsicMetadata {
 	fn from(ir: ExtrinsicMetadataIR) -> Self {
 		ExtrinsicMetadata {
 			ty: ir.ty,
-			version: ir.version,
+			version: *ir.versions.get(0).expect("Metadata V14 supports only one version"),


Is there any plan to support several versions here and how to to find the latest version? I guess it will be last item in the Vec?

So my question is really if you need to do a linear search to find the specific version?

Maybe we want to return the lowest version. Because tools are currently using v4, and it is better not to break them by showing version 5 while 4 is still supported.

Then if the tool want to update they will use metadata v16.

Agreed that V14 should continue to return the lowest version supported.

jsdw · 2024-10-31T10:25:37Z

substrate/frame/support/procedural/src/construct_runtime/expand/metadata.rs

@@ -117,7 +117,7 @@ pub fn expand_runtime_metadata(
 					pallets: #scrate::__private::vec![ #(#pallets),* ],
 					extrinsic: #scrate::__private::metadata_ir::ExtrinsicMetadataIR {
 						ty,
-						version: <#extrinsic as #scrate::sp_runtime::traits::ExtrinsicMetadata>::VERSION,
+						versions: #scrate::__private::vec![ <#extrinsic as #scrate::sp_runtime::traits::ExtrinsicMetadata>::VERSION ],


Maybe similar to what Niklas asked but, I guess this should now contain 2 versions, 4 and 5, as of #3685 merging?

ExtrinsicMetadata::VERSION should be a vec itself (or static slice).

jsdw · 2024-10-31T10:28:04Z

substrate/primitives/metadata-ir/src/unstable.rs

+			address_ty: ir.address_ty,
+			call_ty: ir.call_ty,
+			signature_ty: ir.signature_ty,
+			extra_ty: ir.extra_ty,


Just a reminder that we should remove this type from V16 (there will be multiple versions of tx extensions, so this type which only points at one version will not be helpful); can't remember if there is an issue for this already :)

jsdw · 2024-10-31T10:29:34Z

substrate/primitives/metadata-ir/src/unstable.rs

+		ExtrinsicMetadata {
+			versions: ir.versions.into_iter().map(Into::into).collect(),
+			address_ty: ir.address_ty,
+			call_ty: ir.call_ty,


Is this the same as OuterEnums.call_enum_ty? if so, can remove this also!

jsdw · 2024-10-31T10:32:02Z

substrate/primitives/metadata-ir/src/types.rs

@@ -170,8 +170,8 @@ pub struct ExtrinsicMetadataIR<T: Form = MetaForm> {
 	///
 	/// Note: Field used for metadata V14 only.
 	pub ty: T::Type,
-	/// Extrinsic version.
-	pub version: u8,


Reminder that extensions below will also need to have a shape more like Vec<(extensions_version, Vec<TransactionExtensionMetadataIR>)> or something, but as with the other things I assume this will be a separate PR :)

jsdw

LGTM!

I saw a couple of things which I assume will be followup PRs, and I suppose this is mainly just about making V16 exposed, but added notes anyways just in case :)

gui1117 · 2024-11-07T10:45:00Z

substrate/primitives/metadata-ir/src/unstable.rs

+			ir.apis.into_iter().map(Into::into).collect(),
+			ir.outer_enums.into(),
+			// Substrate does not collect yet the custom metadata fields.
+			// This allows us to extend the V15 easily.


Suggested change

// This allows us to extend the V15 easily.

// This allows us to extend the V16 easily.

am I wrong?

gui1117 · 2024-11-07T10:47:42Z

substrate/primitives/metadata-ir/src/unstable.rs

+		TransactionExtensionMetadata {
+			identifier: ir.identifier,
+			ty: ir.ty,
+			additional_signed: ir.implicit,


maybe metadata v16 should also rename it to implicit?

Signed-off-by: Alexandru Vasile <[email protected]>

jsdw · 2024-11-07T17:41:32Z

substrate/primitives/metadata-ir/src/unstable.rs

+		// Presume all extensions work for all versions.
+		// This is true for now.
+		let transaction_extensions_by_version = ir
+			.versions


That's a different version, isn't it? ir.versions is the transaction version (4 or 5 nowadays), but transaction)extensions_by_version maps the transaction extension version to the list of extensions that apply.

Thus, at the moment we'll have just one version, 0u8, which maps to the list of transaction extensions we have. I guess the ExtrinsicMetadataIR should be tweaked to record this information though, so that it's just mapped more 1-to-1 into this ExtrinsicMetadata?

Yeah the extension_version is a different version. Here we should for now just assume 0.

jsdw · 2024-11-07T17:43:01Z

substrate/primitives/runtime/src/traits/mod.rs

@@ -1409,10 +1409,10 @@ impl SignaturePayload for () {

 /// Implementor is an [`Extrinsic`] and provides metadata about this extrinsic.
 pub trait ExtrinsicMetadata {
-	/// The format version of the `Extrinsic`.
+	/// The format versions of the `Extrinsic`.
 	///
 	/// By format is meant the encoded representation of the `Extrinsic`.


Suggested change

/// By format is meant the encoded representation of the `Extrinsic`.

/// By format we mean the encoded representation of the `Extrinsic`.

bkchr · 2024-11-08T11:23:58Z

Cargo.toml

@@ -846,7 +846,7 @@ macro_magic = { version = "0.5.1" }
 maplit = { version = "1.0.2" }
 memmap2 = { version = "0.9.3" }
 memory-db = { version = "0.32.0", default-features = false }
-merkleized-metadata = { version = "0.1.0" }
+merkleized-metadata = { git = "https://github.com/lexnv/merkleized-metadata.git", branch = "lexnv/update-frame-md" }


Suggested change

merkleized-metadata = { git = "https://github.com/lexnv/merkleized-metadata.git", branch = "lexnv/update-frame-md" }

merkleized-metadata = "0.1.1"

bkchr · 2024-11-08T11:26:17Z

substrate/primitives/metadata-ir/src/v15.rs

@@ -100,7 +100,7 @@ impl From<TransactionExtensionMetadataIR> for SignedExtensionMetadata {
 impl From<ExtrinsicMetadataIR> for ExtrinsicMetadata {
 	fn from(ir: ExtrinsicMetadataIR) -> Self {
 		ExtrinsicMetadata {
-			version: ir.version,
+			version: *ir.versions.get(0).expect("Metadata V15 supports only one version"),


Suggested change

version: *ir.versions.get(0).expect("Metadata V15 supports only one version"),

version: *ir.versions.iter().min().expect("Metadata V15 supports only one version"),

bkchr · 2024-11-08T11:28:09Z

substrate/primitives/metadata-ir/src/unstable.rs

+		// Presume all extensions work for all versions.
+		// This is true for now.
+		let transaction_extensions_by_version = ir
+			.versions


Yeah the extension_version is a different version. Here we should for now just assume 0.

lexnv added 30 commits August 7, 2024 18:05

metadata-ir: Introduce PalletAssociatedTypeMetadata

7e8baee

Signed-off-by: Alexandru Vasile <[email protected]>

frame/config: Add associated types to parsed config

1c12f15

Signed-off-by: Alexandru Vasile <[email protected]>

frame/expand: Implement associated types expansion

20b22d8

Signed-off-by: Alexandru Vasile <[email protected]>

frame/expand: Use provided cfgs for the associated types

350efce

Signed-off-by: Alexandru Vasile <[email protected]>

frame/construct_runtime: Extract associated types from pallet config

3e97971

Signed-off-by: Alexandru Vasile <[email protected]>

frame/pallet: Introduce config(without_metadata)

5d8021e

Signed-off-by: Alexandru Vasile <[email protected]>

frame/pallet: Introduce #[pallet::include_metadata] for associated

5679244

types Signed-off-by: Alexandru Vasile <[email protected]>

frame/pallet: Include associated type iff bounds contain TypeInfo

958611e

Signed-off-by: Alexandru Vasile <[email protected]>

frame/pallet: Proper flag for metdata collection

7f26b67

Signed-off-by: Alexandru Vasile <[email protected]>

frame/tests/ui: Fix type in test

cc001f3

Signed-off-by: Alexandru Vasile <[email protected]>

frame/tests/ui: Check config without metadata

b2803e2

Signed-off-by: Alexandru Vasile <[email protected]>

frame/tests/ui: Check config with multiple attributes

94007d8

Signed-off-by: Alexandru Vasile <[email protected]>

frame/tests/ui: Add negative test for duplicate attributes in config

75d2697

Signed-off-by: Alexandru Vasile <[email protected]>

frame/tests/ui: Add negative test for collecting metadata from constants

ebcb4a0

Signed-off-by: Alexandru Vasile <[email protected]>

frame/tests/ui: Add negative test for metadata collection on events

5115021

Signed-off-by: Alexandru Vasile <[email protected]>

frame/tests: Check PalletAssociatedTypeMetadataIR collection

aba029a

Signed-off-by: Alexandru Vasile <[email protected]>

frame/support: Add documentation

6502c83

Signed-off-by: Alexandru Vasile <[email protected]>

Add PRdoc

e9571cb

Signed-off-by: Alexandru Vasile <[email protected]>

Merge remote-tracking branch 'origin/master' into lexnv/metadata-v16-…

31bf284

…associated-types

Update prdoc

7429eb7

Signed-off-by: Alexandru Vasile <[email protected]>

prdoc: Remove unneeded crate

8a0c138

Signed-off-by: Alexandru Vasile <[email protected]>

prdoc: Include frame-support

c6c1800

Signed-off-by: Alexandru Vasile <[email protected]>

Prupdate

90c100c

Signed-off-by: Alexandru Vasile <[email protected]>

metadata-ir: Add v16 wrappers in preparation for frame-md release

fc87031

Signed-off-by: Alexandru Vasile <[email protected]>

Merge remote-tracking branch 'origin/master' into lexnv/metadata-v16-…

f627476

…associated-types Signed-off-by: Alexandru Vasile <[email protected]>

metadata-ir: Add conversion methods

b533f60

Signed-off-by: Alexandru Vasile <[email protected]>

metadata-ir: Enable v16 version as unstable

e298259

Signed-off-by: Alexandru Vasile <[email protected]>

metadata-ir: Fix into stable conversion

d0ff8f4

Signed-off-by: Alexandru Vasile <[email protected]>

frame-md: Use git dependency instead

4060637

Signed-off-by: Alexandru Vasile <[email protected]>

metadata-ir: Adjust for new transaction format

adf7508

Signed-off-by: Alexandru Vasile <[email protected]>

lexnv added 5 commits October 25, 2024 12:51

Merge remote-tracking branch 'origin/master' into lexnv/metadata-v16-…

a95183e

…associated-types-enable-unstable Signed-off-by: Alexandru Vasile <[email protected]>

metadata-ir: Adjust metadata

6ff6f37

Signed-off-by: Alexandru Vasile <[email protected]>

metadata-ir: Prepare metadataIR for multiple version collection

fba3502

Signed-off-by: Alexandru Vasile <[email protected]>

Add prdoc

ca2b0f0

Signed-off-by: Alexandru Vasile <[email protected]>

Merge remote-tracking branch 'origin/master' into lexnv/metadata-v16-…

daf8dc3

…associated-types-enable-unstable

lexnv marked this pull request as ready for review October 25, 2024 10:13

lexnv requested review from a team and koute as code owners October 25, 2024 10:13

lexnv mentioned this pull request Oct 25, 2024

cargo: Update frame-metadata to v17 bkchr/merkleized-metadata#3

Merged

pkhry approved these changes Oct 30, 2024

View reviewed changes

niklasad1 reviewed Oct 30, 2024

View reviewed changes

substrate/frame/metadata-hash-extension/Cargo.toml Outdated Show resolved Hide resolved

Update substrate/frame/metadata-hash-extension/Cargo.toml

379ec83

Co-authored-by: Niklas Adolfsson <[email protected]>

niklasad1 reviewed Oct 30, 2024

View reviewed changes

jsdw reviewed Oct 31, 2024

View reviewed changes

jsdw approved these changes Oct 31, 2024

View reviewed changes

gui1117 reviewed Nov 7, 2024

View reviewed changes

lexnv added 6 commits November 7, 2024 15:38

Update cargo lock with pending branches

475288c

Signed-off-by: Alexandru Vasile <[email protected]>

Update prdoc

195e514

Signed-off-by: Alexandru Vasile <[email protected]>

metadata-ir/v14: Populate with lowest supported version

332a900

Signed-off-by: Alexandru Vasile <[email protected]>

metadata-ir/unstable: Adjust for new frame-md version

f9b635d

Signed-off-by: Alexandru Vasile <[email protected]>

extrinsic: Modify ExtrinsicMetadata to return a Vec of versions

ce42813

Signed-off-by: Alexandru Vasile <[email protected]>

metadata-ir: Use IR versions instead of hardcoding

bdf9e7e

Signed-off-by: Alexandru Vasile <[email protected]>

jsdw reviewed Nov 7, 2024

View reviewed changes

bkchr approved these changes Nov 8, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v16: Expose the unstable metadata v16 #5732

v16: Expose the unstable metadata v16 #5732

lexnv commented Sep 16, 2024 •

edited

Loading

niklasad1 Oct 30, 2024

niklasad1 Oct 30, 2024

gui1117 Nov 7, 2024

jsdw Nov 7, 2024

jsdw Oct 31, 2024

gui1117 Nov 7, 2024

jsdw Oct 31, 2024 •

edited

Loading

jsdw Oct 31, 2024

jsdw Oct 31, 2024 •

edited

Loading

jsdw left a comment

gui1117 Nov 7, 2024

gui1117 Nov 7, 2024

jsdw Nov 7, 2024

bkchr Nov 8, 2024

jsdw Nov 7, 2024

bkchr Nov 8, 2024

bkchr Nov 8, 2024

bkchr Nov 8, 2024

	// This allows us to extend the V15 easily.
	// This allows us to extend the V16 easily.

	/// By format is meant the encoded representation of the `Extrinsic`.
	/// By format we mean the encoded representation of the `Extrinsic`.

	merkleized-metadata = { git = "https://github.com/lexnv/merkleized-metadata.git", branch = "lexnv/update-frame-md" }
	merkleized-metadata = "0.1.1"

	version: *ir.versions.get(0).expect("Metadata V15 supports only one version"),
	version: *ir.versions.iter().min().expect("Metadata V15 supports only one version"),

v16: Expose the unstable metadata v16 #5732

Are you sure you want to change the base?

v16: Expose the unstable metadata v16 #5732

Conversation

lexnv commented Sep 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsdw Oct 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsdw Oct 31, 2024 • edited Loading

Choose a reason for hiding this comment

jsdw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lexnv commented Sep 16, 2024 •

edited

Loading

jsdw Oct 31, 2024 •

edited

Loading

jsdw Oct 31, 2024 •

edited

Loading