Json schemas for VSCode YAML Support

[haixuanTao]

In an effort to address, developers difficulty in discovering dora-rs functionality, I'm adding a way to:

• Generate JSON Schema: https://json-schema.org/ using schemars
• JSON Schema can be interpreted by many extension to give us YAML validation, documentation and suggestions. In my case, I'm using VSCode most popular extension for YAML which is redhat.vscode-yaml

This enables:

• Documentation that is directly generated from the docstring within the rust code
• Suggestions from the field defined in the Rust Code
• Validation of type

Edge cases

Failure to auto-generate schema

Some auto-generation of some of our rust struct definition did not work. They are the one that are using serde(from= such as:

dora/libraries/core/src/descriptor/mod.rs

Lines 270 to 280 in 558fa14

	}
	#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, JsonSchema)]
	#[serde(
	deny_unknown_fields,
	from = "PythonSourceDef",
	into = "PythonSourceDef"
	)]
	pub struct PythonSource {
	pub source: String,
	pub conda_env: Option<String>,
	}

In those case I had to remove the validation so that our current dataflow passes.

Failure to manage #[serde(flatten)] with additional properties

In our Nodes definition, we are using #[serde(flatten)] of an enum with some additional properties.

In order to validate those additional properties, I had to set additionalproperties to true.

Demo

2024-05-15.15-43-22.mp4

[phil-opp]

Do we want to have this (and other) _unstable_* fields in the schema? If not, we should be able to ignore them using a #[schemars(skip)] attribute.

I'm also fine with keeping them in, as the field names makes it clear that they are unstable.

[haixuanTao]

Okay, in the end I have removed unstable features as to not overcrowd: autocomplete and push people on features not yet complete.

[phil-opp]

Why is this replace needed?

[haixuanTao]

// Add additional properties to True, as #[derive(transparent)] of enums are not well handled.
//
// 'OneOf' such as Custom Nodes, Operators and Single Operators overwrite property values of the initial struct `Nodes`.`
// which make the original properties such as `id` and `name` not validated by IDE extensions.

I have added documentation but basically its because we use #[derive(transparent)] on an enum, and the parent properties get forgotten in the JSON Schema.

[phil-opp]

Thanks!