Metaschema refactor phase 1: towards v2

[wendellpiez]

User Story:

The version 1 Metaschema technology we have developed for OSCAL has helped a great deal to enable workable solutions to problems at several levels; but as it has grown more complex it has also become harder to test and maintain. This is largely due to the complexity of pipelining dependencies in the different processes.

At present the system must, from any given metaschema input, be able to produce for each of XML and JSON:

• A schema: XSD or JSON Schema as applicable
• A conversion XSLT to produce the opposite form (XML or JSON)
• Documentation of the schema tuned to the syntax
• Model maps, again tuned to the syntax

This is currently eight (8) outputs for a single input. We currently have four metaschemas (catalog, profile, SSP and component) making 32 artifacts so far.

A revised pipeline will be simpler than the current implementation because it would do more common processing of metaschema input before splitting. Short version: the schemas and ultimately the conversion transformations can be produced from the same (internal) intermediate representation as the model maps, consolidating logic in one place that is now shared out in many places.

Goals:

A new Metaschema implementation producing all the artifacts above except the conversion pipelines, which will be added in a later phase.

The new implementation must pass extant and new unit tests, and it should be clearer and easier to test and maintain, with better documentation, than the current set of XSLTs.

Processing assumptions regarding Metaschema form and format and applicable constraints should be documented and ideally validated against (Schematron). Also see usnistgov/metaschema-xslt#8.

Metaschema features we have designed and not implemented are in scope (see #19, #31); but new features are not.

Dependencies:

Do we need a working example Metaschema for demo/testing?

Acceptance Criteria

• All website and readme documentation affected by the changes in this issue have been updated. Changes to the website can be made in the docs/content directory of your branch.
• A Pull Request (PR) is submitted that fully addresses the goals of this User Story. This issue is referenced in the PR.
• The CI-CD build process runs without any reported errors on the PR. This can be confirmed by reviewing that all checks have passed in the PR.

[wendellpiez]

A manageable approach to this could be to break the current pipelines out into one or more XProc pipelines, then re-organize and test, then put them back together again under XSLT.

Micropipelining inside the XSLT should be rewritten (externalized) since they are making it more difficult both both to test the logic in isolation, and to reuse it.

[wendellpiez]

All this work proceeds. Currently (in a branch) XSDs and JSON Schemas are successfully emitted from new-model Metaschema under XProc and XSLT; however the Metaschema redesign warrants unit testing.

Unit testing for the XSD production in particular is called for, as that is the big gap.

The smaller gap is documenting how to point the JSON Schema unit testing to the new JSON Schema generator.

[wendellpiez]

March 5 Progress

Things are moving along. Running pipelines, to the new design, produce composed metaschemas, XSD and documentation, including both XML and JSON documentation. Next up, JSON Schema.

[david-waltermire]

@wendellpiez We can close this once #19 and usnistgov/metaschema-xslt#8 are completed.