Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update JSON-XML conversion script production to current Metaschema model #53

Closed
3 tasks
wendellpiez opened this issue May 14, 2020 · 3 comments
Closed
3 tasks
Assignees
Labels
closable Issue is n/a or satisfied enhancement New feature or request XSLT Implementation Issue relates to the XSLT implementation of Metaschema.

Comments

@wendellpiez
Copy link
Collaborator

User Story:

As a developer I need a robust means to produce data converters for Metaschema-defined OSCAL, from XML to JSON and back. (YAML can potentially be supported as an extension on the JSON side.)

As a user I need to be able to perform such conversions and trust they work reliably.

Goals:

Having a new-improved XML-to-JSON pathway (see #51), we also need to go the other way, converting OSCAL JSON into the equivalent XML.

This capability was first prototyped for Milestone 1. However now we have a superior approach, much easier to maintain and test, as well as supporting new Metaschema features such as local definitions.

The new architecture performs conversion by transforming either XML or JSON into a common supermodel representation, which can then be (relatively trivially) serialized back into either syntax.

Dependencies:

For #51 we have built out most of the infrastructure for this including the supermodel serializers. Indeed the JSON -> XML conversion for a given Metaschema might conceivably be derived from the XML->JSON converter, only with match patterns and select expressions mapped from the XML variant into the JSON variant paths.

However, to port this logic over we also need to be able to convert paths addressing the XML into the Metaschema-equivalent paths for the JSON. Development of this capability is being tracked in usnistgov/metaschema-xslt#13.

Acceptance Criteria

  • All website and readme documentation affected by the changes in this issue have been updated. Changes to the website can be made in the docs/content directory of your branch.
  • A Pull Request (PR) is submitted that fully addresses the goals of this User Story. This issue is referenced in the PR.
  • The CI-CD build process runs without any reported errors on the PR. This can be confirmed by reviewing that all checks have passed in the PR.

{The items above are general acceptance criteria for all User Stories. Please describe anything else that must be completed for this issue to be considered resolved.}

@wendellpiez wendellpiez added the enhancement New feature or request label May 14, 2020
wendellpiez added a commit to wendellpiez/metaschema that referenced this issue Jul 8, 2020
…version. Metapath mapping corrections. Nearly working round-trip test conversion.
@wendellpiez
Copy link
Collaborator Author

Templates are being produced to produce "abstracted supermodel" representation from JSON source. They do not yet capture all the data correctly, for example in the case of fields whose values are given directly (because having no without flags), and perhaps in other positions. But close. Still working.

Test data running through a round-trip conversion - XML to JSON back to XML - serves for unit testing.

See usnistgov/OSCAL#633.

wendellpiez added a commit to wendellpiez/metaschema that referenced this issue Jul 10, 2020
wendellpiez added a commit to wendellpiez/metaschema that referenced this issue Jul 14, 2020
…m metaschema source. Many bug repairs and extensions. usnistgov#53.
david-waltermire pushed a commit that referenced this issue Sep 17, 2020
…mapping corrections. Nearly working round-trip test conversion.
david-waltermire pushed a commit that referenced this issue Sep 17, 2020
…m metaschema source. Many bug repairs and extensions. #53.
wendellpiez added a commit to wendellpiez/metaschema that referenced this issue Nov 10, 2020
…version. Metapath mapping corrections. Nearly working round-trip test conversion.
wendellpiez added a commit to wendellpiez/metaschema that referenced this issue Nov 10, 2020
wendellpiez added a commit to wendellpiez/metaschema that referenced this issue Nov 10, 2020
…m metaschema source. Many bug repairs and extensions. usnistgov#53.
david-waltermire pushed a commit that referenced this issue Nov 13, 2020
…mapping corrections. Nearly working round-trip test conversion.
david-waltermire pushed a commit that referenced this issue Nov 13, 2020
…m metaschema source. Many bug repairs and extensions. #53.
david-waltermire pushed a commit that referenced this issue Dec 21, 2020
…mapping corrections. Nearly working round-trip test conversion.
david-waltermire pushed a commit that referenced this issue Dec 21, 2020
…m metaschema source. Many bug repairs and extensions. #53.
wendellpiez added a commit that referenced this issue May 26, 2021
david-waltermire pushed a commit that referenced this issue May 27, 2021
* Many adjustments toward correct converter generation, though still broken - start with unit test toolchains/xslt-M4/metapath/metapath-jsonizing.xspec - this is a fail-safe commit

* Now appears to pass testing scenarios (but more testing is coming)

* Committing a version that handles test data correctly (so far) from rebuilt metaschema composition addressing #51 #53 #76
david-waltermire added a commit that referenced this issue Jun 6, 2021
* Rework of docs focusing on JSON docs and model pipeline
* Improvements to composition toolchain
* Fixed a few small bugs in the metaschema-check. Improved performance of the compose pruning using an accumulator.
* Moved edge-case samples into testing directory
* Made shadowing warning a warning
* Initial commit of an Oxygen Metaschema framework.
* Creation of new compose schematron unit tests.
* Cross-linking XML and JSON syntax pages and other improvements to links
* Now building XML and JSON indexes to reference pages, with links to steps
* Reconfigured docs pipeline (XSLT entry points); adding new files including pipeline steps
* Migrating schema generation tools to new/improved composition pipeline
* Addressing usnistgov/OSCAL#902 thanks for finding this bug
* Enhancements to JSON Schema definition (with better performance too)
* Adding support for json-base-uri as a metaschema property
* Updated JSON schema $id; factoring out common docs XSLT
* Fixing IDs in JSON schema per issue usnistgov/OSCAL#933.
* Addressing datatype validation issues: whitespace collapsing; non-empty values; ncname-workalike in JSON Schema - see usnistgov/OSCAL#911  usnistgov/OSCAL#805 also #33 #67 #68
* Improvements to XSD production; fully aligning 'token' datatype across XSD and JSON Schema implementations.
* Updating bidirectional XML/JSON converter generators (#143)
* Committing a version that handles test data correctly (so far) from rebuilt metaschema composition addressing #51 #53 #76
* Now displaying constraints in documentation at point of definition;
* Docs generation revamp Reworked reference and other pages to sketch - #128 and others

Co-authored-by: Wendell Piez <[email protected]>
@david-waltermire
Copy link
Collaborator

This should be updated based on the changes summarized in #195.

@david-waltermire david-waltermire added the XSLT Implementation Issue relates to the XSLT implementation of Metaschema. label Apr 12, 2022
@david-waltermire david-waltermire added this to the Metaschema 0.9.0 milestone Apr 12, 2022
@david-waltermire david-waltermire removed this from the Metaschema 0.9.0 milestone Feb 9, 2023
@wendellpiez
Copy link
Collaborator Author

Work has been done, currently stable.

@wendellpiez wendellpiez added the closable Issue is n/a or satisfied label Feb 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
closable Issue is n/a or satisfied enhancement New feature or request XSLT Implementation Issue relates to the XSLT implementation of Metaschema.
Projects
None yet
Development

No branches or pull requests

2 participants