Converter Generators Do not Emit IDs for Fields in Certain Assembly Contexts #235

aj-stein-nist · 2022-09-16T13:18:06Z

Describe the bug

The XSLT M4 pipeline in this repo is used by the OSCAL repository to generate converter generators themselves. For certain processing values of Metaschema fields in specific contexts, processing the intermediate supermodel data misses in some assembly contexts, like /system-security-plan/system-characteristics/system-id with its @id and optional @identifier-type, filters out and does not pass along the @id key to subsequent pipeline steps. This leads to an error about a malformed map where an element is missing a key, but it is not clear which key.

More detail can be found in usnistgov/oscal-content#139 (comment).

Who is the bug affecting?

NIST OSCAL developers.

What is affected by this bug?

Proper conversion of Metaschema-based content, specifically Metaschema, from XML to JSON.

When does this occur?

Consistently.

How do we replicate the issue?

See Add Examples of OSCAL Actions Assembly oscal-content#139 (comment) for details and explanation of this occurring as-is with CI/CD build sequencing.

Expected behavior (i.e. solution)

The conversion of such data occurs without error, or error messaging is more explicit.

Other Comments

N/A

The text was updated successfully, but these errors were encountered:

As reported in the issue, debugging indicated that filtering the IDs of some fields in array constructs (in the XSLT M4 supermodel intermediate format) are dropping in later stages as a result, lead to invalid JSON serialization.

aj-stein-nist · 2022-10-03T18:18:23Z

It was unclear to me why I had left this unassigned, but it appears I popped it off the stack of the then current sprint (Sprint 58) and left it for "the next sprint," i.e. the now current Sprint 59. I will have to circle around back to this feedback in #236 (comment) and make that bit happen before it is ready for final review.

aj-stein-nist · 2022-10-21T04:29:07Z

As agreed, @david-waltermire-nist, I know you have been back but it's been a busy week with dev and strategic planning. As discussed during planning meeting, I will circle back on this immediately upon my return on Monday 31 October 2022.

As reported in the issue, debugging indicated that filtering the IDs of some fields in array constructs (in the XSLT M4 supermodel intermediate format) are dropping in later stages as a result, lead to invalid JSON serialization.

aj-stein-nist · 2022-11-14T22:41:59Z

Wendell and I spent a lot of last Thursday and this Monday debugging this further and I am having a lot of trouble retriggering the bug in development locally on my workstation and even in the original problem request in usnistgov/oscal-content#139, especially after rebasing its OSCAL submodule (and nested metaschema submodule). It is seeming our original hypothesis was off-target or wrong. Thankfully, taking the time to confirm the bug and a fix with tests brought us to this point. I will continue to plug away on that sample and then localize a similar construct into tests for the associated PR when I can. It is unclear how this evaporated or what I have missed, but I will update tomorrow and hopefully determine next steps then.

aj-stein-nist · 2022-11-14T23:26:15Z

I added a separate issue to track XProc touchups needed as determined from this work in #255.

aj-stein-nist · 2022-11-14T23:27:06Z

Also odd is now I keep testing and will experiment more with filters, maybe the real issue as reported "as part of usnistgov/oscal-content#139" is really #240 and not this issue. 🤦

https://github.com/usnistgov/oscal-content/actions/runs/3465798740/jobs/5788953312

I need to dig deeper but now I am more unsure of myself than before.

aj-stein-nist · 2022-11-18T16:21:45Z

So I forgot to update this issue in the interim. I cannot seem to track this and confirm the bug with the only example in CI/CD appearing to be adjacent to #240 which breaks the CI/CD build before I can see this bug, if at all. Pulling down the artifacts show a valid JSON conversion, so I am still confused. Stepping off this and rolling back to 240 now.

/cc @wendellpiez I forgot say this explicitly here.

aj-stein-nist · 2022-12-05T22:07:40Z

I am going to tentatively move this up to Sprint 61, but it is unclear if this bug can be reproduced in the original context it was reported or not. I will hap to wrap this up after #240.

aj-stein-nist · 2022-12-29T15:17:34Z

I will come back to this and see if I can reproduce the bug that I was unable to find before.

As reported in the issue, debugging indicated that filtering the IDs of some fields in array constructs (in the XSLT M4 supermodel intermediate format) are dropping in later stages as a result, lead to invalid JSON serialization.

aj-stein-nist · 2023-01-04T03:13:43Z

I have found some other unrelated CI/CD issues that are blocking recreating this bug and may have masked the original issue. I am continuing to investigate.

aj-stein-nist · 2023-01-31T21:03:21Z

Removing OSCAL work board per discussion with @david-waltermire-nist.

aj-stein-nist · 2023-02-16T15:20:24Z

It does seem this is rearing it's ugly head in touchups for testing CI/CD upgrades in oscal-content and I am having trouble figuring out what that is. Conversions are failing.

@wendellpiez, let me know your availability, we need to work on this ASAP.

aj-stein-nist · 2023-02-16T22:49:43Z

I was able to pull this down again with a SSP (oscal_leveraging-example_ssp.json) and get back to where we were all those months back. Not sure where that leaves us with testing this issue and remediating in this repo. But is the exact scenario reported here with the JSON XDM to be converted to JSON:

<array key="system-ids"><map><string>saas_system_iaas_customer</string></map></array>

This is what is causing the following error when using the converter generator:

System ID: /home/me/code/json-xdm-to-json.xsl
Scenario: json-xdm-to-json
XSL file: /home/me/code/json-xdm-to-json.xsl
XML file: /home/me/code/gha-logs/oscal/4046515155-6959397364/git-content/src/examples/ssp/xml/oscal_leveraging-example_ssp_json-xdm.xml
Engine name: Saxon-PE 10.6
Severity: fatal
Problem ID: FOJS0006
Description: xml-to-json: Child elements of <map> must have a key attribute
Start location: 9:52
URL: http://www.w3.org/TR/2005/WD-xpath-functions-20050211/#ERRFOJS0006

I am using this basic JSON in XDM form to XML converter transform for debugging the intermediate format.

aj-stein-nist · 2023-02-17T22:44:57Z

It looks like I went down the wrong rabbit hole and maybe the super-model-to-json.xsl stylesheet isn't the issue, certain keys for assemblies in XPath JSON form are missing. I am still investigating this but I resolved the issues with the test-json-conversion.xpl pipeline in incorrect/obsolete paths and figure that out more.

More to follow.

wendellpiez · 2023-02-21T15:00:34Z

The XDM JSON does show a problem in a missing key on the string, as we should have:

<array key="system-ids"><map><string key="THEKEY">saas_system_iaas_customer</string></map></array>

where THEKEY is either in the supermodel (if the bug is in the JSON-writer) or further upstream (if the bug is in the JSON-to-supermodel 'up' step).

wendellpiez · 2023-02-21T15:52:36Z

This might be happening because the system-id object with no flags at all would ordinarily be represented straight up, not as an object property (for the field value), so

<array key="system-ids"><string>saas_system_iaas_customer</string></array>

We need to distinguish between cases where a field casts to an object even when it has only one property (because a flag is permitted) vs when it can be 'naked' since no flags are permitted.

aj-stein-nist · 2023-02-21T16:51:17Z

This might be happening because the system-id object with no flags at all would ordinarily be represented straight up, not as an object property (for the field value), so
<array key="system-ids"><string>saas_system_iaas_customer</string></array>
We need to distinguish between cases where a field casts to an object even when it has only one property (because a flag is permitted) vs when it can be 'naked' since no flags are permitted.

Weirdly our current patch in #236 fixes some of this, but there is something going on with assembly blocks that are converted into groups in the supermodel. I am currently looking into system-information/information-types are holding up the specific example data from the oscal_leveraging_ssp.json that is breaking in the oscal-content CI/CD build that is the origin of this bug report.

In short: sounds right, but there is a little more breakage. Either it moved around, or there is another part we need to fix.

- Updated data types to ensure consistency between specification and implementation. - Identify locations of data type schemas in data type specifications. - Discuss use of regular expression dialects used in XML and JSON schema. #235

As reported in the issue, debugging indicated that filtering the IDs of some fields in array constructs (in the XSLT M4 supermodel intermediate format) are dropping in later stages as a result, lead to invalid JSON serialization.

- Updated data types to ensure consistency between specification and implementation. - Identify locations of data type schemas in data type specifications. - Discuss use of regular expression dialects used in XML and JSON schema. #235

Data type improvements: - Relocated data types into their own file for easier maintenance. - Updated data types to ensure consistency between specification and implementation. - Identify locations of data type schemas in data type specifications. Documentation Improvements: - Discuss use of regular expression dialects used in XML and JSON schema. #235 - Improved documentation around the regex dialects used in the data types. - Added documentation around whitespace restrictions to make clear how the data type implementations restrict leading and trailing whitespace. - Cleaned up and simplified the terminology. - Refactored and made terminology more internally consistent. - Aligned tutorial with updated terminology. - Moved some content out of the overview page into use case page. - Fixed inconsistencies in the tutorial. - Added a complete computer build example. - Removed many OSCAL references. - Added discussion of collapse groups. - Wrapped most terminology in italics. - Improved introduction section and terminology consistency. - Added the following sections: - Conventions Used in this Document: Discusses use of capitalized requirement words. - Graph Theoretical Basis of Metaschema: Discusses how a Metaschema module represents a graph. - Object-Oriented Basis of Metaschema: Discusses how a Metaschema module describes OO classes. - Reorganized specification pages into smaller sections. - Added a syntax outline that better supports reference navigation into the specification. - Some TODO markers remain for work that needs to be completed in a future PR. Publishing an incomplete specification to encourage more review of current materials. Other Improvements: - Integrated new work towards a USWDS 3.x Hugo theme. - Adding NPM files. Added NodeJS setup to website build. - Configured dart-sass to be installed on Hugo build. Had to relocate the package.json, etc to project root to make Hugo postcss work. - Cleaned up some extra unneeded files and corrected some content that was left in error. --------- Co-authored-by: Wendell Piez <[email protected]> Co-authored-by: A.J. Stein <[email protected]>

As reported in the issue, debugging indicated that filtering the IDs of some fields in array constructs (in the XSLT M4 supermodel intermediate format) are dropping in later stages as a result, lead to invalid JSON serialization.

aj-stein-nist added the bug Something isn't working label Sep 16, 2022

aj-stein-nist mentioned this issue Sep 16, 2022

Update Converter Generators to Emit Missing IDs for Fields in Certain Assembly Contexts #236

Merged

7 tasks

david-waltermire linked a pull request Sep 23, 2022 that will close this issue

Update Converter Generators to Emit Missing IDs for Fields in Certain Assembly Contexts #236

Merged

7 tasks

aj-stein-nist self-assigned this Oct 3, 2022

aj-stein-nist added a commit to aj-stein-nist/metaschema that referenced this issue Nov 3, 2022

Add tests for usnistgov#235 regression.

4985b40

aj-stein-nist added a commit to aj-stein-nist/metaschema that referenced this issue Nov 3, 2022

Add tests for usnistgov#235 regression.

86c58e8

aj-stein-nist added a commit to aj-stein-nist/metaschema that referenced this issue Nov 3, 2022

Add tests for usnistgov#235 regression.

ed3159d

aj-stein-nist added a commit to aj-stein-nist/metaschema that referenced this issue Nov 9, 2022

Add tests for usnistgov#235 regression.

910c8c3

aj-stein-nist added a commit to aj-stein-nist/metaschema that referenced this issue Nov 9, 2022

Add tests for usnistgov#235 regression.

549a1f7

aj-stein-nist mentioned this issue Nov 14, 2022

Debugging XProc Pipelines Using Outdated Paths #255

Closed

aj-stein-nist added a commit to aj-stein-nist/metaschema that referenced this issue Dec 29, 2022

Add tests for usnistgov#235 regression.

04bc6e5

This was referenced Dec 30, 2022

Docker Container Doesn't Build with Muti-Stage and Caching apt Packages Across Images usnistgov/OSCAL#1597

Closed

Draft Profiles Resolve Against No Longer Existing Import usnistgov/oscal-content#168

Closed

aj-stein-nist mentioned this issue Jan 6, 2023

Profile Check Schematron Enhancements Fail in GHA Due to Relative Path Issues usnistgov/OSCAL#1603

Closed

david-waltermire added the XSLT Implementation Issue relates to the XSLT implementation of Metaschema. label Feb 9, 2023

aj-stein-nist mentioned this issue Feb 17, 2023

Upgrade GitHub CI/CD Workflows to Current Stable Version usnistgov/oscal-content#116

Closed

6 tasks

aj-stein-nist added a commit to aj-stein-nist/metaschema that referenced this issue Mar 15, 2023

Add tests for usnistgov#235 regression.

4f7fd6e

aj-stein-nist added a commit to aj-stein-nist/metaschema that referenced this issue Mar 15, 2023

Add tests for usnistgov#235 regression.

f29902c

david-waltermire pushed a commit that referenced this issue Mar 27, 2023

Add tests for #235 regression.

b4be227

david-waltermire added this to the Metaschema 0.9.0 milestone Jun 27, 2023

david-waltermire closed this as completed Jun 27, 2023

nikitawootten-nist pushed a commit to nikitawootten-nist/metaschema-xslt that referenced this issue Jul 21, 2023

Add tests for usnistgov/metaschema#235 regression.

893ba38

github-actions bot mentioned this issue Dec 21, 2023

Scheduled Check of Markdown Documents Found Bad Hyperlinks usnistgov/OSCAL#1971

Closed

github-actions bot mentioned this issue Feb 15, 2024

Scheduled Check of Markdown Documents Found Bad Hyperlinks usnistgov/OSCAL#1981

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Converter Generators Do not Emit IDs for Fields in Certain Assembly Contexts #235

Converter Generators Do not Emit IDs for Fields in Certain Assembly Contexts #235

aj-stein-nist commented Sep 16, 2022

aj-stein-nist commented Oct 3, 2022

aj-stein-nist commented Oct 21, 2022

aj-stein-nist commented Nov 14, 2022

aj-stein-nist commented Nov 14, 2022

aj-stein-nist commented Nov 14, 2022 •

edited

Loading

aj-stein-nist commented Nov 18, 2022

aj-stein-nist commented Dec 5, 2022

aj-stein-nist commented Dec 29, 2022

aj-stein-nist commented Jan 4, 2023

aj-stein-nist commented Jan 31, 2023

aj-stein-nist commented Feb 16, 2023

aj-stein-nist commented Feb 16, 2023

aj-stein-nist commented Feb 17, 2023

wendellpiez commented Feb 21, 2023

wendellpiez commented Feb 21, 2023

aj-stein-nist commented Feb 21, 2023

Converter Generators Do not Emit IDs for Fields in Certain Assembly Contexts #235

Converter Generators Do not Emit IDs for Fields in Certain Assembly Contexts #235

Comments

aj-stein-nist commented Sep 16, 2022

Describe the bug

Who is the bug affecting?

What is affected by this bug?

When does this occur?

How do we replicate the issue?

Expected behavior (i.e. solution)

Other Comments

aj-stein-nist commented Oct 3, 2022

aj-stein-nist commented Oct 21, 2022

aj-stein-nist commented Nov 14, 2022

aj-stein-nist commented Nov 14, 2022

aj-stein-nist commented Nov 14, 2022 • edited Loading

aj-stein-nist commented Nov 18, 2022

aj-stein-nist commented Dec 5, 2022

aj-stein-nist commented Dec 29, 2022

aj-stein-nist commented Jan 4, 2023

aj-stein-nist commented Jan 31, 2023

aj-stein-nist commented Feb 16, 2023

aj-stein-nist commented Feb 16, 2023

aj-stein-nist commented Feb 17, 2023

wendellpiez commented Feb 21, 2023

wendellpiez commented Feb 21, 2023

aj-stein-nist commented Feb 21, 2023

aj-stein-nist commented Nov 14, 2022 •

edited

Loading