-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Converter Generators Do not Emit IDs for Fields in Certain Assembly Contexts #235
Comments
As reported in the issue, debugging indicated that filtering the IDs of some fields in array constructs (in the XSLT M4 supermodel intermediate format) are dropping in later stages as a result, lead to invalid JSON serialization.
It was unclear to me why I had left this unassigned, but it appears I popped it off the stack of the then current sprint (Sprint 58) and left it for "the next sprint," i.e. the now current Sprint 59. I will have to circle around back to this feedback in #236 (comment) and make that bit happen before it is ready for final review. |
As agreed, @david-waltermire-nist, I know you have been back but it's been a busy week with dev and strategic planning. As discussed during planning meeting, I will circle back on this immediately upon my return on Monday 31 October 2022. |
As reported in the issue, debugging indicated that filtering the IDs of some fields in array constructs (in the XSLT M4 supermodel intermediate format) are dropping in later stages as a result, lead to invalid JSON serialization.
As reported in the issue, debugging indicated that filtering the IDs of some fields in array constructs (in the XSLT M4 supermodel intermediate format) are dropping in later stages as a result, lead to invalid JSON serialization.
Wendell and I spent a lot of last Thursday and this Monday debugging this further and I am having a lot of trouble retriggering the bug in development locally on my workstation and even in the original problem request in usnistgov/oscal-content#139, especially after rebasing its OSCAL submodule (and nested metaschema submodule). It is seeming our original hypothesis was off-target or wrong. Thankfully, taking the time to confirm the bug and a fix with tests brought us to this point. I will continue to plug away on that sample and then localize a similar construct into tests for the associated PR when I can. It is unclear how this evaporated or what I have missed, but I will update tomorrow and hopefully determine next steps then. |
I added a separate issue to track XProc touchups needed as determined from this work in #255. |
Also odd is now I keep testing and will experiment more with filters, maybe the real issue as reported "as part of usnistgov/oscal-content#139" is really #240 and not this issue. 🤦 https://github.com/usnistgov/oscal-content/actions/runs/3465798740/jobs/5788953312 I need to dig deeper but now I am more unsure of myself than before. |
So I forgot to update this issue in the interim. I cannot seem to track this and confirm the bug with the only example in CI/CD appearing to be adjacent to #240 which breaks the CI/CD build before I can see this bug, if at all. Pulling down the artifacts show a valid JSON conversion, so I am still confused. Stepping off this and rolling back to 240 now. /cc @wendellpiez I forgot say this explicitly here. |
I am going to tentatively move this up to Sprint 61, but it is unclear if this bug can be reproduced in the original context it was reported or not. I will hap to wrap this up after #240. |
I will come back to this and see if I can reproduce the bug that I was unable to find before. |
As reported in the issue, debugging indicated that filtering the IDs of some fields in array constructs (in the XSLT M4 supermodel intermediate format) are dropping in later stages as a result, lead to invalid JSON serialization.
I have found some other unrelated CI/CD issues that are blocking recreating this bug and may have masked the original issue. I am continuing to investigate. |
Removing OSCAL work board per discussion with @david-waltermire-nist. |
It does seem this is rearing it's ugly head in touchups for testing CI/CD upgrades in oscal-content and I am having trouble figuring out what that is. Conversions are failing. @wendellpiez, let me know your availability, we need to work on this ASAP. |
I was able to pull this down again with a SSP ( <array key="system-ids"><map><string>saas_system_iaas_customer</string></map></array> This is what is causing the following error when using the converter generator:
I am using this basic JSON in XDM form to XML converter transform for debugging the intermediate format. |
It looks like I went down the wrong rabbit hole and maybe the More to follow. |
The XDM JSON does show a problem in a missing key on the
where |
This might be happening because the
We need to distinguish between cases where a field casts to an object even when it has only one property (because a flag is permitted) vs when it can be 'naked' since no flags are permitted. |
Weirdly our current patch in #236 fixes some of this, but there is something going on with In short: sounds right, but there is a little more breakage. Either it moved around, or there is another part we need to fix. |
- Updated data types to ensure consistency between specification and implementation. - Identify locations of data type schemas in data type specifications. - Discuss use of regular expression dialects used in XML and JSON schema. #235
- Updated data types to ensure consistency between specification and implementation. - Identify locations of data type schemas in data type specifications. - Discuss use of regular expression dialects used in XML and JSON schema. #235
As reported in the issue, debugging indicated that filtering the IDs of some fields in array constructs (in the XSLT M4 supermodel intermediate format) are dropping in later stages as a result, lead to invalid JSON serialization.
As reported in the issue, debugging indicated that filtering the IDs of some fields in array constructs (in the XSLT M4 supermodel intermediate format) are dropping in later stages as a result, lead to invalid JSON serialization.
As reported in the issue, debugging indicated that filtering the IDs of some fields in array constructs (in the XSLT M4 supermodel intermediate format) are dropping in later stages as a result, lead to invalid JSON serialization.
- Updated data types to ensure consistency between specification and implementation. - Identify locations of data type schemas in data type specifications. - Discuss use of regular expression dialects used in XML and JSON schema. #235
- Updated data types to ensure consistency between specification and implementation. - Identify locations of data type schemas in data type specifications. - Discuss use of regular expression dialects used in XML and JSON schema. #235
- Updated data types to ensure consistency between specification and implementation. - Identify locations of data type schemas in data type specifications. - Discuss use of regular expression dialects used in XML and JSON schema. #235
- Updated data types to ensure consistency between specification and implementation. - Identify locations of data type schemas in data type specifications. - Discuss use of regular expression dialects used in XML and JSON schema. #235
- Updated data types to ensure consistency between specification and implementation. - Identify locations of data type schemas in data type specifications. - Discuss use of regular expression dialects used in XML and JSON schema. #235
Data type improvements: - Relocated data types into their own file for easier maintenance. - Updated data types to ensure consistency between specification and implementation. - Identify locations of data type schemas in data type specifications. Documentation Improvements: - Discuss use of regular expression dialects used in XML and JSON schema. #235 - Improved documentation around the regex dialects used in the data types. - Added documentation around whitespace restrictions to make clear how the data type implementations restrict leading and trailing whitespace. - Cleaned up and simplified the terminology. - Refactored and made terminology more internally consistent. - Aligned tutorial with updated terminology. - Moved some content out of the overview page into use case page. - Fixed inconsistencies in the tutorial. - Added a complete computer build example. - Removed many OSCAL references. - Added discussion of collapse groups. - Wrapped most terminology in italics. - Improved introduction section and terminology consistency. - Added the following sections: - Conventions Used in this Document: Discusses use of capitalized requirement words. - Graph Theoretical Basis of Metaschema: Discusses how a Metaschema module represents a graph. - Object-Oriented Basis of Metaschema: Discusses how a Metaschema module describes OO classes. - Reorganized specification pages into smaller sections. - Added a syntax outline that better supports reference navigation into the specification. - Some TODO markers remain for work that needs to be completed in a future PR. Publishing an incomplete specification to encourage more review of current materials. Other Improvements: - Integrated new work towards a USWDS 3.x Hugo theme. - Adding NPM files. Added NodeJS setup to website build. - Configured dart-sass to be installed on Hugo build. Had to relocate the package.json, etc to project root to make Hugo postcss work. - Cleaned up some extra unneeded files and corrected some content that was left in error. --------- Co-authored-by: Wendell Piez <[email protected]> Co-authored-by: A.J. Stein <[email protected]>
As reported in the issue, debugging indicated that filtering the IDs of some fields in array constructs (in the XSLT M4 supermodel intermediate format) are dropping in later stages as a result, lead to invalid JSON serialization.
Describe the bug
The XSLT M4 pipeline in this repo is used by the OSCAL repository to generate converter generators themselves. For certain processing values of Metaschema
field
s in specific contexts, processing the intermediate supermodel data misses in some assembly contexts, like/system-security-plan/system-characteristics/system-id
with its@id
and optional@identifier-type
, filters out and does not pass along the@id
key to subsequent pipeline steps. This leads to an error about a malformed map where an element is missing a key, but it is not clear which key.More detail can be found in usnistgov/oscal-content#139 (comment).
Who is the bug affecting?
NIST OSCAL developers.
What is affected by this bug?
Proper conversion of Metaschema-based content, specifically Metaschema, from XML to JSON.
When does this occur?
Consistently.
How do we replicate the issue?
Expected behavior (i.e. solution)
The conversion of such data occurs without error, or error messaging is more explicit.
Other Comments
N/A
The text was updated successfully, but these errors were encountered: