Tracker ticket for profile resolution samples

[bradh]

Describe the bug

Profile resolution is being worked by @wendellpiez over on #559. That has some "known answer" files for the result of processing / resolving the profiles. Not all of them are necessarily correct. This ticket tracks status

Who is the bug affecting?

At this stage it only affects those working in #559. Indirectly it affects everyone who is trying to use the profile resolution logic.

What is affected by this bug?

This is a blocker to completion of #559.

When does this occur?

On resolution.

Files that are OK (per spec, noting #579 and #580)

• broken_profile.xml
• circular_profile.xml
• home_profile.xml

Files needing work

• base2-test_profile.xml has an incorrect catalog link. See Fix catalogs path for profile examples wendellpiez/OSCAL#12
• output-expected/base-test_profile-RESOLVED.xml is missing the sub-sub-control at the end. See Update "expected" profile resolver output for the base tests. wendellpiez/OSCAL#15
• output-expected/base2-test_profile-RESOLVED.xml should look the same as base-test_profile-RESOLVED.xml except for the metadata link href attribute value. See Update "expected" profile resolver output for the base tests. wendellpiez/OSCAL#15
• output-expected/exclude-call-test-test_profile_RESOLVED.xml is missing the groups from the catalog. See Update expected resolution result for exclude-call-test profile example wendellpiez/OSCAL#16
• output-expected/full-test_profile_RESOLVED.xml is not schema valid. It needs to include the controls inside their parent group. Also looks like some punctuation differences. Possibly other changes. See Fix profile resolver results for full-test_profile.xml example wendellpiez/OSCAL#14
• output-expected/include-all-no-children-test_profile_RESOLVED.xml is not schema valid. It looks like it needs to include the controls and parameters inside their parent group. See Update the all-no-children profile resolution example result wendellpiez/OSCAL#20
• output-expected/include-all-test_profile_RESOLVED.xml is not schema valid. It looks like it needs to include the controls and parameters inside their parent group. See Update all-test profile resolution result. wendellpiez/OSCAL#17
• output-expected/include-all-implicit-test_profile_RESOLVED.xml is missing the groups and controls from the catalog. See Update the expected implicit "all" profile resolver result. wendellpiez/OSCAL#18
• output-expected/include-call-with-children-test_profile_RESOLVED.xml is missing the parent groups and the last control. See Update include-call-with-children profile resolution example wendellpiez/OSCAL#21
• include-loose-param-test_profile.xml looks like it is importing from the wrong catalog: if its meant to result in a loose param, then it needs to come from abc-full-catalog.xml rather than abc-simple-catalog.xml. See Fix catalog reference in loose param test profile example wendellpiez/OSCAL#13
• output-expected/include-loose-param-test_profile_RESOLVED.xml is missing the parent group and param entries. See Update loose param profile resolution example wendellpiez/OSCAL#19
• merge-implicit-keep_profile.xml has the wrong resource href value (missing the catalogs/ part). See Fix catalogs path for profile examples wendellpiez/OSCAL#12
• output-expected/merge-implicit-keep_profile_RESOLVED.xml is missing the groups and controls. See Update merge-keep profile resolution examples. wendellpiez/OSCAL#22
• merge-keep_profile.xml has the wrong resource href value (missing the catalogs/ part). See Fix catalogs path for profile examples wendellpiez/OSCAL#12
• output-expected/merge-keep_profile_RESOLVED.xml is missing the groups and controls. See Update merge-keep profile resolution examples. wendellpiez/OSCAL#22

Files to be analysed

• import-twice_profile.xml

Other test profiles we might need

• More checks for citation logic (spec section 3.6) - that is quite complicated
• Controls within nested groups (say three or four levels deep)
• Selecting a sub-control without explicitly selecting parent control(s)

Other things we should / could do

• Fix up the versions to match the current or next milestone?
• Validate all the Profile files against the schema
• Validate all the Catalog files against the schema
• Validate all the expected output files against the schema
• Convert all the files to equivalent JSON files
• Convert all the files to equivalent YAML files

[david-waltermire]

@bradh I think @wendellpiez accepted all your PRs. Can you verify and check them off above?

I am also working on integrating the profile resolution unit tests into the CI/CD (#561), which should cover some aspects of the "Other things we should / could do".

We still need to fix the versions.

[wendellpiez]

The PRs into my repository have been merged. If all goes according to plan, corresponding failures of the current (unfinished) resolver should next emerge in unit testing (see #561), wherever @bradh has made corrections (even arguable ones) to the tests.

This will give us our first set of corrections to make to the resolver, but more importantly, opportunity to flesh out and correct the relevant language in the specification.

If for whatever reason the unit tests pass (call me skeptical, but maybe I messed up the PRs?) we can plunge onward to address list items and make more tests as noted. (Along with the spec work.)

[bradh]

Its certainly possible I messed up the tests too - could be poor typing, or poor understanding. Concur with the opportunity to clarify (and hopefully simplify) the spec.

[wendellpiez]

Sprint 26 Update Jan 30 2020

Profile resolution is stable if unfinished in my branch. See PR #559. I could continue to make forward progress on testing anytime but the critical path goes through unit testing, so dependent on #561.

[wendellpiez]

Taking on a work item to produce an Issue for stabilizing profile resolution in view of recent schema/model changes (especially #568 set-param but also #612 (resource and citation).

[wendellpiez]

Sprint 27 Update Feb 13 2020

Now we have two Issues -- #622 and #625.

[wendellpiez]

March 12 Update

Still pending integration work.