Modularize and standardize GitHub Actions workflows

[aj-stein-nist]

Committer Notes

This work closes #1223.

Improvements of note:

• Properly configuring dependabot updates for necessary package ecosystems supported:
  • git submodules
  • GitHub Actions
  • Docker
  • npm for NodeJS
  • pip for Python
  • mvn and pom.xml for Java utilities
  • determine if there is a go mod or Golang method for managing the hugo-extended dependency as well
• Consistent handling of GitHub Action versions by git SHA1 manifest and not mutable tags, to follow best practices
• Preparation for use of reusable workflows by other downstream repos such as metaschema, oscal-content, and GSA/fedramp-automation
• Review of error-handling logic as applicable and part of Review Error Behavior in OSCAL CI/CD Scripts and GitHub Actions Automation #1222

All Submissions:

• Have you selected the correct base branch per Contributing guidance?
• Have you set "Allow edits and access to secrets by maintainers
  "?
• Have you checked to ensure there aren't other open Pull Requests for the same update/change?
• Have you squashed any non-relevant commits and commit messages? [instructions]
• Do all automated CI/CD checks pass?

By submitting a pull request, you are agreeing to provide this contribution under the CC0 1.0 Universal public domain dedication.

Changes to Core Features:

• Have you added an explanation of what your changes do and why you'd like us to include them?
• Have you written new tests for your core changes, as applicable?
• Have you included examples of how to use your new feature(s)?
• Have you updated all OSCAL website and readme documentation affected by the changes you made? Changes to the OSCAL website can be made in the docs/content directory of your branch.

[aj-stein-nist]

Talked with Dave and team during the status update. He would prefer I continue to keep it as agile and scoped as possible.

He recommended and I agree to not build separate Actions in unique, independent repos and continue to move forward with reusable workflows as mentioned above, as opposed to "composite" GH Actions as an alternative. Dave also requested I keep an eye on properly handling the dependency requirements for Saxon and XML Calabash properly with Maven as just Maven and Saxon (and not their transitive deps) will not cut it. Also, Dave wants me to find a solution but also try to complete around the ETA of Tuesday (July 5th). If I can find a path forward in the short-term, worth considering it, if not, just keep current non-Maven approach.

More updates to come and will resync with Dave and team accordingly.

[aj-stein-nist]

Dave and I converged on a good mvn installation with pox.xml and some improved configuration recs from Dave. I am getting some weird errors with ./OSCAL/build/scripts/run-all.sh from local container testing, need to match with GitHub Actions deps install and config and workaround /opt/oscal permissions. Getting closer.

[aj-stein-nist]

Making progress on the Go bit of this from local testing and will finish clean up of that and port it to the GitHub Actions workflow now to wrap that up and finish clean up of deps and pre-req of different GHA workflows and their respective jobs.

[aj-stein-nist]

I have been testing dependabot functionality and I will be removing the comment after uses: action/example-action@3cea5372237819ed00197afe530f5a7ea1234 # current: tag name because dependabot does not update the comments and this will get increasingly confusing for GH Action modules. It does update them and properly identify them in the PR though.

[aj-stein-nist]

Wrapping up testing, seems lycheeverse/lychee#631 is not in place where I can get it to work the way I want to deprecate markdown-link-check entirely. Going to move this along and stabilize it, check off error management up and wrap up for final review.

[mre]

Hey @aj-stein-nist, sorry for chiming in.
I saw your link to lycheeverse/lychee#631 and wanted to see if I could help with the lychee config.
Looking at your commit, I can see that there's a typo in the excludes section:

exclude = [
    ""https://defense.gov/",
    "http://fedramp.gov/ns/oscal",
    "https://fedramp.gov/ns/oscal",
    "http://www.first.org/cvss/v2.0",
    "http://www.first.org/cvss/v3.0",
    "http://www.first.org/cvss/v3.1",
    "https://tools.ietf.org/html.*",
    "http://csrc.nist.gov/ns/.*",
    "http://csrc.nist.gov/oscal",
    "https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/xxxx",
    "https://cdn.telos.com/wp-content/uploads/2021/06/22150746/Xacta-360-EULA-US.pdf",
    "https://search.usa.gov/search"
]

At the beginning of the first line it should be one quote instead of two. 😉
Source:
https://github.com/usnistgov/OSCAL/compare/13f6c714bc87ad3d5e3d23a2e2a71c296f428c6d..832673e2a3cc21baedd9ba54408f020e07451ba2

It's a shot in the dark, but maybe that's what's causing the issue. Maybe you want to try that.
However, don't see this as a blocker for the merge of course; we can also change it in a follow-up PR.

[aj-stein-nist]

It's a shot in the dark, but maybe that's what's causing the issue. Maybe you want to try that. However, don't see this as a blocker for the merge of course; we can also change it in a follow-up PR.

Thanks, @mre. I will follow up. I did notice that typo, and got tired of waiting for latency of GH Actions, so I tested the binary locally after that fixing that typo and didn't push it up here. I just pulled the code out for now. Pushed the proper config back up though in the backup branch I made last night to handle that for later.

When I run this locally in Git Bash (to be lazy) with this command and still got remap specific error around ../../projects and didn't know why. Not only is the remap pattern not working, the wildcard file argument doesn't seem to work for me.

But I only tried for a bit. Will follow up later.

ajs16@MYCOMPUTERNAME MINGW64 ~/code/OSCAL (1223-modularize-all-github-actions-setups-backup-lychee-config-remap-experiments)
$ lychee --config ./build/config/lychee_config.toml './**/*.md' --exclude .github
? [EXCLUDED] file:///C:/Users/ajs16/code/OSCAL/.github/workflows | Excluded
? [EXCLUDED] file:///C:/Users/ajs16/code/OSCAL/.github/CODEOWNERS | Excluded
? [EXCLUDED] file:///C:/Users/ajs16/code/OSCAL/.github/ISSUE_TEMPLATE | Excluded
? [EXCLUDED] file:///C:/Users/ajs16/code/OSCAL/.github/issue_template.md | Excluded
? [EXCLUDED] file:///C:/Users/ajs16/code/OSCAL/.github/ISSUE_TEMPLATE/question.md | Excluded
? [EXCLUDED] file:///C:/Users/ajs16/code/OSCAL/.github/ISSUE_TEMPLATE/feature_request.md | Excluded
? [EXCLUDED] file:///C:/Users/ajs16/code/OSCAL/.github/ISSUE_TEMPLATE/bug_report.md | Excluded
? [EXCLUDED] https://github.com/usnistgov/OSCAL/discussions/categories/q-a | Excluded
? [EXCLUDED] https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork | Excluded
? [EXCLUDED] https://github.com/usnistgov/OSCAL/blob/main/CONTRIBUTING.md | Excluded
? [EXCLUDED] https://github.com/usnistgov/OSCAL/pulls | Excluded
Error: Invalid path to URL conversion: projects