Code coverage metrics

[naveensrinivasan]

The code coverage has been downward trending in these last six months.

https://app.codecov.io/gh/ossf/scorecard

It was at 58%, and now (8/27/2022) is around 42%.

My recommendation is to enforce coverage metrics to have higher levels of reliability. This is critical for NEW contributors as it gives them confidence in their patches.

[naveensrinivasan]

@ossf/scorecard-maintainers FYI...

[naveensrinivasan]

Based on our bi-weekly meeting (9/8/22) I am enabling the Code Coverage as a required check
Let’s enable “fail PR if test coverage is reduced”; we’ll separately investigate if E2E testing is included in the test coverage We can override in specific cases if warranted, but normally shouldn’t need to override

[spencerschrock]

One thing I noticed, as part of #2195, is that tests outside the package aren't counted towards coverage.

For example line 34 of line 34 in checks/evaluation/security_policy.go was marked as not having coverage.
https://github.com/ossf/scorecard/pull/2195/files#annotation_5755382660. There were plenty of tests added in checks/security_policy_test.go, but the coverage profiles generated by go test don't count cross-package tests.

It's possible to count cross package tests if that's something we'd want to try by adding -coverpkg=./... to the unit-test target in the Makefile

[spencerschrock]

Closing as Naveen got us up to 75%, and our codecov check looks for patch coverage around there so that we maintain it.