tests: Include testing code in code coverage

[webmiche]

In a recent PR the question came up whether we want to include the testing code in the code coverage.

I am not sure what makes more sense here. From a conceptual perspective, I feel that the code coverage should only contain parts of the library and I don't consider the tests part of that. However, this has the advantage of knowing which tests are actually executed, which can be very insightful as well.

Notice how this influences the code coverage as well. Writing testing code (that obviously gets executed) will always improve coverage, so the exact number is always higher in this case and there is a bias to write a lot of tests (maybe very dumb ones) to increase the code coverage.

[math-fehr]

I would argue that if we can do it easily, we should try to cover the testing code as well.
The reason is mostly to have an idea if the tests themselves are sensible, and if they make sense.
Otherwise, what would be the reason in having unreached code in testing?

However, I do not have strong feelings about this, so both options are fine for me. After all, we only "publish" the non-testing code.

[webmiche]

How do you plan to include it into the CI run though? I guess we could try to enforce that it is always >99% or something...

[math-fehr]

Well, currently, we just report the coverage on each run, maybe we should just do the same for testing?

[webmiche]

In which way report it is my question. There is (a) what is in the logs of the CI and there is (b) the stuff on the coverage page and the PR comment that it updates. Having (a) is pretty much free, I am not sure about (b) though as I haven't set that up personally. Only having (a) though is not very insightful, that's why I suggested to make it such that the MLIR-based CI enforces a testing code coverage of more than 99% (or whatever makes sense) and fails otherwise. The coverage reported in the CI can then only be the actual code coverage without tests.

[math-fehr]

Hmm, I'm not sure if having a CI for x% is good or not.
Because essentially, this means we won't look at it until the threshold get passed, and it would be at that moment that we see that some code wasn't executed.
And there is also the fact that operation definitions are not marked as covered sometimes it seems, so we might pass the threshold without making any mistakes.
If we only show the values in the logs, I'm not sure this is good as well, since I doubt we are gonna look at it ;)

[webmiche]

Yes, I agree with both points...

It would be cool to be have two measurements in the Codecov comment, one for tests and one for actual code...

[georgebisbas]

I am happy either way. I would vote to keep covering the tests until the project scales...e.g. v2.0...