Unit tests results should not be displayed in 'automake output' mode #3232
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
what
Updated
unit_tests
program to not show individual unit tests results in automake output.why
Showing individual results breaks the collection of results when a test fails bypassing its detection, which makes
make check
show incorrect results.The issue is that the individual test results include escape characters to colorize the output (to show in red that a test failed for example) and this makes the generated
.log
&.trs
file binary, which breaksgrep
parsing intest/custom-test-driver
and the collection of test results fromautotest
.NOTE: The issue is not limited to the escape characters to colorize output as there are tests that include binary data in the input & output of the tests, which when display trigger the same issue.
For example, in the context of #3231, if you introduce a bug in the
ParityOdd7bit
transformation (by switching theinplace
template parameter fromtrue
tofalse
, which makes it behave like theParityEven7bit
in the new shared implementation), three of the tests should fail. However, if you run the tests withmake check
, they complete with zero errors! If you look at the output of the execution, though, you can see that there are grep errors:misc
Piggyback on this PR to complete update from PR #3229. 🙏