Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testsuite: Treat not tested as non failure and reduce verbosity #1665

Merged
merged 13 commits into from
Oct 22, 2024

Conversation

rdmark
Copy link
Member

@rdmark rdmark commented Oct 21, 2024

Overhaul of testsuite to reduce default verbosity for more readable test logs, and treat NOT TESTED and SKIPPED as non-failure for more consistent results in a CI environment.

Restore a handful of tests that were made functional by recent code improvements.

Standardize on -v for medium verbosity and -V for high level of verbosity logging.

Refashion the -x (Exclude) option for flagging tests that are failing with Netatalk 4.0 for unknown reasons. Investigation required.

Colorize also the SKIPPED test results.

All of this results in a cleaner test log and test report generated by spectest.sh that is much more readable. Sample:

===================
FPAddAPPL page 94
-------------------
FPAddAPPL:test214: test appl - NOT TESTED
===================
FPAddComment page 96
-------------------
FPAddComment:test55: add comment - NOT TESTED
===================
FPByteRangeLock page 101
-------------------
FPByteRangeLock:test60: illegal fork - FAILED
FPByteRangeLock:test80: Resource Fork FPByteLock Read write - NOT TESTED
FPByteRangeLock:test329: FPByteLock 2users DATA FORK - NOT TESTED
FPByteRangeLock:test330: pre OSX trash folder - NOT TESTED
FPByteRangeLock:test410: FPByteLock 2users DATA FORK - NOT TESTED
FPByteRangeLock:test366: Locks released on exit - SKIPPED (need to run individually with -f)
===================
FPByteRangeLockExt page 105
-------------------
FPByteRangeLockExt:test195: illegal fork - FAILED
===================
FPCatSearch page 110
-------------------
FPCatSearch:test225: Catalog search - NOT TESTED

@rdmark rdmark linked an issue Oct 21, 2024 that may be closed by this pull request
@rdmark rdmark changed the title testsuite: Treat not tested as non failure testsuite: Treat not tested as non failure and reduce verbosity Oct 21, 2024
@rdmark rdmark force-pushed the 1663-testsuite-treat-not-tested-as-non-failure branch from 579de2c to a757160 Compare October 21, 2024 22:54
@rdmark rdmark marked this pull request as ready for review October 21, 2024 22:56
@rdmark rdmark force-pushed the 1663-testsuite-treat-not-tested-as-non-failure branch from 835b3ad to 89be90c Compare October 22, 2024 13:04
@rdmark rdmark force-pushed the 1663-testsuite-treat-not-tested-as-non-failure branch 6 times, most recently from 15827b5 to 254a667 Compare October 22, 2024 17:37
@rdmark rdmark force-pushed the 1663-testsuite-treat-not-tested-as-non-failure branch from 254a667 to 9b04a06 Compare October 22, 2024 17:55
Copy link

sonarcloud bot commented Oct 22, 2024

@rdmark rdmark merged commit afa781f into main Oct 22, 2024
15 checks passed
@rdmark rdmark deleted the 1663-testsuite-treat-not-tested-as-non-failure branch October 22, 2024 18:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

testsuite: Treat NOT TESTED as non-failure
1 participant