Randomized testing

[dcoeurjo]

PR Description

Add an option to cmake to only run and test a random selection of unit-tests (e.g. only 30% of them). Each time cmake is run, a new random selection is performed.

This may speed up CI bots since all PR have many commits involved (and the probability of missing an important test decreases with the number of commits;))

TODO:

• add a whitelist of tests that must be run/compile (eg, the one impacted by the last commits ?)(I have no idea about how to do that)
• Do the same for examples
• force a full test cmake .. -DTHRESHOLD_RANDOM_TESTING=100 when merging a PR

Checklist

• Unit-test of your feature with Catch.
• Doxygen documentation of the code completed (classes, methods, types, members...)
• Documentation module page added or updated.
• New entry in the ChangeLog.md added.
• No warning raised in Debug cmake mode (otherwise, Travis C.I. will fail).
• All continuous integration tests pass (Travis & appveyor)

[dcoeurjo]

I will do the same for the examples to speed up the CI bots.
Maybe DGtalTools as well ?
@kerautret

[dcoeurjo]

Ideas for the whitelist (to force some tests):

• Pass the test in cmake CLI https://stackoverflow.com/questions/33242956/cmake-passing-lists-on-command-line/33248574
• add a check in the "dgtal_add_test"
• In Travis, check the "test*" in the last git commits and add them to the whitelist

[kerautret]

nice idea to reduce time, and the random select could be a good compromise instead selecting all tests impacted by a commit.

[kerautret]

I will do the same for the examples to speed up the CI bots.
Maybe DGtalTools as well ?
@kerautret
For the test itself, we don't have any tests in DGtalTools, however we can see a way to reduce the build with random tools selection ?

[dcoeurjo]

Yes, I meant that in dgtal, Travis also builds the entire dgtaltools for a given dgtal commit (to make sure that the commit does not break up dgtaltools).
This could also be randomized for the CI

[kerautret]

yes agree to also doing it in the tools!

[dcoeurjo]

Great 👍
I'll do it for DGTAL examples as well (I have an idea to whitelist some tests from the git history)

[kerautret]

nice the example will looks simple to transpose in the tools, and me know to test or help

[dcoeurjo]

Randomized testing enabled in appveyor as well (not the whitelist that would require a shell command)

[kerautret]

looks great! I try to apply the same for DGtalTools, perhaps I will have some other comment else I will approve ;)

[kerautret]

Did it impact the auto main doxygen generation for including examples like these page: https://dgtal-team.github.io/doc-nightly/examples.html ?

[kerautret]

@dcoeurjo looks fine for me, perfectly works on my computer. I also do same for DGtalTools here: DGtal-team/DGtalTools#416 and do a related PR to your branch: dcoeurjo#30

[dcoeurjo]

Thx @kerautret