Add retry and timeout in unit test

[ianco]

Signed-off-by: Ian Costanzo [email protected]

Increases likelihood of test passing from roughly 50% to 90%

[sovbot]

Can one of the admins verify this patch?

[WadeBarnes]

[ci] please test

[WadeBarnes]

(ci) please test

[WadeBarnes]

(ci) test this please

[Toktar]

Hello Ian,
thanks for your emphasize an issue with failed tests. I really apologize if the logic of these tests is a bit implicit. But these are simulation tests for checking the consensus. A random seed is selected with each new launch. That is, if you just restart a test, then you ignore a possible critical issue in the consensus logic (or view change in this case). Could you please write down the seed on which the test falls in the list of exceptions if you do not have time to figure out what is wrong?
I suggest not to put into the master branch rule to ignore tests that can find real bugs in the code.
As far as I understand it's a start of creating simulation tests:
#1242
#1246
It's a random article about seeds in simulation or randomization testing: https://sciprincess.wordpress.com/2019/03/14/how-to-select-a-seed-for-simulation-or-randomization/
Tomorrow I'll ask an author of these tests to get a better link to describing this approach.
UPD: The link to article from our expert: https://alexwlchan.net/2016/06/hypothesis-intro/

[ianco]

Hello Ian,
thanks for your emphasize an issue with failed tests. I really apologize if the logic of these tests is a bit implicit. But these are simulation tests for checking the consensus. A random seed is selected with each new launch. That is, if you just restart a test, then you ignore a possible critical issue in the consensus logic (or view change in this case). Could you please write down the seed on which the test falls in the list of exceptions if you do not have time to figure out what is wrong?
I suggest not to put into the master branch rule to ignore tests that can find real bugs in the code.
As far as I understand it's a start of creating simulation tests:
#1242
#1246
It's a random article about seeds in simulation or randomization testing: https://sciprincess.wordpress.com/2019/03/14/how-to-select-a-seed-for-simulation-or-randomization/
Tomorrow I'll ask an author of these tests to get a better link to describing this approach.

@Toktar The unit test creates a simulated pool using a random number of nodes, and then creates a random number of votes, so there is not a single "seed" per se that causes the test to fail. We suspect it is a timing issue hence the retries.

There are too many layers of logic for me to try to dig into it unfortunately :-( If you can get someone to take a look, they just need to retry the test several times and they should get some failures - for me & Wade this test fails about half the time (or about 10% of the time with the retries).

[Toktar]

@ianco In our case a seed is a random parameter and the number that you can see near the name of a failed test in logs. Looks like
...
test_new_view_combinations(440868)
...
The fails reproduce locally for me. And it's really a consensus problem for a ViewChange protocol but just with a medium risk.
I created an issue with description of the failed case #1506
And I'll send a PR with skipping tests for problem seeds. Could you please extend this exception list if find a new one?

[ianco]

@Toktar thanks for the notes, I'll re-test and try to track the failing seeds.