test_loss.test_hinge_loss has fixed seed that can mask flakiness

[szha]

The unit test in title have been using fixed seed to mask flakiness. Suggested action:

1. Evaluate whether the test is flaky without fixed seed. If not, remove seed. Else move to 2
2. If test is flaky, determine whether it's an actual uncaught edge case. If so, fix the operator. Else move to 3
3. If numerical instability is inevitable, adjust tolerance level appropriately.

[apeforest]

Thanks for filing this issue. We will investigate this Flaky test

[zachgk]

I ran the test numerous times. It only failed at one instance with the original tolerance, however I did not record the seed that caused the failure. After ~40k additional iterations, I have been unable to reproduce another failure with the original tolerance. However, the increased tolerance would have accepted that failure, so I updated it.

I then ran tests/python/unittest/test_loss.py:test_hinge_loss and tests/python/gpu/test_operator_gpu.py:test_hinge_loss 10k times each without producing the error at an increased tolerance

I raised a PR #12020 to increase the tolerance and remove the fixed seed

CPU Run with increased tolerance:

ubuntu@ip-172-31-27-232:~/incubator-mxnet$ MXNET_TEST_COUNT=10000 nosetests --logging-level=DEBUG --verbose -s tests/python/unittest/test_loss.py:test_hinge_loss

Produces:

[DEBUG] 9999 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=2140149943 to reproduce.
0.0
[DEBUG] 10000 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=552569265 to reproduce.
0.0
ok

----------------------------------------------------------------------
Ran 1 test in 7793.215s

OK

GPU Run with increased tolerance:

ubuntu@ip-172-31-27-232:~/incubator-mxnet$ MXNET_TEST_COUNT=10000 nosetests --logging-level=DEBUG --verbose -s tests/python/gpu/test_operator_gpu.py:test_hinge_loss

Produces:

[DEBUG] 9999 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=175424733 to reproduce.
0.0
[DEBUG] 10000 of 10000: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=745050216 to reproduce.
0.0
ok

----------------------------------------------------------------------
Ran 1 test in 8036.187s

OK

[haojin2]

@sandeep-krishnamurthy Related PR is merged, should be good to close.