Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some tests are failing randomly from time to time #279

Open
olekscode opened this issue May 5, 2022 · 8 comments
Open

Some tests are failing randomly from time to time #279

olekscode opened this issue May 5, 2022 · 8 comments

Comments

@olekscode
Copy link
Member

olekscode commented May 5, 2022

This is probably because they don't set the random state.
For example:

  • PMQRTest >> testMPInverse
@hemalvarambhia
Copy link
Contributor

I've begun looking into this. Could you clarify what should happen when random state is set, please? As an alternative, I wondered whether a simpler test could be written. The current one is difficult for me to follow.

@hemalvarambhia
Copy link
Contributor

hemalvarambhia commented Jun 26, 2022

I've begun carefully refactoring the testMoorePenroseInverse test to enhance clarity (Four Elements of Simple Design).

To effectively test the Moore Penrose Inverse computation, is it necessary to set random state, @olekscode ? Due to my complete ignorance I am unsure of why.

Thank you.

@hemalvarambhia
Copy link
Contributor

Another test that's failing is

PMNumericalMethodsTestCase
[1496](https://github.com/PolyMathOrg/PolyMath/runs/7117052304?check_suite_focus=true#step:4:1497)
 ✗ #testOptimizeOneDimension (12ms)

@hemalvarambhia
Copy link
Contributor

Another test that's failing erratically:

PMGeneralFunctionFitTest
[1484](https://github.com/PolyMathOrg/PolyMath/runs/7119999338?check_suite_focus=true#step:4:1485)
 ✗ #testMaximumIterationsProbabilistic (1009ms)

@hemalvarambhia
Copy link
Contributor

testQRFactorisation in PMQRTest fails erratically

@hemalvarambhia
Copy link
Contributor

hemalvarambhia commented Jul 3, 2022

@olekscode I think I managed to corner the input matrix that is responsible for the erratically failing test:

testInputOfErraticFailure

| a |
a := PMSymmetricMatrix rows:
    #( #( 0.41929313699681925 0.05975350554089691
          0.2771676258543356 0.35628773381760703 )
       #( 0.05975350554089691 0.12794227252152854
          0.3257742693302102 0.28814463284245906 )
       #( 0.2771676258543356 0.3257742693302102 0.8468441832097453
          0.9101872061892353 )
       #( 0.35628773381760703 0.28814463284245906
          0.9101872061892353 0.5163744224777326 ) ).

self assert: (a mpInverse closeTo: a inverse)

This fails for the same reason as highlighted during the CI run. Thus we have a deterministic test we can use to corner the failure much more.

If you'd like to investigate with me, you can find this in investigate-erratic-test

Does the above matrix relate in some way to your hypothesis of not setting the random state.

My investigate points to an array (3 4 3 nil) - the nil appears to be the cause.

@jecisc
Copy link
Member

jecisc commented Nov 29, 2022

Another test that's failing is

PMNumericalMethodsTestCase
[1496](https://github.com/PolyMathOrg/PolyMath/runs/7117052304?check_suite_focus=true#step:4:1497)
 ✗ #testOptimizeOneDimension (12ms)

I confirm this random failure:

PMNumericalMethodsTestCase
✗ #testOptimizeOneDimension (12ms)
TestFailure: Assertion failed
PMNumericalMethodsTestCase(TestAsserter)>>assert:description:resumable:
PMNumericalMethodsTestCase(TestAsserter)>>assert:description:
PMNumericalMethodsTestCase(Object)>>assert:
PMNumericalMethodsTestCase>>testOptimizeOneDimension ...assert: (maximum - 5) abs < 1.0e-6
PMNumericalMethodsTestCase(TestCase)>>performTest

@jecisc
Copy link
Member

jecisc commented Mar 13, 2023

This is probably because they don't set the random state. For example:

* PMQRTest >> testMPInverse

I fixed this one and all related to random matrix and vectors: PolyMathOrg/vector-matrix@5fb38e9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants