Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use consistent tests #762

Merged
merged 1 commit into from
Dec 4, 2021
Merged

Use consistent tests #762

merged 1 commit into from
Dec 4, 2021

Conversation

akaszynski
Copy link
Collaborator

@akaszynski akaszynski commented Dec 4, 2021

Ping @germa89:

Looks like #758 didn't quite resolve our issues. We're still seeing intermittent failures in test_load_array. You can validate this locally with:

pytest tests/test_mapdl.py -k test_load_array --count 100 -x

This fails because when our input array is shaped (10, 1), MAPDL will output a flat array (10,). Since numpy outputs flat arrays when the second dimension is 1, (e.g. numpy.arange), this PR changes the test to account for this edge case.

Lessons learned

If we want to test for a variety of edge cases, use parameterized inputs:

@pytest.mark.parametrize("dimx", [1, 3, 10])
@pytest.mark.parametrize("dimy", [1, 3, 10])
def test_load_array(mapdl, dimx, dimy):
    my_conv = np.random.rand(dimx, dimy)
    mapdl.load_array("my_conv", my_conv)

    # flatten as MAPDL returns flat arrays when second dimension is 1.
    if dimy == 1:
        my_conv = my_conv.ravel()
    assert np.allclose(mapdl.parameters["my_conv"], my_conv, rtol=0.0001)

This way, we test a variety of edge cases without resorting to random integers. Tracking down flaky tests is a nightmare. Let's avoid it by making tests consistent. If we want complete testing, let's focus on parameterized tests, not randomized tests. If we really want to do randomized tests correctly, checkout the hypothesis library. Disclaimer: it's going to make our tests longer and harder to maintain.

@akaszynski akaszynski self-assigned this Dec 4, 2021
@akaszynski akaszynski merged commit 0c46e1d into main Dec 4, 2021
@akaszynski akaszynski deleted the fix/load_array_stability branch December 4, 2021 07:39
@germa89
Copy link
Collaborator

germa89 commented Dec 4, 2021

Hi @akaszynski
Thank you very much for working on this. This was a difficult one to debug (I believe). Happy to learn which steps you followed to discover the reason.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants