python3.pkgs.scipy ignores when tests failed

[yu-re-ka]

Describe the bug

Test summary info reports that some tests failed, but the package still built

Steps To Reproduce

Steps to reproduce the behavior:
nix build github:yu-re-ka/nixpkgs/b1ec44240c44960791760c86ad9703fbf43d3517#pkgsMusl.python3.pkgs.scipy --impure

Expected behavior

The build should fail if tests fail

Screenshots

[...]
python3.10-scipy> =========================== short test summary info ============================                                                                    
python3.10-scipy> FAILED lib/python3.10/site-packages/scipy/spatial/transform/tests/test_rotation.py::test_as_euler_symmetric_axes - AssertionError:                  
python3.10-scipy> = 1 failed, 42000 passed, 2825 skipped, 135 xfailed, 11 xpassed, 4657 warnings in 182.86s (0:03:02) =                                               
python3.10-scipy> /build/source                                                                                                                                       
python3.10-scipy> installCheckPhase completed in 3 minutes 7 seconds                                                                                                  
python3.10-scipy> pythonCatchConflictsPhase                                                                                                                           
python3.10-scipy> pythonRemoveBinBytecodePhase
python3.10-scipy> pythonImportsCheckPhase
python3.10-scipy> Executing pythonImportsCheckPhase
python3.10-scipy> pytestcachePhase
python3.10-scipy> pytestRemoveBytecodePhase
python3.10-scipy> pytestXdistHook

$ 

Additional context

N/A

Notify maintainers

@doronbehar @FRidh

Metadata

N/A

[doronbehar]

Thanks for the report. I have a vague memory that we handled that with sys.exit in #239969 but I guess not. A fix is upcoming (tested as of currently).

Do you want to also fix that failing test before staging-next is merged to master? Should be rather simple to only disable it.

[doronbehar]

Should be fixed in #245935 . Closing. Fixing the failing test can be done in a separate commit, if at all.

[ghost]

Should be fixed in #245935 .

It would be really nice if 01e53f7 had a PR (not just the staging-next mega-PR) as a place to discuss it. I'm getting test failures ever since that commit:

       > FAILED lib/python3.10/site-packages/scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints2-<lambda>-kwds2-SLSQP] - AssertionError:
       > FAILED lib/python3.10/site-packages/scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints2-Bounds-kwds0-SLSQP] - AssertionError:
       > FAILED lib/python3.10/site-packages/scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints2-Bounds-kwds1-SLSQP] - AssertionError:
       > FAILED lib/python3.10/site-packages/scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints2-Bounds-kwds2-SLSQP] - AssertionError:
       > FAILED lib/python3.10/site-packages/scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints4-<lambda>-kwds0-SLSQP] - AssertionError:
       > FAILED lib/python3.10/site-packages/scipy/optimize/tests/test_optimize.py::test_equal_bounds[None-constraints4-Bounds-kwds0-SLSQP] - AssertionError:
       > FAILED lib/python3.10/site-packages/scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints2-<lambda>-kwds0-SLSQP] - AssertionError:
       > FAILED lib/python3.10/site-packages/scipy/optimize/tests/test_optimize.py::test_equal_bounds[callback-constraints2-<lambda>-kwds1-SLSQP] - AssertionError:
       > FAILED lib/python3.10/site-packages/scipy/sparse/linalg/_isolve/tests/test_iterative.py::test_x0_equals_Mb[bicgstab-nonsymposdef] - assert -11 == 0
       > = 17 failed, 41989 passed, 2820 skipped, 136 xfailed, 10 xpassed, 4658 warnings in 255.44s (0:04:15) =
       For full logs, run 'nix log /nix/store/535rl8abn08xvvikpirxxgsjd4kxa562-python3.10-scipy-1.11.1.drv'.

Fixing the failing test can be done in a separate commit, if at all.

?!?!?! This is baffling. If the test fails, the package fails to build, and none of its dependencies can be built.

IMHO in a situation like this where tests were ignored, the PR that makes them no longer be ignored needs to either

a. Disable the failing tests or
b. Disable all the tests (set doCheck=false)

... otherwise the PR is actually increasing the level of breakage...

[doronbehar]

It would be really nice if 01e53f7 had a PR (not just the staging-next mega-PR) as a place to discuss it. I'm getting test failures ever since that commit

Interesting, I wonder how come.. On Hydra and locally I don't get tests failures. That's why I didn't create a PR because I thought (and still think) everything is OK. Let's discuss the details of your failure in a new issue.