-
-
Notifications
You must be signed in to change notification settings - Fork 47
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
TEST: 1.19.x + blas variants #196
Conversation
Hi! This is the friendly automated conda-forge-linting service. I just wanted to let you know that I linted all conda-recipes in your PR ( |
This looks quite a bit better than on the scipy-side. Still it did catch some errors that might not show up anywhere else in CI. I mentioned in conda/conda-build#3947 that IMO, foundational packages like numpy/scipy should be tested across all available blas variants. This would - effectively - provide some further CI on the various blas builds, but also make sure that no broken variants are shipped. Note: The cells of the matrix have the same behaviour for all python versions (cpython 3.6-3.8, pypy 3.6) unless otherwise noted - hence the simplified matrix
There were 4 additional failures not shown in the matrix above:
linux + x86 + netlib/openblas + pypy: segfault, test suite aborted
linux + ppc64le + openblas + cpython 3.6: 1 test failure (TestCholesky.test_basic_property)
linux + ppc64le + openblas + cpython 3.7: 4 test failures
linux + ppc64le + openblas + cpython 3.8: 3 test failures
|
Maybe worth pinging @conda-forge/help-pypy on this. The segfaults on linux+pypy for numpy 1.19 affect both this PR and #195. |
What version of OpenBLAS are you testing with? 0.3.10 was released recently and fixes an interaction that would cause a segfault on PyPy on azure machines. |
@mattip There's more stuff that I don't understand, actually. Bumping blis from 0.6.1 to 0.7.0 has introduced a segfault on cpython 3.6, 12 test failures on 3.7, but runs through fine on 3.8...? 🤔 |
After bumping blis to 0.7.0 and openblas to 0.3.10 (see conda-forge/blas-feedstock#45), the picture changed a bit, in some ways unexpectedly. In short:
linux + x86 + mkl + pypy: segfault, test suite aborted
linux + ppc64le + openblas + cpython: 3 test failures
linux + ppc64le + openblas + pypy: 4 test failures
windows + blis + cpython: 11-12 failed testsThis one seems to be a bit fickle as well - in the previous run, it appeared for py36 & py37 but not py38, not it appeared for py37 & py38 but not py36.
|
Thanks for the summary. Numpy regularly builds on travisCI machines with PyPy, both in the numpy/numpy CI and on the MacPython/num-wheels CI. I wonder what is different on the Azure builds? |
In the build log for pypy I am seeing |
You're looking at the right logs, but I think you're mixing up the reference-lapack version numbers with openblas (note: 3.9.0 isn't in conda-forge yet because MKL isn't compatible yet). As I noted in the last comment, the segfault on pypy seems non-deterministic. It appeared for openblas in 2 out of 3 runs, for netlib 1/3 times and for mkl 1/3 times. One of the runs has been deleted by azure, but here's one where the segfault occurred with openblas. I haven't investigated which test follows |
Ahh, right sorry. But in the In NumPy we use add a check like this (from
|
You're right, sorry. The run I linked was before conda-forge/blas-feedstock#45 was merged. The first run after that had the segfault, but got deleted by azure since I restarted before everything was through. Let's try that again. |
this is a tough PR. The starts must align just perfectly for everything to pass in the same run. |
It seems to me the best answer would be to figure out what causes the segfault and fix it 😉 The blis+win builds are also failing consistently, as are the linux+ppc builds. So we'll probably need more than lucky stars to pull this into the green 😅 |
I wonder if there is a path to release a 1.19 NumPy for at least the most common platforms. |
I see that OpenBLAS is being built with
In [this comment](https://github.com/OpenMathLib/OpenBLAS/issues/2388#issuecomment-582267407), and elsewhere, they recommends using `TARGET` as a minimal platform when `DYNAMIC_ARCH` is used which will be the fallback when no better dynamic loop can be found. The default is the build platform, which may be problematic if the build is done on a more advanced machine than the test.
|
No more 1.19 patch releases expected, time to close this PR |
…nda-forge-pinning 2022.04.12.21.56.28
This reverts commit 6dc144f.
b1d98ba
to
f5d4c8b
Compare
Revival (new PyPy builds and BLAS updates): all green except PPCDue to rebuilding 1.19 for pypy3.8/3.9, much less several relevant BLAS (& infrastructure) changes, I thought I'd revive this PR for one last update. From 9 failures out of 92 runs, we're now at 12 failures (PPC-only) out of 108 runs. Notable
Details
Build logs: |
Based on #195; doing a similar investigation as in conda-forge/scipy-feedstock#130.