Missing symlinks for libblas.so and liblapack.so

[SylvainCorlay]

The current version of the openblas package does not include the .so symlinks. Only libblas.so.3 and liblapack.so.3 are available.

[SylvainCorlay]

Ah it seems that the symlink were removed by @isuruf in conda-forge/openblas-feedstock@0d1073b.

We are relying on this in xtensor-blas + cling. Including headers of xtensor-blas causes the shared object to be loaded, but without the number suffix.

[isuruf]

xtensor-blas should be fixed so that it looks for the symlinks. Otherwise, xtensor-blas will be incompatible with mkl, blis and other blas implementations which only add libblas.so.3

[jschueller]

I was needing this also for omcompiler : at runtime it needs to link executables with "-lapack -lblas", I was able to work around it with --with-lapack="${PREFIX}/lib/liblapack.so.3 ${PREFIX}/lib/libblas.so.3"

[SylvainCorlay]

xtensor-blas should be fixed so that it looks for the symlinks.

This is a runtime thing in cling, and to my knowledge, there is not means to pass the version suffix to the pragma from some sort of runtime detection.

[isuruf]

You could detect the soname of the library at build time and hardcode that into the header

[isuruf]

This is going to be messy in cmake though. So, I'm okay with adding the symlink to blas-feedstock

[isuruf]

I'm not so sure about this anymore. I've seen a couple of projects (scipy and another) accidentally linking to libblas.so in conda-build because of missing libblas in meta.yaml and end up linking to libopenblas.so because of the soname.

How about a blas-devel package with the symlinks?

[jakirkham]

Are there any cases where the symlinks would be needed in the installation environment?

[isuruf]

@jakirkham, only when a package would dlopen the library at runtime.

[isuruf]

@SylvainCorlay, can you add a dependency on blas for xtensor-blas?

[isuruf]

ping @SylvainCorlay

[isuruf]

@SylvainCorlay, since you didn't comment, I'll assume that you are okay with moving the symlink to blas package.

[minrk]

Just ran into this on conda-forge/petsc-feedstock#66

Are there any cases where the symlinks would be needed in the installation environment?

Any package that does runtime-compilation with -lblas etc. needs this as well (fenics, petsc, cling-derived packages, inline Cython, etc.). I don't really understand any benefit to the libblas package not having a libblas.so. What was the reasoning there? I can't find any discussion about this solving a problem and no other library package omits its canonical linking name, to my knowledge. My reading of the PRs is that anything that does runtime compilation should now add the blas metapackage as a direct runtime dependency, so I'm going to start doing that.

The biggest direct problem I see is that libblas*_netlib includes libblas.so while libblas*_other don't. It seems like something's clearly wrong for different libblas packages to have different capabilities, and to me, this is that you can compile against libblas_netlib, but you can't compile against libblas_anythingelse.

So I'd be a big +1 to reverting #38 to make all libblas builds consistent with each other and consistent with other libfoo packages. I haven't seen any downsides to the symlinks being present.

[isuruf]

but you can't compile against libblas_anythingelse.

This is the reason why I want the symlinks removed. We don't want packages accidentally linking to one of these implementations. We only want the netlib packages for building.

Removing .so is similar to what debian does where the .so is only in the dev package.

[minrk]

Then I would argue that creation of the libblas-devel package should be a prerequisite for removing the symlinks from libblas. Right now, the only solution I see is for packages that do runtime compilation is to depend on blas as a runtime dep, so I'm going to go ahead and do that for now as it's the only way to create a working package.

[minrk]

To be consistent, libblas_netlib should also remove libblas.so, otherwise it doesn't really make sense.

[isuruf]

PRs welcome. This should also include the header files as well.
There are other options as well.

1. vendor .so files from netlib into the other packages
2. Do what debian does and create a stub library libblas.so with libblas.so.3 as the soname.

Right now, the only solution I see is for packages that do runtime compilation is to depend on blas as a runtime dep, so I'm going to go ahead and do that for now as it's the only way to create a working package.

Sure. Note that you'll have to be careful in downstream packages about not linking to the shared library from an implementation (instead of netlib ones)

[minrk]

PRs welcome

Will do when I get a chance. I'd start with reverting the symlinks back into libblas so things are consistent again while we get libblas-devel set up, if that's okay.

Note that you'll have to be careful in downstream packages about not linking to the shared library from an implementation (instead of netlib ones)

Yeah, thanks. Not a concern for the packages I'm currently looking at because these are runtime compilation, not build stages for redistribution. Failure to compile with libblas_openblas is indeed a bug in the package for these cases.

[isuruf]

Something like below works. I initially thought about making libblas-devel, etc, but that would add to the complexity. I like blas-devel because it conveys that that packages like petsc would depend on a development package at runtime.

diff --git a/recipe/meta.yaml b/recipe/meta.yaml
index dc13d0c..b279381 100644
--- a/recipe/meta.yaml
+++ b/recipe/meta.yaml
@@ -200,6 +200,13 @@ outputs:
         - test -f $PREFIX/lib/liblapacke.{{ version_major }}.dylib   # [osx]
         - if not exist %LIBRARY_BIN%/liblapacke.dll exit 1           # [win]
 
+  - name: blas-devel
+    requirements:
+      run:
+        - openblas   # [blas_impl == "openblas"]
+        - mkl-devel  # [blas_impl == "mkl"]
+        - blas {{ version }} *{{blas_impl}}
+
 about:
   home: https://github.com/conda-forge/blas-feedstock
   license: BSD-3-Clause

[minrk]

Fixed by #39, thanks @isuruf!

[isuruf]

@minrk, @SylvainCorlay, have you added a requirement of blas-devel to the packages yet? It's been more than a year and I think it's time to revert #39

[isuruf]

Ping again

[minrk]

I had to pull back from a lot of conda-forge stuff for a while, so I just forgot about it. That's on me, though and conda-forge/petsc-feedstock#106 should do it, since all the packages I deal with ultimately depend on petsc.

[minrk]

Hm, there appear to be some hiccups.

blas-devel should be in the host dependencies of every recipe that links libblas, right? Since there shouldn't be libblas.so in any libblas package. Even numpy hasn't been updated to use blas-devel in the host dependencies.

I also see something weird in priority resolution:

conda create -n petsc petsc=3.14.5

requests libblas-3.9.0-8_openblas

However, adding blas-devel switches the default to mkl:

conda create -n petsc petsc=3.14.5 blas-devel

gets libblas-3.9.0-8_mkl

Similarly, if I try to get blas-devel with openblas:

conda install openblas blas-devel

I get libblas_mkl and openblas.

Additionally, conda install blas-devel=*=*openblas to explicitly request blas-devel with openblas fails in the first solve:

$ conda create -n blasdev blas-devel=*=*_openblas
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done

[minrk]

It looks like blas-devel package is still incomplete. Tasks left to do before blas-devel can be adopted:

• add missing netlib variant of blas-devel (add build number to blas-devel build string lapack-feedstock#50)
• remove unversioned lib*.so from lib* netlib variants (see build netlib variants of blas, blas-devel in this repo #66 for necessarily coordinating PRs to both this and liblapack)

Otherwise, blas-devel cannot be used because it conflicts with the libblas=*=netlib used in all conda-forge builds:

$ conda create -n blas libblas=*=*netlib blas-devel
Collecting package metadata (current_repodata.json): done
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: |
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package libblas conflicts for:
blas-devel -> libblas[version='3.8.0|3.9.0',build='0_h43114d0_netlib|7_openblas|7_blis|8_mkl|8_openblas|8_blis|7_mkl|7_hd44dcd8_netlib']
libblas[build=*netlib]
blas-devel -> liblapack==3.9.0[build=*netlib] -> libblas[version='3.8.0.*|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.8.0|3.9.0|3.9.0|3.9.0|3.9.0.*|3.9.0|3.9.0|3.9.0|3.9.0|3.9.0',build='7_h4bb4525_netlib|15_blis|16_openblas|17_openblas|18_mkl|21_mkl|0_h508aa58_netlib|2_blis|3_blis|4_blis|5_openblas|6_blis|6_mkl|6_openblas|5_mkl|5_blis|4_mkl|4_openblas|3_openblas|2_openblas|20_mkl|19_mkl|16_mkl|16_blis|15_mkl|15_openblas|14_mkl|14_openblas|14_blis']
Note that strict channel priority may have removed packages required for satisfiability.

[minrk]

Hm, looks like it's there but out-of-date due to the build string: conda-forge/lapack-feedstock#50

[dalcinl]

I'm wondering whether conda-forge should slowly move to use FlexiBlas, just as the Fedora community recently did [link].

[minrk]

conda-forge effectively already has that behavior - we build and link against libblas (etc.) provided by netlib during build, then at runtime you can select any blas/lapack provider - openblas, netlib, mkl, etc. We haven't had a blas-provider build matrix for some time, thanks to the wonderful work of @isuruf.

This issue is just about exactly what package has libblas.so vs libblas.so.3.

[dalcinl]

@minrk What do you mean by runtime? conda install time? Or can users really switch the implementation via config files of environ vars?

[SylvainCorlay]

We had been working around that issue in the xtensor-blas binder by simply requiring blas * *openblas" for now. I think making blas-devel a dependency of xtensor-blas is probably a better solution.

Thank you for working this out.

[minrk]

@dalcinl yes, conda install time. The blas providers are mutually exclusive, so cannot be in the same env.