-
-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
error loading shared object with conda-build 2 #21
Comments
Disabling symbolic linking, and forcing copy (within the docker linux-anvil build image) fixed this for conda-build 2... |
Logs: with sym-links:
with copy:
|
Within my docker container (condaforge/linux-anvil) I am able to reproduce the linking problem with conda-build 2 iff the conda-build root is I'm still trying to verify this, what is confusing is that I'm not entirely convinced it is a long prefix issue:
I suppose it is conceivable that one of the dependencies has a build prefix length of less than 80 even and is therefore unable to be installed in a 80 character long prefix... |
Does |
Also, |
Do you mean something like this |
FYI @mariusvniekerk offered to lend a hand trying to rebuild |
Any updates here? |
So i'm having build issues trying to get libarchive working with conda-build 2. Seems to not be able to find the lzma stuff despite xz building and testing fine. |
Do you have a log and other info you can share? |
So I'm getting failures building deps for cmake. perl with conda-build 2.
This seems to be related to something that conda build is doing with $PATH manipulation? m4 with conda-build 2
This seems to be related to http://trac.clfs.org/ticket/1078 |
I looked at a couple of libraries on Setting
seems to have fixed the cmake problem in my feedstock, https://circleci.com/gh/GorgonCryoEM/gorgon-feedstock/146. |
Here is a summary of some relevant commits from my feedstock.
|
Why is this thread gone silent? Are there other blockers or issues more pressing than this that delay it? Upgrading to conda-build 2 feels important, if I am not missing something. |
Something that may be relevant is this change: https://github.com/conda/conda-build/pull/1336/files I'm not sure if that change makes binaries built with v1 incompatible with those built by v2. Do we have a complete v2-built foundation here? |
I can't see |
Sorry, I can't say I really understand what's going on in that test. @mingwandroid can you please chime in here? |
Are we interested in why v1 works , but v2 breaks? Or, do we want to fix whatever is incompatible or broken that causes the problems with v2? |
Well, given how pressing this issue, the inactivity of this thread drives me nuts. Maybe, overstepping, but since I haven't got any feedback on my previous |
Is there a PR that fixes things? |
@shadowwalkersb : From your comment #21 (comment)
Without a doubt, these libraries need to be rebuilt. Soft-linking is inherently incompatible with $ORIGIN-relative RPATHs, and IMHO for this reason soft-linking should be removed altogether from conda: |
Not on conda-forge. I didn't want to put much effort before getting some feedback here. But, it worked on my CI, see #21 (comment) and #21 (comment). |
@shadowwalkersb, if you open a PR, I think we'll have something to go from. |
The solution I would propose would be adding this to the recipes.
Which recipe would you suggest to open a PR? Or, maybe patch |
Or, PR on cmake-feedstock and then try the fix on previously failing feedstocks? |
This is a workaround. Please read my comment: #21 (comment) |
Where exactly do we do that? I mean, switiching to copying. |
I don't know enough about the http://conda.pydata.org/docs/config.html#advanced-configuration |
So was debating disabling We don't really have a way to disable that on a per feedstock basis ATM (aside from hacking the CI files). Something like PR ( conda-forge/conda-smithy#285 ) would give us that ability, but there are still some points of disagreement that are yet to be resolved. Something like |
So, do you suggest going the RPATH fixing route for now, @jakirkham ? |
What if we set |
|
@mingwandroid , if you don't mind, what are your concerns here? |
I don't think there's any can of worms disabling symlinks for conda installs. My concerns are given in my previous comment that links to the two issues that describe in some detail, why symlinks are not compatible with conda rpath handling on Linux. |
I'm sure all those discussions that have been going on for a couple months need to be resolved, but I don't think we should wait for them for a few more conda and conda-build releases to solve the problem here. ATM we can't upgrade to conda-build v2. I'm sure not upgrading causes a lot of problems. And the major blocker seems to be cmake issues which are discussed here. As hacky they could be, can't we just go ahead with something, no matter how hacky or how ugly a workaround, to fix cmake and start rebuilds? I'm sorry for all the noise, but, I really can't see why this is not going forward. |
Have you had time to look into this issue more, @mingwandroid ? |
Is this still a problem? I tried to replicate the original issue, building |
Are you sure it was building with Just opened this PR ( conda-forge/blosc-feedstock#7 ), which demonstrates this exact problem. Feel free to build off of it if you wish to experiment. |
Thanks for the example @jakirkham. I wonder if the different was the conda version used, I was using 4.2.12, the blosc example uses 4.1.12. I'll look into this further tonight and try to debug what the issue is. |
I created a PR (conda-forge/blosc-feedstock#8) to test using the latest versions of conda-build: Perhaps @msarahan would know what changed between conda |
Sorry, I don't know what about conda's behavior in this area changed between 4.1.x and 4.2.x. My guess is that the warnings aren't showing up because conda 4.2.x is inappropriately hijacking the logger output. That is a known issue. As for why it works anyway, I'm at a loss. When conda-build is unable to install some package into a long-prefix build environment (due to the package having short binary prefixes), it falls back to creating an 80-character build prefix. You can see that this is happening (despite Conda silencing conda-build's logger messages) by the duplicated install section at
If anything has changed between 4.1.x and 4.2.x, it is more intelligent symlink handling in conda. Conda-build 2 made changes to rpath stuff that probably broke incorrect assumptions that conda made about rpaths. The conda-build changes were a necessary fix. Essentially anything that had embedded binaries outside of the standard bin location was broken. We saw this in particular with repackaging Java binaries from Azul Systems. See conda/conda-build#1336 for more details on when this changed. |
Thanks for the information @msarahan, I think I have a better understanding on what is going on here, namely that an 80-character build prefix package is being created. I believe the root of the issue is that three of the build requirements for blosc are 80-character prefix packages in the For conda-forge to move to conda build Unrelated, it looks like the conda build configuration settings listed in conda documentation are not supported by conda |
That was supposed to be fixed ( conda/conda#3744 ). 😕 Are you sure you are using |
So if upgrading |
Still seeing the buggy behavior in |
Yup although it does not seem as if the patch fixed the actual issue. |
Thanks for helping get issue ( conda/conda#3744 ) actually fixed. |
seems fixed, closing |
I have been able to reproduce the problem identified in #14, and can confirm that simply changing from conda-build 2 to conda-build 1 fixes the problem.
Some pertinent output:
I will use this issue as a place to document findings...
The text was updated successfully, but these errors were encountered: