Feature Request: Preventing formulae updates from breaking their dependencies

[ahundt]

Key problems

1. Package A is updated from 1.0 to 2.0, breaking package B which depends on A 1.0.
   • breakage is not always detected
     • Example: ceres depends on an exact version of eigen (a simple bottle refresh could fix it)
   • breakage can't always be fixed
     • B may not support A 2.0 until its next release a month from now
2. Package B,C depend on different versions of package A. Package D depends on both B, C and sometimes A directly. (other variations on this theme exist)
3. Options sometimes break invisibly
4. Cross tap dependencies may change

Solutions with most backing

These are reasonably easy, well liked ideas that will mitigate (but may not completely solve) the above

• allow a limited set of versions into homebrew supported taps with an aliasing mechanism for the latest version so dependent libraries can have a soft landing until they support new versions
• define criteria for accepting requests/proposals to make certain options recommended by default

proposed solutions

This is the full list of proposals (not definite upcoming features). Check marks indicate they are implemented

• checking all formulae for broken dylib linkage
• rearranged brew audit output format
• allowing versions for widely used directly into homebrew supported taps where appropriate
  • this seems particularly popular and supported by several commenters
• "maximal options" test
• dependencies on specific versions in formulas
• allowing version phase-ins / phase-outs
• side by side install of versions (where possible)
• keg only installs of versions
• alias the latest version i.e. boost is boost160
• allow multiple acceptable dependency versions to be specified perhaps with a preferred option
• easy way to remap dependencies like brew install mypackage --override-dep boost ahundt/boost155. could be a normal brew command or perhaps an easy way to change the formula files?
• move libraries and system services (not end-user applications or GUIs) that have dependencies in the Homebrew org to being versioned on the next major upgrade
• allow multiple versions to be specified in a single formula
• define criteria where proposals to make certain options recommended by default is encouraged

posed problems without solutions

these are from the ongoing conversation beyond the initial request posed above

• version dependency conflicts for multiple packages. Here is an example:

A: depends on boost (latest)
B: depends on boost155 (newest feasible version)
C: depends on A, B, sometimes various boost versions

• cross tap dependencies may change

Original Feature Request

I keep having libraries I depend on get broken because those dependencies' own dependencies are changed out from under them. In particular, this happens most frequently when flags other than the default are needed.

Two examples:
https://github.com/Homebrew/homebrew-science/issues/3454
https://github.com/Homebrew/homebrew-science/pull/3512

Perhaps there is a way to avoid these problems that also meets the crucial criteria of the homebrew team being willing to put it into practice? :-)

Thanks for your thoughts and considering my request!

[tdsmith]

In particular, this happens most frequently when flags other than the default are needed.

This is a difficult problem because we are unable to test all combinations of all formulas with all options. Formulas with fewer options are inherently less fragile.

We do test the default configurations where we have tests available. If the broken formulas lack tests, adding a test do... block that tests the default configuration will help the bot detect breakage and prevent us from pulling an incompatible update in the future.

Of course, we do not want to break packages. I think there is not a generic solution but maybe you have some ideas.

[ilovezfs]

As peculiar as it may sound, this is basically by design. If you need something more stable, I'd suggest maintaining your own tap with the working set of tools pegged to a particular version and backing up older bottles and build-time resources that you need. brew deps and brew uses --recursive are the commands to understand the scope of what you'd need to capture in such a tap.

Note that you can also generate your own bottles using install --build-bottle + brew bottle. The test-bot command is helpful for this, but it roughly does this:

#!/bin/bash
set -e
brew fetch --retry --build-bottle "$1"
brew install --only-dependencies --verbose --build-bottle "$1"
brew install --verbose --build-bottle "$1"
set +e
brew audit "$1" --strict --online
brew style "$1"
set -e
brew bottle --verbose --rb "$1"
brew bottle --merge --write --no-commit "./$1"*rb
brew uninstall -f "$1"
brew install "./$1"*gz
mv "$1"*gz ~/Library/Caches/Homebrew

Cf. Homebrew/homebrew-bundle#162

(Another option is to use virtualization and/or file-system level snapshotting).

[apjanke]

Yeah, adding tests will help in some cases. And minimizing options in the first place will help.

TL;DR: Add test do blocks to all your formulae so test-bot can test them when upgrading their dependencies. Probably would have caught those two breaks before they happened.

Details:

The Homebrew/homebrew-science#3454 ceres-solver breakage looks like it was a dylib version linkage problem. It didn't get caught first time around because ceres-solver did not have a test do. Adding a test do, even a basic one like --version, will generally be enough to catch dylib linkage problems for direct dependencies, because when we test a version bump PR, that includes testing all the bottled dependents. (At least the ones with direct dependencies.) For this kind of breakage, making sure all formulae have tests, even if they're trivial tests, will help. brew audit now requires a test do for all formulae, so we're making progress there.

Homebrew/homebrew-science#3512 PCL doesn't have a test do in its formula either, so it's not being tested when its dependencies change. I can't tell from skimming that issue whether it was a dylib version problem or a different issue, but a test might have caught it.

I'm not sure what happens in test_bot when testing a PR for foo and there's a testable bar with depends_on "foo" => "with-some-option". For dependencies besides the tested formula, it will install the dependency with the option. But I'm not sure what happens when the dep is the formula under test. It probably should reinstall it with that option. It probably doesn't do it 100% correctly now: our dependency resolution isn't perfect, and it's something we're working to get more robust in the longer term.

The full solution of testing all combinations of options is prohibitively expensive. (Testing N options takes N or 2^N runs, depending on how thorough you are.)

We could do a smaller version where we just test the option combinations which are used in depends_on xxx => "with_foo" dependencies in the central taps. (And maybe we effectively already do, depending on how the test-bot dependency resolution works.) But even that could blow up the test bots: we're already pushing run time limits with larger formulae that have lots of dependencies, and that's where some of the dependencies with options happen. Could be helped by having test-bot examine the dependents and consolidate dependents that need options toward the end of the list, to avoid redundant builds. Calculating that could get hairy once indirect dependencies are considered, though.

We could also detect dylib versions by interrogating files with otool and storing that data for bottles some where. But that would introduce a whole new data set for brew to manage, and would be mostly redundant with test do blocks, which we want to encourage, and which can test more things. So I don't think we'd want to do that.

The best immediate approach is probably "make sure all your formulae have tests". I think that would have caught both of the issues you're referencing here.

Open to other suggestions. I may have blinders on for this issue since I'm so used to looking at test-bot throughput limitations.

(EDIT: Note that this ability is currently limited because we're not using uses --recursive in the test-bot, for performance reasons. Open issue for it: Homebrew/legacy-homebrew#50256)

[ahundt]

@tdsmith Maybe it is possible to compile once with the default and once with all options? I know mutually exclusive options won't work there but it would probably eliminate a large subset of these problems while only requiring 1 additional continuous integration run.

[ilovezfs]

A pull request implementing a brew install --maximal-configuration foo would be welcome for review.

[ilovezfs]

@ahundt FYI Homebrew/legacy-homebrew#49338 (comment) ;)

[tdsmith]

A pull request implementing a brew install --maximal-configuration foo would be welcome for review.

Would it? What's the goal here?

Reopened for now; I'm not sure the discussion is over.

[tdsmith]

while only requiring 1 additional continuous integration run.

I'm not sure doubling the CI load is acceptable either; we haven't found a way to move our CI infrastructure into the cloud and right now we aren't willing to invest in the capital costs or the maintenance burden of additional CI builders.

We could maybe do some heuristics about whether it looks like dylib names are likely to change, but that seems less effective than adding tests to formulas.

[apjanke]

We would probably have to alter the formula DSL to express option conflicts, too, or explicitly specify maximal_configuration option sets in the formulae themselves. IIRC, there's a lot of formulae that have options or optional dependencies which conflict with each other (like openssl vs libressl). I'm not sure we'd want to do that.

[ahundt]

maybe an "advanced build" command where the rb script can specify a second reasonable configuration to test with extra options set? that's something I'd know enough to submit pull requests for as an occasional, very minor contributor.

@ilovezfs thanks for the link, I'd be sad if there were no options because I'd be back to compiling from source all the time... I'm using stuff like cuda and image tools. For one data point I'd say about 90% of the time I type brew install commands I run use options. I'm probably an unusual case though.

[DomT4]

Maybe it is possible to compile once with the default and once with all options?

There's no way this'll fly on Homebrew's current CI.

It's an outlier of an example but doing Qt5 per option (which is the safest way to handle possible conflicting options) would take ~10 hours of CI time, per VM. You're talking 30 hours of CI time total, for a single formula. Doing it in a way where we do an initial run and then one with all options afterwards (potentially risking conflict) would still take upwards of ~4 hours per CI time, per VM.

Short of someone providing Homebrew with a large enough donation to enable us to buy, run and maintain a significantly larger number of hardware we're limited on what we can reasonably do. We tried to farm out some of the CI load to Travis but have ended up pulling back on that for various reasons. CI is a struggle for us, and it chews up a lot of hours behind-the-scenes keeping it going.

The answer here to an extent is that people need to be careful which options are added and only add the ones with community demand and value. Every option added to an official Homebrew formulae is more or less a promise that we'll keep trying to support it, as removing options is always treated as more or less a last resort.

brew install --maximal-configuration foo

For reasons stated by others above and myself before I still consider this a "fairly bad idea" ™️.

[sjackman]

Options in Homebrew are inherently fragile, because the options are not tested by the test-bot, and the interactions between options are definitely not tested. Using any option basically puts you into uncharted territory. Within the current constraints of not testing options with CI, the only way to avoid breakage is not to use options. If there's an option that you use frequently, you could open an issue to discuss making that option the default behaviour for the formula, and then it would be tested by CI.

[DomT4]

Increasingly we're asking the question if an option is popular/valuable enough to merit inclusion is it popular/valuable to merit being turned on by default. We haven't come up with a firm answer yet, but on a case-by-case basis it's worth considering.

[ahundt]

@sjackman an "advanced build" option could reduce the surface area that is untested while keeping compile times lower than the full combinatorial explosion of possibilities.

Or, perhaps more options should generally be included by default? I'd be happy to add some pull requests adding :recommended tags (or stronger) to the options I personally require that aren't on by default in homebrew-science and other formulae repositories for libraries like opencv, vtk, pcl, eigen, etc. Just have to look up how to do it right... wish I knew ruby better... :-)

[MikeMcQuaid]

Options in Homebrew are inherently fragile, because the options are not tested by the test-bot, and the interactions between options are definitely not tested. Using any option basically puts you into uncharted territory.

This is a good point and options should be considered "dangerous" by anyone relying on CI to test formulae. Really, I think on most formulae they add more user pain than they save...

CC @Homebrew/science here. This is a reason brew audit demands a test do block. Without one our reverse dependency testing is ineffective.

Or, perhaps more options should generally be included by default?

I'm fairly keen to keep formulae's default options on the stuff that's useful for 99% of users. This has made me remember that we should be making a note with our new analytics of what options are used for formulae so we can consider making commonly used ones the default.

Just have to look up how to do it right... wish I knew ruby better... :-)

You'll learn. I'd never used Ruby before I started working on Homebrew and now I do Ruby development full-time 😉

[sjackman]

CC @Homebrew/science here. This is a reason brew audit demands a test do block. Without one our reverse dependency testing is ineffective.

Yes tests are absolutely necessary. We insist on tests now, but there was a time when we didn't, and some formula haven't changed since that time. Currently 486/581 (84%) have tests.

You'll learn. I'd never used Ruby before I started working on Homebrew

Ditto. Homebrew was my first Ruby project as well. I quite like the language now.

[sjackman]

Without one our reverse dependency testing is ineffective.

I've wondered how this system works. If app depends on lib, and lib is rebuilt changing the dylib name (or soname), and the test-bot notices that app breaks, does it build a new bottle for app, and does brew pull --bottle change the bottles for both lib and app? I haven't seen breakage due to a changed dylib name in a long time, so I figured that there's some system in place, but I've never seen brew pull --bottle modify more than just the one formula being updated.

[UniqMartin]

@sjackman A classic example are poppler version bumps (e.g. Homebrew/legacy-homebrew#50249). Tests for some dependent formulae will fail and if the OP doesn't notice/understand themselves, we ask them to bump the revision of those formulae, so they are rebuilt and rebottled in the same PR job.

[sjackman]

Ah, I see. Thanks for the explanation. Do you modify the original formula and bump the revisions of the dependents all in the same commit?

[DomT4]

Ideally one commit per formula bump, with the following style:

* poppler 0.43.0
* diff-pdf: revision for poppler
* pdf2htmlex: revision for poppler

And so on.

[UniqMartin]

Do you modify the original formula and bump the revisions of the dependents all in the same commit?

We haven't been entirely consistent with this, unfortunately. I think the preferred style is to have one commit per formula. But you can also find instances of

• formula bump in one commit and revision bumps for affected dependents in a second commit
• formula bump and revision bumps of affected dependents all in a single commit

in the recent Git history of homebrew/core.

[apjanke]

@ahundt: Do you have a list of some other homebrew/science formulae which tend to get broken dependencies like this? It would be helpful to have some more concrete examples. And we can make sure they are testable so they're at least taking advantage of Homebrew's current QA process.

[skystrife]

I think icu4c is another concrete example of this problem. See https://github.com/Homebrew/homebrew-php/issues/2544#issuecomment-208770839.

While having the repos all separate is great for separation of concerns, it's a little troublesome to update a library in homebrew-core that other formulae depend on from other repos. In icu4c's case, we've got dependants in homebrew-php, homebrew-games, homebrew-science, and homebrew-core itself.

Dealing with the homebrew-core dependents is easy: just ensure that the pull request also includes revision bumps for all formulae that have an explicit (non-optional) dependency on icu4c. The existing CI infrastructure works well to tell you if there are any formulae in homebrew-core that need to have their bottles remade (provided, of course, that they actually have tests defined. Let's just assume that they do for now, for the sake of argument).

Dealing with the other repos' dependents is harder. CI currently does detect the breaks in the other repos. While we can immediately issue pull requests on those repos as soon as the version bump is merged into homebrew-core (which is what I did for this icu4c version bump), there is still going to be a window of time where packages in the other repos will break because of the dependency version bump. (Only one of the two pull requests in the other repos has merged, so currently there are packages that are broken because they are looking for the older version of icu4c in homebrew-games and homebrew-science.)

I'm not sure about the right way to solve this. One option would be to somehow coordinate between the repositories so that all of the library-version-bump-related PRs are merged (nearly) simultaneously. This seems hard to do, though, since unpredictable CI errors do happen (something builds just fine locally, but not on e.g. a different OS X version).

Another option would be to have the formulae that list icu4c as a dependency specify a specific version that they target. In theory, couldn't brew then detect that installing a newer icu4c formula would break the existing formulae installed on the system that have not been updated and either spout a warning or just outright bail from updating icu4c until they're ready?

[apjanke]

Do you modify the original formula and bump the revisions of the dependents all in the same commit?

The one vs many commits is more a stylistic issue. Regardless of how many commits it is, for formulae in the same repo, all the commits go in the same PR which is merged all-or-nothing, so they're tested and committed as a unit, and users will pick up all of them in the same brew update, so there's never a time when they are out of sync for a given Homebrew installation.

But for formulae in separate repos, skystrife hits it on the head:

While having the repos all separate is great for separation of concerns, it's a little troublesome to update a library in homebrew-core that other formulae depend on from other repos.
...
While we can immediately issue pull requests on those repos as soon as the version bump is merged into homebrew-core (which is what I did for this icu4c version bump), there is still going to be a window of time where packages in the other repos will break because of the dependency version bump.

We do not have a mechanism for linking PRs across repos, testing them together, or making sure they get tested and deployed as a unit. So the current mechanism actually requires this out-of-sync window, because the PRs for dependent repos' revision bumps can only be run in CI after the version bumps to core are committed. (And since we only have 1 CI worker per OS, the PRs for multiple dependent repos must be serialized. And wait on whatever other CI traffic there is.)

I don't know how to solve this currently. I think that "coordinating between repositories" would require adding some special metadata in PRs that brew test-bot recognized and used to test them as a unit. It's brew test-bot, not Jenkins code, which pulls in the formula PRs for testing, so that could be solved with custom Homebrew code, and not require GitHub and Jenkins to support it. But probably non-trivial.

The limiting factor here is again whether the dependent formulae are testable. Right now, that means having a test do block. So, no test do in a dependent, and we still wouldn't see the breakage reported in a test run. I don't have numbers to support this, but I suspect most of the dependency-version breakage users are actually encountering is probably due to lack of testing, and not repo synchronization. The sync breakage window is maybe an hour or so. The breakage window for a dependent that is missing a test do is until somebody notices it's broken, reports it, and gets a revision-bump PR through. And unfortunately it's probably going to be a user noticing it in this case.

Perhaps we could also add an automatic dylib linkage test that's done regardless of whether a test do is defined: install the dependents, check each with brew linkage, and consider broken link references to be a test failure. That way all dependents, regardless of their test do definition, will have dylib version breakage caught.

Both of these are probably necessary to get to a "perfect" state, because they're independent sources of possible breakage. (Plus restoring --recursive to brew uses, or flattening all dependencies.)

couldn't brew then detect that installing a newer icu4c formula would break the existing formulae

IIRC, brew actually does something like this already: it checks for dylib linkage, and if the old version of icu4c has installed formulae still linked to its versioned dylibs, it will not be uninstalled until those dependencies have been uninstalled. For Homebrew installations with existing formula installations, it's the removal of the old version, not installation of the new version, which breaks things.

This doesn't cover the case of keeping bottles in sync, though: the case that breaks more often is when you do a fresh installation of icu4c and some dependents. In that case, there's no prior installation for brew to hold on to for dylib linkage reasons.

[apjanke]

That brings up another question, @ahundt: Are you seeing this breakage on systems where these things are already installed, and a brew upgrade breaks them? Or is it that sometimes when newly installing things, you end up with a broken installation?

[derrabus]

@apjanke: If on a fresh system after the icu4c 56.1 bump you installed the php56-intl package, you received icu4c 56.1 as well as binaries for php56-intl that were compiled against icu4c 55.1. So yes, this applies to newly installing packages as well.

[MikeMcQuaid]

@scpeters In that case the formula should depend on whatever is the newest version of boost it supports.

[scpeters]

@MikeMcQuaid thanks for the clarification. What would happen when trying to install both packages at the same time? Would the older version of boost be installed as keg-only for the sake of the package that requires it?

Also, if there was a 3rd package that depended on packages that have varying support for the latest versions of boost, they might run into a conflict.

A: depends on boost (latest)
B: depends on boost155 (newest feasible version)
C: depends on A, B

Would the same version of boost be used when installing C? Seems like if the boost versions are different that there could be linking errors.

[MikeMcQuaid]

@scpeters They would both be installed either side-by-side or keg-only depending on conflicts and if we're able to resolve them.

In that case it'd be up to the author of C to resolve the issue. Currently that formula would not work at all or have the same problem with homebrew/versions. I'd rather focus on known issues with formulae rather than speculative issues; we can always iterated and improve any approach we take.

[sjackman]

Since boost155 is installed :keg_only, I don't seen any problem with the above situation. C shouldn't see an issue. It has no direct dependency on boost.

[scpeters]

Thanks for the responses. I spend a big part of my time maintaining software for Ubuntu/debian, which define distributions with specific versions of each package, which is a different model than homebrew, which generally uses the latest version of everything. In my head, the example about packages A,B,C seemed relevant, but maybe it's more relevant to the Ubuntu/debian model.

I'll see if I can think of a concrete example for which it would impact homebrew.

[MikeMcQuaid]

@scpeters At this point it's probably worth critiquing implementations rather than theory so let's hold off until we have that.

[ahundt]

Sorry for my lack of responses, I've been traveling the last two weeks. I like the ideas that have been discussed, is there something that can be put into effect?

@MikeMcQuaid I've actually encountered the "theoretical" problem mentioned by @scpeters in real use cases. Particularly with libraries that support OpenCV 2.x vs OpenCV 3.x, vtk 5 vs 6 and 6 vs 7, Qt 4.x vs Qt 5.x, and a variety of boost versions, particularly around the introduction of new boost libraries (boost.geometry in my case) between 1.4x and 1.5x, and compilation of libraries during the transition between cmake versions 2.8.x with x<10 to 3.x. I don't think this situation is uncommon as different libraries take different amounts of time to support newer versions of their dependencies, particularly widely used ones.

Here is a specific comment on the effects of version transitions from another user: PointCloudLibrary/pcl#1563 (comment)

[ahundt]

Regarding #62 where I'd sometimes like to remap dependencies, could there be a command line command like brew install mypackage --override-dep boost ahundt/boost155. If we can't automate the solution perhaps something like that may allow workarounds to be effective with less effort.

[apjanke]

A couple of the smaller ones are now in effect.

• We're now checking all formulae for broken dylib linkage, not just testable ones. So we can catch in-tap dependencies that need revision bumps that were overlooked by PR submitters. We don't have the resources to run this cross-tap at this point, though.
• I rearranged the brew audit output format to work better when auditing large batches of formulae, like brew audit --strict a*.rb. We can use this to go through homebrew/science to clean up formulae and add missing tests, making more of them testable so they get covered by test-bot when there are changes. This should catch some additional breakage. I'll be doing this as I have time over the next couple weeks, and other contributors with free time can as well. The test bot queues are fast enough now that I think we're okay to take audit-fix-only PRs, and we're always happy accepting PRs that add test dos to formulae that lack them.
  • And if we get enough of them testable, we can take a pass at running brew test-bot on all of them (on home machines, not the CI farm). Could get us to a more stable baseline.

Don't know about the others. And this thread has gotten long enough that I've kind of lost track of all the items which were proposed.

[MikeMcQuaid]

Regarding #62 where I'd sometimes like to remap dependencies, could there be a command line command like brew install mypackage --override-dep boost ahundt/boost155. If we can't automate the solution perhaps something like that may allow workarounds to be effective with less effort.

I can 100% guarantee we won't support something like that that will make things extremely hard to debug.

I don't think this situation is uncommon as different libraries take different amounts of time to support newer versions of their dependencies, particularly widely used ones.

I think in this case it's on the packager to avoid such conflicts.

[ahundt]

@apjanke I tried to add a summary of proposals and additional problems to the very top of this. here is the tl;dr

Key problems

1. Package A is updated from 1.0 to 2.0, breaking package B which depends on A 1.0.
   • breakage is not always detected
     • Example: ceres depends on an exact version of eigen (a simple bottle refresh could fix it)
   • breakage can't always be fixed
     • B may not support A 2.0 until its next release a month from now
2. Package B,C depend on different versions of package A. Package D depends on both B, C and sometimes A. (other variations on this theme exist)
3. Options sometimes break invisibly
4. Cross tap dependencies may change

Solutions with most backing

These are reasonably easy, well liked ideas that will mitigate (but may not completely solve) the above

• allow a limited set of versions into homebrew supported taps with an aliasing mechanism for the latest version so dependent libraries can have a soft landing until they support new versions
• [define criteria for accepting requests/proposals to make certain options recommended by default](make certain options recommended by default)

[MikeMcQuaid]

Package B,C depend on different versions of package A. Package D depends on both B, C and sometimes A. (other variations on this theme exist)

I don't actually think this is a problem; it's something that blocks package D from being packaged in pretty much any package manager and is a problem with their development process.

Otherwise: yeh, I agree 👍

[CamJN]

I just stumbled across this issue and I have been frustrated with version incompatibilities before (reverting QT5.6 to 5.5 for the capybara-webkit gem was annoying and isn't going to be fixed so now qt5 has to be pinned forever, and I can never build anything that requires QT5.6+ at build time without remembering to install the new version and then switch the links back when I'm done so that the next time I bundle update everything's not broken again).

However I'd rather deal with that migraine than this bit: For mysql I see us suffixing the binary names and data directories but probably not e.g. ports. Breaking the data directory for a database across versions? Where does the data actually live? Is the data migrated forward with each upgrade? Even major upgrades? But what if something depends on the old db version (the whole point), do you have to create a cluster from all versions and sync the data constantly? Because that often isn't possible across DB versions (every database I've ever used has done this at one point or another; opentsdb/influx/mysql/postgres just recently). If you want to avoid a huge support nightmare I'd steer away from that idea hard. It's already hard enough to talk to the postgres/mysql people because homebrew's not a blessed/supported config so they can just point at that and not help.

I think there's a lot of good that could come from versioning some libraries but please please please don't do it to the databases.

[MikeMcQuaid]

I think there's a lot of good that could come from versioning some libraries but please please please don't do it to the databases.

We will version databases in some form, just not sure what that will be yet. As a counterpoint under the current system if you have MySQL 5.6 data and upgrade to 5.7 temporarily then getting your data 5.6 compatible again is a non-trivial process.

[ahundt]

Package B,C depend on different versions of package A. Package D depends on both B, C and sometimes A. (other variations on this theme exist)

I don't actually think this is a problem; it's something that blocks package D from being packaged in pretty much any package manager and is a problem with their development process.

You're right. Thinking about it more, I guess I mostly run into this when adding new dependencies which supply new functionality to software I'm developing myself, or when porting software from another OS or package management system like ubuntu. I guess I was just thinking of "things that would be nice to avoid" and not "things a package manager can actually do anything about", haha.

[geoff-nixon]

Thinking about it more, I guess I mostly run into this when adding new dependencies which supply new functionality to software I'm developing myself ...

I'm probably going to catch some shit for even mentioning this, but for several years now I've been using a personal fork of homebrew which does not use dynamic linkage at all. Its a pretty simple hack, just replacing every instance of "./configure" with ./configure --disable-shared --enable-static --with-pic for autotools-based formulae, and adding -DBUILD_SHARED_LIBS=0 to "cmake" for cmake-based based formulae.

This (obviously) takes care of this issue (and most other linkage headaches) for 95% of the formulae I use, and (in most likelihood) at least half of all formulae in general. For the rest I use --interactive to eliminate dynamic linkage manually.

There are so many reasons to do this, the most important of which, for me, as I'm developing my own software, is to avoid things like this; but when developing an app/program that will be distributed in binary form, I actually don't really understand how anyone would be able to use homebrew without doing something like this.

[MikeMcQuaid]

@geoff-codes The opposite: I'm pretty interested in learning more about that and what it would take to make your fork a bit easier.

[geoff-nixon]

Oh!

In the past I've had several unpleasant run-ins with some homebrew maintainers who vociferously advocated for dynamically linking as much as possible (that issue I linked above doesn't address this exact idea per se, but I trust you can gather my meaning). I have thus largely avoided proposing anything related to what is now the "brew" repo, for some time now. I have even avoided putting anything in a public fork: my worry was someone might come across it, adopt it, and suddenly a few weeks I'd be maintaining a fork to which the upstream maintainers were diametrically opposed.

But I know there's been a significant "changing of the guard" since that period, so, if you're serious:

---

There are actually several "side-advantages" of doing this as well, namely,

• Many, many formulae that are now either keg-only or in -dupes no longer need be so, because there is no issue any with library paths or such nonsense, nor worries about ABI compatibility with system libraries. I actually usually intentionally shadow some system libraries. For example, I can (and do) use the newer, and significantly faster whilst also being able to target older SDKs (I usually build against 10.7). And I even have fully-linked, non-sequestered libiconv installed, believe it or not 😈.

I get faster code and great compatibility across systems. All it costs me is a few kb's in my executables.

• It also makes a homebrew install almost completely portable between directories and systems. I don't having to worry about instal_name_tool -changeing thousands of executables and libraries. I have an expanded brew script that basically:
  1. Caches its own location, and checks to make sure it hasn't been moved since the last invocation.
  2. If it detects a move, it 'grep-and-sed's any references to the location that exist in text files (we're basically only talking about .pc files, then reinstalls openssl, which is one a small handful of formulae that hardcode their absolute paths within the binaries themselves.
• It also makes cross-platform compatibility significantly less painful. I use the same basic principal (and a lot of the same code) with a very modified linuxbrew that produces binaries and libraries that are largely interchangeable between distros.

Presently, the code I have is pretty crude, and mostly consists of shell scripts that copy formulae to a temporary directory, 'grep-and-sed's as appropriate; then installs the formula using a file:// URI.

So, if indeed this is something you'd want to consider experimenting with, I'd be happy to clean it all up and implement it in Ruby for something you could pull into a topic branch...?

[sjackman]

with a very modified linuxbrew that produces binaries and libraries that are largely interchangeable between distros.

Linuxbrew bottles can run on any distro. The bottles target glibc 2.19 and gcc 4.8. If the host's versions are older than this, it seamlessly installs the Linuxbrew glibc and gcc, which takes about five minutes, since gcc is bottled. I think your goal however is to distribute your compiled executables outside of Linuxbrew?

[sjackman]

As an alternative to static linking, you can copy all the shared executables that an executable depends upon into its libexec directory and put that directory first in its RPATH. I prefer that solution to static linking myself.

[sjackman]

If the executable is installed in …/bin and its libraries in …/libexec and you set the RPATH to $ORIGIN/../libexec, the executable is relocatable without needing to fix up the executable using patchelf.

[scpeters]

@geoff-codes the static linking idea sounds to me a bit like Ubuntu snappy, which I think is great for leaf applications. If you want to build a framework to use in other applications, my impression is that static linking would be less helpful. I wonder what percentage of homebrew installs are for leaf applications vs. frameworks.

[geoff-nixon]

@sjackman

I think your goal however is to distribute your compiled executables outside of Linuxbrew?

It's both. That is, one goal is to be able to compile executables that can be distributed completely outside/without brew, while still being able to use brew handle building libraries, dependencies, as well as building utilities, toolchain, etc.

As an alternative to static linking, you can copy all the shared executables that an executable depends upon into its libexec directory and put that directory first in its RPATH. I prefer that solution to static linking myself.

Yeah, I'm just going to have to disagree with you there.

@scpeters Yes, I agree its a bit like snappy, but it is exactly sta.li.

In fact, their FAQ makes all the points I would make on this argument.
And its highly recommended reading to anyone interested, no matter which side of the aisle you're on on this topic.

[MikeMcQuaid]

Closing in favour of #620. Thanks for all the discussion, folks.