podman machine start cannot find gvproxy when brew is not installed in the standard directory #12161

carlos-algms · 2021-11-01T22:22:17Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

It is not possible to run podman machine start when brew is not installed in the default location.

Steps to reproduce the issue:

Install brew on your user directory, as described in the official docs:
mkdir ~/homebrew && curl -L https://github.com/Homebrew/brew/tarball/master | tar xz --strip 1 -C ~/homebrew
Install podman

brew install podman
podman machine init
podman machine start

Describe the results you received:

podman machine start returns:

Error: unable to start host networking: 
"could not find \"gvproxy\" in one of 
[
  /usr/local/opt/podman/libexec 
  /opt/homebrew/bin /opt/homebrew/opt/podman/libexec 
  /usr/local/bin /usr/local/libexec/podman 
  /usr/local/lib/podman 
  /usr/libexec/podman 
  /usr/lib/podman
]"

Describe the results you expected:
Podman should be able to find its install location and pick up the gvproxy bin inside its libexec folder.

I can confirm that gvproxy IS inside ~/homebrew/Cellar/podman/3.4.1/libexec/ folder.

Output of podman version:

Cannot connect to Podman ....

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
MacOS Big Sur 11.3.1

The text was updated successfully, but these errors were encountered:

Luap99 · 2021-11-02T10:10:02Z

see #11960
I think this should be reported against homebrew, there is no way that podman can know about custom install locations.
@ashley-cui PTAL

carlos-algms · 2021-11-02T10:40:50Z

I reported it here because I thought it was podman installing and executing gvproxy as a dependency under the libexec folder.

As it is only located inside podman's installation folder: ~/homebrew/Cellar/podman/3.4.1/libexec/

ashley-cui · 2021-11-02T14:25:15Z

gvproxy is installed by homebrew as a dependency in the podman libexec folder. homebrew does not recommend installing in another directory, and in the docs there is a warning that things may break when homebrew is installed in another place.

carlos-algms · 2021-11-02T14:36:40Z

I know, but gvproxy is the only missing part, if I link it to one of the paths podman listed that it is looking for it, everything works as expected.

Can it then be a new feature?
Where podman just reads its realpath and look for gvproxy in one more folder?
Its own libexec folder?

It would be great for people that do not have root access, as it is blocked by the Enterprise restrictions, so installing in other places is the only alternative.

Thanks.

ashley-cui · 2021-11-02T17:48:59Z

@afbjorklund for most places, it would link to $HOMEBREW_PREFIX/bin or $HOMEBREW_PREFIX/lib, but libexec is special in Homebrew, it doesn't get linked to the homebrew prefix

afbjorklund · 2021-11-02T19:09:04Z

Unfortunately, it looks like the formula is broken.

    resource("gvproxy").stage do
      system "make", "gvproxy"
      libexec.install "bin/gvproxy"
    end

It installs under #{prefix}/libexec/gvproxy

It was supposed to be in a "podman" subdirectory ?

So those hardcoded paths are all wrong:

	// Homebrew install paths
	"/usr/local/opt/podman/libexec",
	"/opt/homebrew/bin",
	"/opt/homebrew/opt/podman/libexec",
	"/usr/local/bin",

Or at least different to the normal ones:

	"/usr/local/libexec/podman",
	"/usr/local/lib/podman",
	"/usr/libexec/podman",
	"/usr/lib/podman",

Maybe other binaries go straight into libexec...

Then again, upstream is even worse:

make: *** No rule to make target 'install'.  Stop.

So I guess it needs to be "translated":

--- a/Formula/podman.rb
+++ b/Formula/podman.rb
@@ -37,6 +37,9 @@ class Podman < Formula
   def install
     os = OS.kernel_name.downcase
 
+    inreplace "vendor/github.com/containers/common/pkg/config/config_linux.go", "/usr/local/libexec/podman", libexec
+    inreplace "vendor/github.com/containers/common/pkg/config/config_darwin.go", "/usr/local/libexec/podman", libexec
+
     system "make", "podman-remote-#{os}"
     if OS.mac?
       bin.install "bin/#{os}/podman" => "podman-remote"

Then the hardcoded paths aren't needed.

@@ -17,12 +17,6 @@
 }
 
 var defaultHelperBinariesDir = []string{
-       // Homebrew install paths
-       "/usr/local/opt/podman/libexec",
-       "/opt/homebrew/bin",
-       "/opt/homebrew/opt/podman/libexec",
-       "/usr/local/bin",
-       // default paths
        "/usr/local/libexec/podman",
        "/usr/local/lib/podman",
        "/usr/libexec/podman",

ashley-cui · 2021-11-02T19:31:53Z

#{prefix} is /usr/local/Cellar/foo/0.1/. $HOMEBREW_PREFIX is different, it's /usr/local.
every formula in cellar is also symlinked into $HOMEBREW_PREFIX/opt/foo. The docs are pretty confusing on this, but I just verified this on my machine.

afbjorklund · 2021-11-02T19:34:36Z

Right, here they translate to:

HOMEBREW_PREFIX: /home/linuxbrew/.linuxbrew

prefix: /home/linuxbrew/.linuxbrew/Cellar/podman/3.4.1

And have to use podman-remote

It installs as a special user (with sudo), just to share precompiled binaries.

I could have installed it under /home/anders, and compiled everything...

https://docs.brew.sh/Homebrew-on-Linux

The installation script installs Homebrew to /home/linuxbrew/.linuxbrew
using sudo if possible and in your home directory at ~/.linuxbrew otherwise.

brew install --build-from-source podman

github-actions · 2021-12-03T00:04:34Z

A friendly reminder that this issue had no activity for 30 days.

rhatdan · 2021-12-03T16:04:09Z

Where do we stand on this issue? Is it still a problem?
@ashley-cui @afbjorklund @carlos-algms WDYT?

ashley-cui · 2021-12-03T16:12:24Z

I don't think this is an issue that should be fixed on our end, its more of a homebrew install issue, and homebrew specifically states that some packages may break if not installed in their default location

afbjorklund · 2021-12-03T16:40:12Z

They forgot to patch the default config, with the current path to use.

Could try a PR (for brew), to replace the hardcoded values ?

#12161 (comment)

I guess any Linux distribution not using /usr prefix has same issue ?

podman: Change hardcoded libexec path Homebrew/homebrew-core#90379

carlos-algms · 2021-12-03T16:49:23Z

I would say it is good for the community in general to have this fixed.
As it was pointed out, the paths are hardcoded and It would be great to have a dynamic check on the installation folder before giving up looking for executables.

I raised this ticket because if your computer is blocked by the company you work for, and you don't have admin/sudo rights, installing on a custom directory is the only option.

afbjorklund · 2021-12-03T16:54:06Z

You can provide your custom directory in configuration, if Homebrew and Podman doesn't want to fix their packaging

afbjorklund · 2021-12-03T18:45:07Z

Homebrew prefers to have their packages unpatched, so left all the unused paths in there.

~~/usr/local/opt/podman/libexec~~
~~/opt/homebrew/bin~~
~~/opt/homebrew/opt/podman/libexec~~
~~/usr/local/bin~~
/usr/local/libexec/podman -> #{HOMEBREW_PREFIX}/Cellar/#{name}/#{version}/libexec
/usr/local/lib/podman
/usr/libexec/podman
/usr/lib/podman

carlocab · 2021-12-06T07:36:37Z

I don't think this is an issue that should be fixed on our end, its more of a homebrew install issue

It seems weird to call this a Homebrew install issue when this is also causing problems for users who aren't even using Homebrew. See #11960 (also referenced above).

afbjorklund · 2021-12-06T08:36:04Z

Technically it is a problem for all "weird" installations that don't use the "usual" /usr (or /usr/local).

But there really isn't any good substitute for ./configure --prefix=$PREFIX either, so needs patching*...

* note: you can use sed, if not allowing patch

Anyway, the path locations needs to be updated:

gvproxy (/usr/libexec/podman/gvproxy)
catatonit (/usr/libexec/podman/catatonit)
conmon (/usr/libexec/podman/conmon)

bowbahdoe · 2022-02-15T15:57:00Z

This is an issue for nix installs

maximsmol · 2022-02-21T20:46:12Z

as a workaround, add this to ~/.config/containers/containers.conf:

[engine]
helper_binaries_dir = [
  "{PATH_TO_GPROXY_BIN}",
  "/usr/local/libexec/podman",
  "/usr/local/lib/podman",
  "/usr/libexec/podman",
  "/usr/lib/podman",
]

on my Mac system using Nix the path is /nix/store/k92v259vn6bv0shsifxhjvavbj436906-gvproxy-0.3.0/bin
you could probably generate this via home-manager but there are other settings that get in the way

I'll work on a PR for nixpkgs soon

afbjorklund · 2022-02-21T21:23:26Z

It could be a problem to set one "helper" dir, if you spread the podman helpers into different packages.

On some system they have relocated them into the podman libexec, while others use a global libexec.

https://packages.debian.org/sid/catatonit
/usr/libexec/podman/catatonit ->
/usr/bin/catatonit

https://github.com/macports/macports-ports/blob/master/net/gvisor-tap-vsock/Portfile
/opt/local/libexec/gvproxy
https://github.com/macports/macports-ports/blob/master/sysutils/podman/files/patch-defaultHelperBinariesDir-for-MacPorts.diff

Possibly one could use symlinks, or something...

Use one podman libexec prefix, linking to the others ?

carlocab · 2022-02-21T22:18:00Z

@ashley-cui I think at this stage it's fairly clear that this issue isn't specific to Homebrew.

Perhaps you'd reconsider implementing a solution in this project? It seems really inefficient for your various downstream distributors to have to carry around duplicate pieces of code to work around this issue.

ashley-cui · 2022-02-21T22:31:57Z

@carlocab I'll bring it up with the team and see what the consensus is. I think we've gone back and forth a lot on where this gvproxy binary should live, but I can bring it up again.

carlocab · 2022-02-22T00:23:40Z

I don't think it's really a matter of where it should live: I think the above thread shows that that's something difficult to control.

To me, it's a matter of providing a mechanism for telling podman where gvproxy will live and documenting this. Whether this mechanism is only invoked at build-time or is also available at run-time is up to you.

bowbahdoe · 2022-02-22T05:24:10Z

@carlocab I will add that there is no way i have found to resolve this when installing podman via nix. Even manually telling it where gvproxy is via editing the config does not let things work

carlocab · 2022-02-22T05:48:59Z

@bowbahdoe something that works for us currently is to replace these lines:

podman/vendor/github.com/containers/common/pkg/config/config_darwin.go

Line 26 in 0cfb5d7

"/usr/local/libexec/podman",

podman/vendor/github.com/containers/common/pkg/config/config_linux.go

Line 40 in 0cfb5d7

"/usr/local/libexec/podman",

with the absolute path to directory we expect to contain gvproxy. You can see this here. That may or may not be something you can also do in nix, depending on when and on what the hashes used to determine the install path is computed. (I don't use nix, so I don't actually know.)

maximsmol · 2022-02-22T05:58:27Z

@bowbahdoe I'm guessing you tried adding my example path? If your nixpkgs is at all different you will get a different derivation and output path. You can get yours by running nix repl '<nixpkgs>' and taking the value of pkgs.gvproxy.outPath. If that's not it, would be great to see what error you are getting so we can figure out whether this is really blocking in some cases on nix

Example:

$ nix repl '<nixpkgs>'
Welcome to Nix 2.6.0. Type :? for help.

Loading '<nixpkgs>'...
Added 15937 variables.

nix-repl> pkgs.gvproxy.outPath
"/nix/store/k92v259vn6bv0shsifxhjvavbj436906-gvproxy-0.3.0"

ashley-cui · 2022-02-22T19:30:22Z

Alright, we'll look into a solution where a location can be set during compile-time using our makefile.

afbjorklund · 2022-02-22T19:35:54Z

I don't understand how Nix can work, without some PATH directory to put the symlinks in (like brew) ?

bin/podman-remote -> ../Cellar/podman/3.4.4/bin/podman-remote

Surely you don't add the installation directory for each and every little package, to the global variable ?

EDIT: Right, it was called a "user environment" (the symlink forest)

But the main hack is to make sure that gvisor-tap-proxy is part of the podman package, not separate.

./Cellar/podman/3.4.4/libexec/gvproxy

(brew and port decided to drop the "podman" dir)

Had it been a real program, it would already have been in the PATH and easy to find at runtime.

Here it is in the podman spec: https://src.fedoraproject.org/rpms/podman/blob/rawhide/f/podman.spec

%package gvproxy

afbjorklund · 2022-02-22T20:08:24Z

@ashley-cui : afaik, you already set config._installPrefix from the Makefile ?

PREFIX ?= /usr/local
BINDIR ?= ${PREFIX}/bin
LIBEXECDIR ?= ${PREFIX}/libexec
LIBEXECPODMAN ?= ${LIBEXECDIR}/podman
MANDIR ?= ${PREFIX}/share/man
SHAREDIR_CONTAINERS ?= ${PREFIX}/share/containers
ETCDIR ?= ${PREFIX}/etc

	-X $(LIBPOD)/config._installPrefix=$(PREFIX) \
	-X $(LIBPOD)/config._etcDir=$(ETCDIR) \

So need to do something similar for the helpers ($(LIBEXECPODMAN)), perhaps.

ashley-cui · 2022-03-01T00:04:28Z

Closed via: #13372

Example usage: make podman-remote HELPER_BINARIES_DIR=/my/location/prefix

deadpyxel · 2022-03-15T16:23:52Z

I'm facing the same issue on MacOs 11.6.4, attempting to get the latest version of podman up and running. I see this has been fixed but is there any special step I need to do before podman machine start?

I have installed the latest qemu but it does not have any gvproxy binary anywhere, neither can a find that binary in my system.

Everything was done using homebrew.

maximsmol · 2022-03-15T17:20:27Z

@deadpyxel there is always this workaround: #12161 (comment)

but you should really open an issue on the homebrew formula, it's in the hands of the package maintainers now

deadpyxel · 2022-03-16T13:01:34Z

I'll check that. It seems homebrew is installing the latest version, but maybe it is something with the package as you say.

Problem is that I can't apply the patch proposed since there's no gvproxy binary available.

This should make it easier to locate helper binaries relative to where the main binary was installed, which should be useful in installations such as Homebrew which install packages under a versioned directory. Use a `$BINDIR` magic token as a prefix in the helper path to indicate it should be relative to the directory where the binary is located. This is somewhat familiar to the syntax used in the shell and Makefile and is still quite explicit about the behavior (as opposed to, say, making all relative paths be relative to the directory of the binary.) Update `podman` config on Darwin to look for helpers such as `gvproxy` under `$BINDIR/../libexec/podman`, which is the ultimate objective of this code change. Tested: Updated vendored package in podman, built it with `podman-remote`, copied `gvproxy` to a `libexec/podman` at the same level as `bin/podman` and confirmed that `podman machine start` worked as expected. Fixes: containers/podman#12161 Related: PR containers#946 Signed-off-by: Filipe Brandenburger <[email protected]>

openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Nov 1, 2021

This comment has been minimized.

Sign in to view

github-actions bot added the stale-issue label Dec 3, 2021

rhatdan removed the stale-issue label Dec 3, 2021

This comment has been minimized.

Sign in to view

afbjorklund mentioned this issue Dec 3, 2021

podman: Change hardcoded libexec path Homebrew/homebrew-core#90379

Closed

6 tasks

This comment has been minimized.

Sign in to view

rhatdan closed this as completed Dec 3, 2021

ashley-cui reopened this Feb 22, 2022

This was referenced Feb 28, 2022

Add additionalHelperBinariesDir containers/common#946

Merged

Allow setting binarypath from Makefile #13372

Merged

ashley-cui closed this as completed Mar 1, 2022

filbranden mentioned this issue Aug 22, 2022

config: Allow specifying helper dirs with $BINDIR as base directory containers/common#1127

Merged

una-tapa mentioned this issue Nov 4, 2022

Let's try InstantOn on Intel Mac OpenLiberty/blogs#2670

Merged

cdrage mentioned this issue Nov 30, 2022

feat: add way to provide additional path for podman binary podman-desktop/podman-desktop#941

Merged

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 20, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 20, 2023

podman machine start cannot find gvproxy when brew is not installed in the standard directory #12161

podman machine start cannot find gvproxy when brew is not installed in the standard directory #12161

Comments

carlos-algms commented Nov 1, 2021 • edited Loading

Luap99 commented Nov 2, 2021

carlos-algms commented Nov 2, 2021 • edited Loading

ashley-cui commented Nov 2, 2021

carlos-algms commented Nov 2, 2021

This comment has been minimized.

ashley-cui commented Nov 2, 2021

This comment has been minimized.

afbjorklund commented Nov 2, 2021 • edited Loading

ashley-cui commented Nov 2, 2021 • edited Loading

afbjorklund commented Nov 2, 2021 • edited Loading

github-actions bot commented Dec 3, 2021

rhatdan commented Dec 3, 2021

ashley-cui commented Dec 3, 2021

afbjorklund commented Dec 3, 2021 • edited Loading

This comment has been minimized.

carlos-algms commented Dec 3, 2021 • edited Loading

afbjorklund commented Dec 3, 2021 • edited Loading

This comment has been minimized.

afbjorklund commented Dec 3, 2021 • edited Loading

carlocab commented Dec 6, 2021 • edited Loading

afbjorklund commented Dec 6, 2021 • edited Loading

bowbahdoe commented Feb 15, 2022

maximsmol commented Feb 21, 2022

afbjorklund commented Feb 21, 2022 • edited Loading

carlocab commented Feb 21, 2022

ashley-cui commented Feb 21, 2022

carlocab commented Feb 22, 2022

bowbahdoe commented Feb 22, 2022 • edited Loading

carlocab commented Feb 22, 2022

maximsmol commented Feb 22, 2022

ashley-cui commented Feb 22, 2022

afbjorklund commented Feb 22, 2022 • edited Loading

afbjorklund commented Feb 22, 2022 • edited Loading

ashley-cui commented Mar 1, 2022

deadpyxel commented Mar 15, 2022 • edited Loading

maximsmol commented Mar 15, 2022

deadpyxel commented Mar 16, 2022 • edited Loading

carlos-algms commented Nov 1, 2021 •

edited

Loading

carlos-algms commented Nov 2, 2021 •

edited

Loading

afbjorklund commented Nov 2, 2021 •

edited

Loading

ashley-cui commented Nov 2, 2021 •

edited

Loading

afbjorklund commented Nov 2, 2021 •

edited

Loading

afbjorklund commented Dec 3, 2021 •

edited

Loading

carlos-algms commented Dec 3, 2021 •

edited

Loading

afbjorklund commented Dec 3, 2021 •

edited

Loading

afbjorklund commented Dec 3, 2021 •

edited

Loading

carlocab commented Dec 6, 2021 •

edited

Loading

afbjorklund commented Dec 6, 2021 •

edited

Loading

afbjorklund commented Feb 21, 2022 •

edited

Loading

bowbahdoe commented Feb 22, 2022 •

edited

Loading

afbjorklund commented Feb 22, 2022 •

edited

Loading

afbjorklund commented Feb 22, 2022 •

edited

Loading

deadpyxel commented Mar 15, 2022 •

edited

Loading

deadpyxel commented Mar 16, 2022 •

edited

Loading