Intel macOS qemu_podman-machine-default.sock: connect: no such file or directory #13609

fithisux · 2022-03-23T12:55:37Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I upgraded my podman with brew to 4.0.2. I tried to use it but it does not work.

Steps to reproduce the issue:

brew uninstall podman
brew install podman

podman machine rm
podman machine init
podman machine start

Describe the results you received:

Starting machine "podman-machine-default"
INFO[0000] waiting for clients...
ERRO[0000] Error listening on socket: /Users/vassilisanagnostopoulos/.local/share/containers/podman/machine/podman-machine-default/podman.sock: listen unix /Users/vassilisanagnostopoulos/.local/share/containers/podman/machine/podman-machine-default/podman.sock: bind: invalid argument
Error: dial unix /var/folders/70/tpb1l8jd45s8zj742ljrmbs80000gp/T/podman/qemu_podman-machine-default.sock: connect: no such file or directory

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman. failed to create sshClient: Connection to bastion host (ssh://core@localhost:51528/run/user/502/podman/podman.sock) failed.: dial tcp [::1]:51528: connect: connection refused

Output of podman info --debug:

Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman. failed to create sshClient: Connection to bastion host (ssh://core@localhost:51528/run/user/502/podman/podman.sock) failed.: dial tcp [::1]:51528: connect: connection refused

Package info (e.g. output of rpm -q podman or apt list podman):

(paste your output here)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
Darwin C02CH2W2MD6R.local 20.6.0 Darwin Kernel Version 20.6.0: Wed Jun 23 00:26:31 PDT 2021; root:xnu-7195.141.2~5/RELEASE_X86_64 x86_64

The text was updated successfully, but these errors were encountered:

Luap99 · 2022-03-23T13:18:53Z

On darwin and other BSD operation system the maximum path length for the socket path seems to be 104 chars. On linux it is 108 chars. Your path is 105 chars long.

Can you try to create a user with a shorter name and see if this works?

baude · 2022-03-23T13:20:34Z

consider the following:

podman system connection ls ; how many listings are there. Are you certain the default listing is correct for the default machine? if you dont know and don't care, rm the machine; then double check system connections. if some still exist, remove them and start over at init.
do the dirs being cited exist? do you have write permissions to them?
are there any security policies on your mac that are blocking things?

baude · 2022-03-23T13:20:57Z

wooh, @Luap99 nice find.

Luap99 · 2022-03-23T13:22:03Z

@baude Maybe this workaround helps:

podman/cmd/rootlessport/main.go

Lines 201 to 217 in f049cba

    
           // workaround to bypass the 108 char socket path limit 
        
           // open the fd and use the path to the fd as bind argument 
        
           fd, err := unix.Open(socketDir, unix.O_PATH, 0) 
        
           if err != nil { 
        
           	return err 
        
           } 
        
           socket, err := net.ListenUnix("unixpacket", &net.UnixAddr{Name: fmt.Sprintf("/proc/self/fd/%d/%s", fd, cfg.ContainerID), Net: "unixpacket"}) 
        
           if err != nil { 
        
           	return err 
        
           } 
        
           err = unix.Close(fd) 
        
           // remove the socket file on exit 
        
           defer os.Remove(socketfile) 
        
           if err != nil { 
        
           	logrus.Warnf("Failed to close the socketDir fd: %v", err) 
        
           } 
        
           defer socket.Close()

But I have no idea if this would work on darwin.

baude · 2022-03-23T13:22:14Z

@Luap99 or @fithisux you could also create a shortened machine name. i.e. podman init --now foo

baude · 2022-03-23T13:27:26Z

verified on my m1 that indeed the long name throws the same errors.

baude · 2022-03-23T14:05:06Z

i got this one ..,.

fithisux · 2022-03-23T14:14:27Z

On darwin and other BSD operation system the maximum path length for the socket path seems to be 104 chars. On linux it is 108 chars. Your path is 105 chars long.

Can you try to create a user with a shorter name and see if this works?

Unfortunately this is not possible. But my issue is that the files / sockets is not there at all

`Last login: Wed Mar 23 15:14:37 on ttys001
➜ ~ ls /var/folders/70/tpb1l8jd45s8zj742ljrmbs80000gp/T/podman

➜ ~ ls -l /Users/vassilisanagnostopoulos/.local/share/containers/podman/machine/podman-machine-default

➜ ~`

fithisux · 2022-03-23T14:16:25Z

podman init --now foo

This does not seem to work

➜ ~ podman init --now foo
Error: unknown flag: --now
See 'podman init --help'
➜ ~

Luap99 · 2022-03-23T14:21:11Z

use podman machine init ...

fithisux · 2022-03-23T14:38:06Z

use podman machine init ...

Works now. Thank you.

if the machine name is long and/or the username is long, it is possible to exceed the 104byte limit for filenames in MacOS. This PR shortcuts things to $HOME/.podman instead. Fixes: containers#13609 [NO NEW TESTS NEEDED] Signed-off-by: Brent Baude <[email protected]>

afbjorklund · 2022-03-26T15:15:35Z

You should still be able to use ssh sockets just fine, it's just the legacy unix sockets that have this limit.

to avoid errors on macos, we use symlinks to long socket names. Fixes: containers#12751 Fixes: containers#13609 Signed-off-by: Brent Baude <[email protected]> [NO NEW TESTS NEEDED] Signed-off-by: Brent Baude <[email protected]>

pigping88 · 2022-12-16T04:11:20Z

mac apple A1 , podman 4.3.1
podman machine init
podman machine start

Starting machine "podman-machine-default"
Waiting for VM ...
Error: dial unix /var/folders/f3/v585c0593yg1v4qqs5tjnw600000gn/T/podman/podman-machine-default_ready.sock: connect: no such file or directory

podman machine ssh

Connecting to vm podman-machine-default. To close connection, use `~.` or `exit`
Fedora CoreOS 37.20221211.2.0
Tracker: https://github.com/coreos/fedora-coreos-tracker
Discuss: https://discussion.fedoraproject.org/tag/coreos

ssbarnea · 2023-01-25T10:45:02Z

Reopening because this issue is still happening with GHA runners as today, see https://github.com/ansible/vscode-ansible/actions/runs/4005037985/jobs/6874912745

2023-01-25T10:34:30.5842600Z Downloading VM image: fedora-coreos-37.20230110.2.0-qemu.x8…
2023-01-25T10:34:30.7628400Z �[1A�[JDownloading VM image: fedora-coreos-37.20230110.2.0-qemu.x8…
2023-01-25T10:34:37.9151470Z Extracting compressed file
2023-01-25T10:35:09.9899520Z Image resized.
2023-01-25T10:35:09.9923940Z Machine init complete
2023-01-25T10:35:09.9935290Z Starting machine "podman-machine-default"
2023-01-25T10:35:10.5067840Z Waiting for VM ...
2023-01-25T10:35:13.5424550Z Error: dial unix /var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/podman/podman-machine-default_ready.sock: connect: no such file or directory
2023-01-25T10:35:13.5717740Z task: Failed to run task "setup": exit status 125
2023-01-25T10:35:13.5751490Z ##[error]Process completed with exit code 1.

I think that the command that caused it to fail was podman machine init --now.

ssbarnea · 2023-01-25T10:56:20Z

@baude Any idea what could have cause this regression? What can we do to make the podman initialization reliable on GHA?

baude · 2023-01-25T14:41:30Z

this is sort of a general error for "something went wrong". it is entirely possible that it is different than this issue .... does removing the machine and recreating it help?

ssbarnea · 2023-01-25T14:54:06Z

While ok locally, turn-in-off-and-on does not really work with GHA. Still, I observed that while it does reproduce, it does not always reproduce, so there is a level of randomness in it.

If you could provide some hints regarding how we can get extra logs when happens, I might be able to alter the GHA pipelines to collect the extra information.

If we manage to get podman to be reliable on GHA, we might have a chance on convincing github to add it to the default runner image.

Due to podman bootrapping flakiness on Github Runners, we make the macos testing non-voting for the final outcome. We continue to run it order to see how it evolves in time, hopefully find a permanent fix for it. Related: containers/podman#13609

ctrought · 2023-03-12T00:21:33Z

My system (m1 mac) hit an out of memory condition while podman was running which led to this issue on startup #16945 , rebooted and then podman won't startup now hitting the error in this issue :/ Any workaround that does not involve wiping the original podman machine?

$ podman machine start  --log-level debug
INFO[0000] podman filtering at log level debug
Starting machine "podman-machine-default"
[/opt/podman/qemu/bin/gvproxy -listen-qemu unix:///var/folders/5k/865_x9vd2_3f3k7bw36k87vc0000gp/T/podman/qmp_podman-machine-default.sock -pid-file /var/folders/5k/865_x9vd2_3f3k7bw36k87vc0000gp/T/podman/podman-machine-default_proxy.pid -ssh-port 50044 -forward-sock /Users/XXXXXXXXXXXX/.local/share/containers/podman/machine/podman-machine-default/podman.sock -forward-dest /run/user/502/podman/podman.sock -forward-user core -forward-identity /Users/XXXXXXXXXXXX/.ssh/podman-machine-default --debug
Error: dial unix /var/folders/5k/865_x9vd2_3f3k7bw36k87vc0000gp/T/podman/qmp_podman-machine-default.sock: connect: no such file or directory

spencerrung · 2023-03-16T16:19:17Z

Agree this should be fixed but an easy workaround for M1 mac users

podman machine stop
podman machine rm
podman machine init
podman machine start

vrothberg · 2023-10-18T15:28:51Z

That has been fixed in the meantime, closing.

openshift-ci bot added kind/bug Categorizes issue or PR as related to a bug. kind/feature Categorizes issue or PR as related to a new feature. labels Mar 23, 2022

fithisux changed the title ~~Intel Macos qemu_podman-machine-default.sock: connect: no such file or directory~~ Intel macOS qemu_podman-machine-default.sock: connect: no such file or directory Mar 23, 2022

Luap99 removed the kind/feature Categorizes issue or PR as related to a new feature. label Mar 23, 2022

baude self-assigned this Mar 23, 2022

Luap99 added the machine label Mar 23, 2022

baude added the 4.1 label Mar 23, 2022

baude mentioned this issue Mar 24, 2022

[WIP]change macos directory for machine sockets #13654

Closed

baude mentioned this issue Apr 1, 2022

machine refactor 3: add symlinks for sockets #13750

Merged

openshift-merge-robot closed this as completed in #13750 Apr 6, 2022

ssbarnea reopened this Jan 25, 2023

ssbarnea mentioned this issue Jan 25, 2023

Add extra logging to macos podman machine start sequence ansible/vscode-ansible#767

Merged

ssbarnea mentioned this issue Mar 10, 2023

Temporary make macos testing non-voting ansible/vscode-ansible#794

Merged

This was referenced Sep 26, 2023

MacOS and nix ES-Nix/get-nix#7

Open

The podman statically compiled and QEMU ES-Nix/podman-rootless#15

Open

vrothberg closed this as completed Oct 18, 2023

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Jan 17, 2024

github-actions bot locked as resolved and limited conversation to collaborators Jan 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Intel macOS qemu_podman-machine-default.sock: connect: no such file or directory #13609

Intel macOS qemu_podman-machine-default.sock: connect: no such file or directory #13609

fithisux commented Mar 23, 2022 •

edited

Loading

Luap99 commented Mar 23, 2022

baude commented Mar 23, 2022

baude commented Mar 23, 2022

Luap99 commented Mar 23, 2022

baude commented Mar 23, 2022

baude commented Mar 23, 2022

baude commented Mar 23, 2022

fithisux commented Mar 23, 2022

fithisux commented Mar 23, 2022

Luap99 commented Mar 23, 2022

fithisux commented Mar 23, 2022

afbjorklund commented Mar 26, 2022

pigping88 commented Dec 16, 2022

ssbarnea commented Jan 25, 2023

ssbarnea commented Jan 25, 2023

baude commented Jan 25, 2023

ssbarnea commented Jan 25, 2023

ctrought commented Mar 12, 2023 •

edited

Loading

spencerrung commented Mar 16, 2023 •

edited

Loading

vrothberg commented Oct 18, 2023

Intel macOS qemu_podman-machine-default.sock: connect: no such file or directory #13609

Intel macOS qemu_podman-machine-default.sock: connect: no such file or directory #13609

Comments

fithisux commented Mar 23, 2022 • edited Loading

Luap99 commented Mar 23, 2022

baude commented Mar 23, 2022

baude commented Mar 23, 2022

Luap99 commented Mar 23, 2022

baude commented Mar 23, 2022

baude commented Mar 23, 2022

baude commented Mar 23, 2022

fithisux commented Mar 23, 2022

fithisux commented Mar 23, 2022

Luap99 commented Mar 23, 2022

fithisux commented Mar 23, 2022

afbjorklund commented Mar 26, 2022

pigping88 commented Dec 16, 2022

ssbarnea commented Jan 25, 2023

ssbarnea commented Jan 25, 2023

baude commented Jan 25, 2023

ssbarnea commented Jan 25, 2023

ctrought commented Mar 12, 2023 • edited Loading

spencerrung commented Mar 16, 2023 • edited Loading

vrothberg commented Oct 18, 2023

fithisux commented Mar 23, 2022 •

edited

Loading

ctrought commented Mar 12, 2023 •

edited

Loading

spencerrung commented Mar 16, 2023 •

edited

Loading