Podman container on Windows cannot access host by ip or dns. Linux and Mac OS do not have this issue.

[AddictArts]

/kind bug

Description

On Windows the container cannot access the host network, localhost, 127.0.0.1, ::1, or by ip assigned with DHCP. An example is podman run -d --name hasura -p 8080:8080 -e HASURA_GRAPHQL_DATABASE_URL=postgres://postgres:[email protected]:5432/postgres -e HASURA_GRAPHQL_ENABLE_CONSOLE=true -e HASURA_GRAPHQL_DEV_MODE=true -e HASURA_GRAPHQL_ENABLE_TELEMETRY=false docker.io/hasura/graphql-engine:v2.5.0

The hasura container will not see the postgres instance on 192.168.1.5 at port 5432. The exact same on Mac OS does not have this issue. It can see the host services without any special config.

Steps to reproduce the issue:

1. Run the command above with a local running postgres.

Describe the results you received:

Error the container tries to connect and fails.

Describe the results you expected:

I expect it to connect and finish running.

Output of podman version:

4.0.3

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.24.3
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.1.0-2.fc35.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 16
  distribution:
    distribution: fedora
    variant: container
    version: "35"
  eventLogger: file
  hostname: DESKTOP-73DQB37
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.16.3-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: journald
  memFree: 50316668928
  memTotal: 53706846208
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.4.4-1.fc35.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.4.4
      commit: 6521fcc5806f20f6187eb933f9f45130c86da230
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.12-2.fc35.x86_64
    version: |-
      slirp4netns version 1.1.12
      commit: 7a104a101aa3278a2152351a082a6df71f57c9a3
      libslirp: 4.6.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.3
  swapFree: 13958643712
  swapTotal: 13958643712
  uptime: 30h 56m 17.1s (Approximately 1.25 days)
plugins:
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.7.1-2.fc35.x86_64
      Version: |-
        fusermount3 version: 3.10.5
        fuse-overlayfs: version 1.7.1
        FUSE library version 3.10.5
        using FUSE kernel interface version 7.31
  graphRoot: /home/user/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 3
  runRoot: /run/user/1000/containers
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 4.0.3
  Built: 1648837274
  BuiltTime: Fri Apr  1 11:21:14 2022
  GitCommit: ""
  GoVersion: go1.16.15
  OsArch: linux/amd64
  Version: 4.0.3

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

[rhatdan]

@n1hility PTAL

[n1hility]

Hey @AddictArts , I have some suggestions that might work for you. The way that WSL does networking is a little different than a classic nested NAT. It's effectively two parallel side-by-side networks with a relay from windows -> linux between them. So in a nutshell, it's easier for windows apps to talk to linux apps than the other way around. The way it works you can't route to 127.0.0.1 on the windows system, but you can route to the internal vlan virtual adapter (called "vEthernet (WSL) " as if it was external. So the solution is to use the vEthernet WSL ip address of your host as the bind address. This should be something like " 172.26.128.1" but IIRC there is some randomization on setup. You can get this address by doing running ipconfig. For example on my host I have:

Ethernet adapter vEthernet (WSL):

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::9d6b:48c0:d043:5684%9
   IPv4 Address. . . . . . . . . . . : 172.26.128.1
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . :

So you could configure postgres to listen on whatever your 172 address is, and then this address should route from your containers. Alternatively, you could bind to 0.0.0.0 and rely on Windows firewall policy (e.g. restrict the remote side to something like 172.26.0.0/16). In either case though the traffic will be tagged as coming from public, so if you get a Windows Defender prompt for the first postgres run and you do not check the public network, all packets will be dropped. If thats the case you can jump into the Windows Defender Firewall Security settings and delete the block rules for the postgres process and restart (windows defender will reprompt in that case).

If you did bind to 0.0.0.0 you would be able to also connect to your 10.x adapter address as you mentioned but only if your firewall policy allows inbound connections that are routable from 172.

[AddictArts]

Thank you @n1hility Binding the Windows service postgres to the 172.x.x.x vEthernet address did the trick. However, the Windows firewall was blocking it and not listed in the apps. So adding the postgres.exe in the bin/ dir in Program Files did the trick for the public network.

Now the only issue I have is that the container does not run as a daemon. I have to use podman start --attach hasura to keep it alive.

[n1hility]

@AddictArts Glad that’s working for you! Could you share more about the daemon problem. Are you saying that if you start a container with -d that it stops running immediately? Thanks!

[AddictArts]

@n1hility yes that is exactly right. -d or -dt do not prevent the process from exiting. Using —attach above keeps a stdout etc and the process keeps on executing. I had this exact same issue with running Postgres in a container from podman windows.

[n1hility]

Thanks for the info @AddictArts. Could you give me a set of instructions to reproduce? Does it happen as well on the wsl prompt? (For example, if you do wsl -d podman-machine-default, and then from that prompt run podman commands does that work?)

[AddictArts]

Sure can @n1hility

From a Windows C:\ prompt using CMD.exe run
podman run -d --name postgres -p 5432:5432 -e POSTGRES_PASSWORD=postgres docker.io/library/postgres:14.2

Postgres will not stay running. This is easier than using the one in this issue since it requires a running postgres Also note you cannot have any others running. If another is running, example without -d above then launch another on a different port, that one will stay a daemon oddly. I know that sounds odd, but be sure podman ps shows nothing.

Note: After the above run and it terminating shortly after, if I run podman start --attach postgres then it will continue running of course showing the log or stdout in the CMD. If you CTRL-C, the process will terminate. Also, note -dt does not help either for the initial run.

As to wsl -d podman-machine-default, it is running and it does have poman. However a podman ps or a podman containers list -a does not show the Windows podman executed containers.

[davdr]

Regarding @n1hility's reply #13966 (comment) on the changing WSL2 IP address:

To avoid manually looking up IP's after each WSL2 startup, I make use the fact that this address also shows up as the first address output by hostname -I. When you run your container with something like:

podman run --add-host=host.wsl.internal:$(hostname -I | cut -f 1 -d " ") …​

you can use the "host.wsl.internal" name to connect to ports on Windows side, or ports exposed by other containers even. Of course you can choose whatever name you like.

This assumes executing the podman command in WSL, not on the CMD or PowerShell prompt though.

[n1hility]

Great tip @davdr

I am going to close this one since the remaining issue is in #13965

[n1hility]

BTW you can do something similar in powershell with $myip = Get-NetIpAddress | where { $_.InterfaceAlias -Like '*WSL*' -and $_.AddressFamily -EQ "IPv4" } |select -ExpandProperty IPAddress