Podman 2.2.1 adds an additional 127.0.1.1 to /etc/hosts breaking services binding to hostname when --net=host is specified

[aressem]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When using --net=host and additional entry in /etc/hosts is created where the hostname is bound to 127.0.1.1
This has started to happen for Podman 2.2.1 on RHEL 8.3.1.

Steps to reproduce the issue:

1. Start a container on a host and install ruby:

$ sudo podman run  -ti --net=host docker.io/centos:8 bash
# dnf install -y ruby

2. Execute the ruby script to start a distributed ruby instance:

#!/usr/bin/env ruby

require 'drb'
DRb.start_service("druby://#{`hostname`.chomp}:23456", nil)
DRb.thread.join

3. Log into the host again and inspect the port the ruby server is listening on:

$ netstat -natup | grep LIST
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1679/sshd
tcp        0      0 127.0.1.1:23456         0.0.0.0:*               LISTEN      296089/ruby
tcp6       0      0 :::22                   :::*                    LISTEN      1679/sshd

Describe the results you received:
Content of /etc/hosts:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.1.2.19 vespa.myhost.com
127.0.1.1 vespa.myhost.com vespa.myhost.com flamboyant_mclaren

Describe the results you expected:
Content of /etc/hosts:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.1.2.19 vespa.myhost.com

This is what we get on RHEL 8.3.0 with podman-2.0.5-5.module+el8.3.0+8221+97165c3f.x86_64.

Additional information you deem important (e.g. issue happens only occasionally):
N/A

Output of podman version:

$ podman version
Version:      2.2.1
API Version:  2
Go Version:   go1.14.7
Built:        Mon Feb  8 21:19:06 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.22-3.module+el8.3.1+9857+68fb1526.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.22, commit: a40e3092dbe499ea1d85ab339caea023b74829b9'
  cpus: 24
  distribution:
    distribution: '"rhel"'
    version: "8.3"
  eventLogger: file
  hostname: vespa9.factory.corp.gq1.yahoo.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 100
      size: 1
    uidmap:
    - container_id: 0
      host_id: 93975
      size: 1
  kernel: 4.18.0-240.15.1.el8_3.x86_64
  linkmode: dynamic
  memFree: 34250637312
  memTotal: 50368409600
  ociRuntime:
    name: runc
    package: runc-1.0.0-70.rc92.module+el8.3.1+9857+68fb1526.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.2-dev'
  os: linux
  remoteSocket:
    path: /run/user/93975/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.module+el8.3.1+9857+68fb1526.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 0
  swapTotal: 0
  uptime: 28h 55m 43.95s (Approximately 1.17 days)
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/aressem/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.3.0-2.module+el8.3.1+9857+68fb1526.x86_64
      Version: |-
        fusermount3 version: 3.2.1
        fuse-overlayfs: version 1.3
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /home/aressem/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/93975/containers
  volumePath: /home/aressem/.local/share/containers/storage/volumes
version:
  APIVersion: "2"
  Built: 1612819146
  BuiltTime: Mon Feb  8 21:19:06 2021
  GitCommit: ""
  GoVersion: go1.14.7
  OsArch: linux/amd64
  Version: 2.2.1(paste your output here)

Package info (e.g. output of rpm -q podman or apt list podman):

$ rpm -q podman
podman-2.2.1-7.module+el8.3.1+9857+68fb1526.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

No, I have been using the podman that comes with RHEL 8. Yes, I have checked the troubleshooting guide.

Additional environment details (AWS, VirtualBox, physical, etc.):
Physical machine

[Luap99]

According to the code comment this is done for docker compat.

podman/libpod/container_internal_linux.go

Lines 1820 to 1827 in b64e20a

	if !hasNetNS {
	// 127.0.1.1 and host's hostname to match Docker
	osHostname, _ := os.Hostname()
	hosts += fmt.Sprintf("127.0.1.1 %s %s %s\n", osHostname, c.Hostname(), c.config.Name)
	}
	if netNone {
	hosts += fmt.Sprintf("127.0.1.1 %s %s\n", c.Hostname(), c.config.Name)
	}

However I was unable to reproduce this behaviour with docker 20.10, should we just drop this @mheon?

[mheon]

Hm. I see it on Docker 19.03 with docker run --net=host --hostname test1 -t -i --rm fedora cat /etc/hosts - did they change it since?

[Luap99]

[root@docker ~]#  docker run --net=host --hostname test1 -t -i --rm fedora cat /etc/hosts
Unable to find image 'fedora:latest' locally
latest: Pulling from library/fedora
b1495d80d526: Pull complete 
Digest: sha256:f534c437436eb44b7ac73646e642732fc055a75d84f900f07c3bbaa392007810
Status: Downloaded newer image for fedora:latest
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
[root@docker ~]# docker version
Client:
 Version:           20.10.6
 API version:       1.41
 Go version:        go1.16
 Git commit:        370c289
 Built:             Tue Apr 20 22:03:35 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.6
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16
  Git commit:       8728dd2
  Built:            Tue Apr 20 00:00:00 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.0~rc.1
  GitCommit:        
 runc:
  Version:          1.0.0-rc95
  GitCommit:        4c62ef789fd7a2963bf61ffbf421ce9646063648
 docker-init:
  Version:          0.19.0
  GitCommit:        

I installed moby-engine on fedora34.

[mheon]

I went down the rabbit hole of what's going on here, and tracked down #8054 as the original issue we added this for. It looks like the request was not Docker-compatibility related - we just had a request to ensure /etc/hosts had the hostname even for net=none. Should we retain the hostname, but change 127.0.1.1 to 127.0.0.1? That seems like a reasonable compromise.

The Docker 19.04 vs 20.10 difference is very curious, though. Would have thought this amounted to a breaking change, if they did change it.

[Luap99]

Interesting, they just use /etc/hosts and /etc/resolv.conf unchanged, see: https://github.com/moby/moby/blob/97303df92191878fabe4939da30ad056f7e941ea/daemon/container_operations_unix.go#L391-L399
moby/moby#41022

[mheon]

Damn, entirely correct - my test VM does have a 127.0.1.1 entry.

OK, that seems entirely reasonable, though IMO we should probably continue to do copies, instead of directly mounting in the host versions. Also, to ensure #8054 remains solved, I think we may want to add an entry for the current hostname if it does not exist.

[aressem]

I could understand if you add something when the --hostname test is specified. However when I use --net=host without specifying any --hostname, I would expect that I do not get an extra entry in /etc/hosts where the hostname (on the host where I execute podman) is mapped to a 127.0.x.x address.

[pgrabowski]

Is --no-hosts=true -v /etc/hosts:/etc/hosts a valid workaround?

[mheon]

Until this is fixed, yes, that sounds like a good option.

[aressem]

It's acceptable. I was not aware of the --no-hosts=true option.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[mheon]

Oops, looks like this got forgotten. I'll take it.

[anilwavde]

Can someone confirm if this issue is fixed? if yes, which version of podman it is incorporated in.

[mattymo]

Can we reopen this @mheon?

It looks like the issue is solved incorrectly:
Here's the /etc/hosts in the container:

# Kubernetes-managed hosts file.
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
fe00::0	ip6-mcastprefix
fe00::1	ip6-allnodes
fe00::2	ip6-allrouters
10.244.6.78	kubernetes-pod-name
127.0.1.1 kubernetes-pod-name kubernetes-pod-name container-name

My podman is the latest container build: 94886d4 and it contains your commit.

I have a python application incorrectly detecting its primary IP as 127.0.1.1 instead of 10.244.6.78. As far as I understand, this line shouldn't be present at all. I should also note that I have an interface in the container cni-podman0 that isn't present in podman.

[Luap99]

@mattymo Did you created a new container? I do not know if this works for already existing ones.

[mattymo]

@Luap99 yes new container. I would agree that existing containers would not be affected since container set up only happens once.

[Luap99]

@mattymo Can you share the commands you used to create the container.