Old DNS entries are not cleaned up (using podman-network-create and podman-run)

[eriksjolund]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

The DNS entry is not removed when the container is removed.

Steps to reproduce the issue:

[erik@laptop ~]$ ls bind-utils/
Dockerfile
[erik@laptop ~]$ cat bind-utils/Dockerfile 
FROM docker.io/library/fedora
RUN dnf install -y bind-utils
[erik@laptop ~]$ podman build -t bind-utils bind-utils

[erik@laptop ~]$ podman network create mynet
/home/erik/.config/cni/net.d/mynet.conflist
[erik@laptop ~]$ podman run --rm --network mynet --name foobar localhost/bind-utils host foobar
foobar has address 10.88.6.9
[erik@laptop ~]$ podman run --rm --network mynet --name foobar localhost/bind-utils host foobar
foobar has address 10.88.6.10
foobar has address 10.88.6.9
[erik@laptop ~]$ podman run --rm --network mynet --name foobar localhost/bind-utils host foobar
foobar has address 10.88.6.11
foobar has address 10.88.6.9
foobar has address 10.88.6.10
[erik@laptop ~]$ podman run --rm --network mynet --name foobar localhost/bind-utils host foobar
foobar has address 10.88.6.12
foobar has address 10.88.6.9
foobar has address 10.88.6.10
foobar has address 10.88.6.11
[erik@laptop ~]$ 

Describe the results you received:

[erik@laptop ~]$ podman run --rm --network mynet --name foobar localhost/bind-utils host foobar
foobar has address 10.88.6.12
foobar has address 10.88.6.9
foobar has address 10.88.6.10
foobar has address 10.88.6.11
[erik@laptop ~]$ 

Describe the results you expected:

[erik@laptop ~]$ podman run --rm --network mynet --name foobar localhost/bind-utils host foobar
foobar has address 10.88.6.12
[erik@laptop ~]$ 

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 2.1.0

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.21-1.el8.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 3460cd1ad859a79bd27df1714f39c76926ac1b39-dirty'
  cpus: 16
  distribution:
    distribution: '"centos"'
    version: "8"
  eventLogger: journald
  hostname: laptop.example.com (Manual edit to mask the information)
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1008
      size: 1
    - container_id: 1
      host_id: 427680
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1008
      size: 1
    - container_id: 1
      host_id: 427680
      size: 65536
  kernel: 4.18.0-193.19.1.el8_2.x86_64
  linkmode: dynamic
  memFree: 65648582656
  memTotal: 67206131712
  ociRuntime:
    name: runc
    package: runc-1.0.0-145.rc91.git24a3cf8.el8.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.2-dev'
  os: linux
  remoteSocket:
    path: /run/user/1008/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-2.el8.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
  swapFree: 33810280448
  swapTotal: 33810280448
  uptime: 2h 47m 44.69s (Approximately 0.08 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/erik/.config/containers/storage.conf
  containerStore:
    number: 39
    paused: 0
    running: 1
    stopped: 38
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-1.el8.x86_64
      Version: |-
        fusermount3 version: 3.2.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /home/erik/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 136
  runRoot: /run/user/1008/containers
  volumePath: /home/erik/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1600819270
  BuiltTime: Wed Sep 23 02:01:10 2020
  GitCommit: ""
  GoVersion: go1.13.15
  OsArch: linux/amd64
  Version: 2.1.0

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.1.0-1.el8.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

I've checked the Podman Troubleshooting Guide.
(The latest release 2.1.1 is not yet available in CentOS from Kubic)

Additional environment details (AWS, VirtualBox, physical, etc.):

[Luap99]

I cannot reproduce this on fedora 32. You are using the new rootless networking, right?

[eriksjolund]

Yes

[Luap99]

OK, I can reproduce. You need to have a long running container attached to this network in order to keep the rootless-cni-infra container alive.
like this: podman run -d --network mynet alpine sleep infinity

[AkihiroSuda]

Fix: #7841

[in-fke]

Is this problem recurring in some special context? I am using podman-compose (a development version of 1.0.4) and netavark.

[devops@rocky-vm soo-docker]$ podman exec a getent hosts b
10.89.2.30      b.dns.podman
10.89.2.31      b.dns.podman
10.89.2.39      b.dns.podman
10.89.2.40      b.dns.podman
10.89.2.48      b.dns.podman
[devops@rocky-vm soo-docker]$ podman exec b getent hosts b
10.89.2.48      b b 

[devops@rocky-vm soo-docker]$ podman-compose -v
podman-compose version: 1.0.4
['podman', '--version', '']
using podman version: 4.2.0
podman-compose version 1.0.4
podman --version
podman version 4.2.0
exit code: 0

[devops@rocky-vm soo-docker]$ cat /etc/containers/containers.conf
[network]

# Explicitly use netavark. See https://github.com/containers/podman-compose/issues/455
network_backend = "netavark"

containers/podman-compose#455

[in-fke]

As a follow-up to the above, just noticed that "podman-compose down" does not remove network.
see containers/podman-compose#490
Unfortunately, I am unable to reproduce the problem consistently.