host.docker.internal not resolvable depending on nameserver order

[sandro-h]

Description

host.docker.internal is not resolvable depending on how systemd-resolved orders the nameservers. This only happens when --network-address is used and happens "randomly" depending on how it decides to order them.

Version

colima version 0.7.5
git commit: 1588c066b9ab9dae8205ef265929c7eb43dca473

runtime: docker
arch: x86_64
client: v24.0.1
server: v27.1.1
limactl version 0.23.2
qemu-img version 9.1.0
Copyright (c) 2003-2024 Fabrice Bellard and the QEMU Project developers

Operating System

• macOS Intel <= 13 (Ventura)
• macOS Intel >= 14 (Sonoma)
• Apple Silicon <= 13 (Ventura)
• Apple Silicon >= 14 (Sonoma)
• Linux

Output of colima status

INFO[0000] colima [profile=minimal_test] is running using QEMU 
INFO[0000] arch: x86_64                                 
INFO[0000] runtime: docker                              
INFO[0000] mountType: sshfs                             
INFO[0000] address: 192.168.106.4                       
INFO[0000] socket: unix:///Users/go/.colima/minimal_test/docker.sock 

Reproduction Steps

1. Start colima with --network-address on a MacOS Ventura host:

   colima start minimal_test --network-address
   

2. Try to reach host.docker.internal in a docker container:

   docker run -ti --rm busybox ping host.docker.internal
   

Expected behaviour

The ping succeeds:

PING host.docker.internal (192.168.5.2): 56 data bytes
64 bytes from 192.168.5.2: seq=0 ttl=63 time=0.573 ms
64 bytes from 192.168.5.2: seq=1 ttl=63 time=0.726 ms

Instead, the ping fails with:

ping: bad address 'host.docker.internal'

Additional context

We recently upgraded from Colima 0.4.6 to 0.75. Before the upgrade this wasn't a problem. As far as I understand, the older version wasn't using systemd yet.

As said, it doesn't happen consistently, but depends on how systemd-resolved decides to order the nameservers on start up. I can reliably reproduce it (and fix it) by restarting this service in a running colima instance until the order changes:

1. Restart systemd-resolved:

   colima -p minimal_test ssh sudo systemctl restart systemd-resolved
   

2. Check the order of the nameservers:

   colima -p minimal_test ssh grep 'nameserver' /run/systemd/resolve/resolv.conf
   

   In the good case, the Host IP (192.168.5.2) is first, as expected:

   nameserver 192.168.5.2
   nameserver 192.168.106.1
   nameserver fe80::50:56ff:fe28:9064%3
   

   In the bad case, the Host IP comes after others, in particular after the bridge100 / col0 IP:

   nameserver 192.168.106.1
   nameserver fe80::50:56ff:fe28:9064%3
   nameserver 192.168.5.2
   

Output of colima -p minimal_test ssh resolvectl:

Global
         Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.5.2
       DNS Servers: 192.168.5.2
        DNS Domain: [redacted company domains]

Link 3 (col0)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.106.1
       DNS Servers: 192.168.106.1 fe80::50:56ff:fe28:9064
        DNS Domain: [redacted company domains]

Link 4 (docker0)
    Current Scopes: none
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported