Docs: Extra steps needed for DNS forwarding in Ubuntu 17.10, due to systemd-resolved

[honnibal]

I've been having a terrible time getting DNS forwarding to work on Ubuntu 17.10. I'm not an expert at this stuff, but I think the problems come down to the behaviour of systemd-resolved, and the way it takes ownership of the /etc/resolv.conf file.

Specifically, if I just follow the instructions on https://www.consul.io/docs/guides/forwarding.html , a dns lookup for consul.service.consul or any other consul subdomain fails. However, if I do dig -p 8600 consul.service.consul, it works fine. So consul is behaving correctly -- it's just that the port forwarding isn't working. This occurs with both dnsmasq and bind.

The problem seems to be that /etc/resolv.conf is symlinked to some file within systemd-resolved. After a lot of depressing reading, I guess this is a fairly infamous problem. If I set the /etc/resolv.conf to list 127.0.0.1 as the first nameserver, the port forwarding works okay.

Frustratingly, just disabling the systemd-resolved service doesn't put back the /etc/resolv.conf file --- it's still symlinked. So disabling the service leaves the system in a broken state. What seems to work is then replacing the symlink with an actual file, with the nameservers.

Here are the changes I've cobbled together my Packer image to get this working for me. I don't know whether this is an advisable solution. Hopefully the right thing to do can be added to the docs. Thanks!

Changes to consul.json provisioners:
------------------------------------

 {
    "type": "file",
    "source": "fix-dns",
    "destination": "/tmp/fix-dns",
    "pause_before": "10s"
  },{
    "type": "shell",
    "inline": "sudo /tmp/installers/install-bind9"
}
    

install-bind9
----------------

#!/usr/bin/env bash

apt-get install -y bind9
cp /tmp/fix-dn/bind-consul.conf /etc/bind/consul.conf
cp /tmp/fix-dns/bind-named.conf.options /etc/bind/named.conf.options
systemctl disable systemd-resolved
unlink /etc/resolv.conf
cp /tmp/fix-dns/resolv.conf /etc/resolv.conf
service bind9 enable
service bind9 restart

resolv.conf
--------------

nameserver 127.0.0.1
nameserver 8.8.8.8

bind-named.conf.options
---------------------------------

options {
        directory "/var/cache/bind";
        recursion yes;
        allow-query { localhost; };

        forwarders {
                8.8.8.8;
                8.8.4.4;
        };

        dnssec-enable no;
        dnssec-validation no;

        auth-nxdomain no;    # conform to RFC1035
        listen-on-v6 { any; };
};

include "/etc/bind/consul.conf";

bind-consul.conf
----------------------

zone "consul" IN {
  type forward;
  forward only;
  forwarders { 127.0.0.1 port 8600; };
};

[preetapan]

@honnibal I am closing this out since its not a bug or enhancement request. Feel free to submit a documentation update PR for the above.

[shantanugadgil]

@honnibal,
I have a few suggestions, in case you plan to do large scale stuff with your experiments:

Stick with an LTS version of Ubuntu, its less madness that way: currently 16.04, soon 18.04
Try using Ubuntu Server rather than the Desktop flavor.
Maybe just use CentOS 7 without NetworkManager.

You haven't mentioned what exactly you are trying to do, but there go the free suggestions. :) ;)

HTH,
Shantanu

[sandstrom]

@honnibal I've opened a related issue here: #4155

@shantanugadgil Agree, LTS is much better. Unfortunately I've run into the same issue under 18.04 😄