make Java InetAddress.getLocalHost() (similar to hostname -i) return the weave IP address

[bboreham]

At present, a container started in the way described in the weave README will have two IP addresses: one assigned by Docker and one assigned through weave. So hostname -I will return something like:

root@9b903541c5a7:/# hostname -I
172.17.0.72 10.0.0.61 

however, hostname -i returns the Docker-assigned IP:

root@9b903541c5a7:/# hostname -i
172.17.0.72

I would prefer hostname -i to return the weave address

[rade]

I would prefer hostname -i to return the weave address

Why?

[bboreham]

More detail (I'm posting this on behalf of someone who approached via email)

I'm using Consul for service registration and discovery. While registering services they default to the address provided by the host which is usually in the "172.x." range given by Docker. I can manually switch that to advertise the "10.0.x"-range by querying the interfaces directly with:

IP=`ip -4 -o addr show dev ethwe 2> /dev/null |awk '{split($4,a,"/") ;print a[1]}'`

Also I can do the same with Cassandra and with sed insert that IP to the cassandra.yaml so that they can talk with each other, also across nodes. Inside a single host having them use the "172.x" range is of course okay, but it won't fly across the Weave network.

I would much rather have both Consul registration and Cassandra report the IP of the Weave-interface directly, they both most probably use the address reported by "hostname -i". I got around that with pipework and our own DNS implementation with telling Ubuntu to use DNS before hosts-file, but that is quite kludgy and in this case wouldn't even work as Consul provides the DNS for all the hosts and the hosts register themselves instead of me registering them outside of the container where I can declare the IP for them.

Basically either with metric-information or by otherwise ordering the interfaces I'd like to have Weave return the ethwe IP with "hostname -i". Both IPs are reported with "hostname -I". I think changing the metrics in running containers is prohibited (at least I get an error when I try), so I'm out of ideas there.

[rade]

Would disabling the docker networking work? (--net=none)

[bboreham]

Interestingly, no. Even though that address doesn't work, it is still returned:

root@0c034eb763f3:/# hostname -i
172.17.0.73
root@0c034eb763f3:/# ping `hostname -i`
PING 172.17.0.73 (172.17.0.73): 48 data bytes
ping: sending packet: Network is unreachable

Looks like hostname -i returns whatever IP address hostname resolves to, which is here:

root@0c034eb763f3:/# cat /etc/hosts
172.17.0.73 0c034eb763f3
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
...

Changing the address listed in /etc/hosts has the desired effect on hostname -i

(See also moby/moby#1951, on the subject of editing /etc/hosts inside a container)

[rade]

On my ubuntu machine, having two entries for my hostname in /etc/hosts causes hostname -i to return both.

What do the likes of Consul and Cassandra actually do? Do they literally take the output of hostname -i? If so, what is the behaviour when, as I've seen, that returns multiple results?

[SirIle]

OP here. When I was using pipework (by jpetazzo) and a custom DNS server implementation, I had 'hostname -i' return the DNS server given address for the host by altering the order of lookup in /etc/nsswitch.conf. In that case the interface was created after the container was started and also added to the DNS server with the hostname so that DNS lookup returned the IP I gave to the container.

When the IP address is given to the container by Weave at start-up for the ethwe-interface and Consul provides the DNS, it seems that Consul registers the host with the IP address that the hosts-file provides for localhost. This also seems to be what 'hostname -i' returns. In Cassandra documentation it is mentioned that if the bind address is left empty, it uses InetAddress.getLocalHost() to resolve.As described above, I can override and parse the IP address from ethwe interface, but things get quite kludgy.

I try to make the applications in the container as agnostic about the environment as possible and in this case the container thinking it has the Weave provided IP address (instead of the Docker provided node local one) would make sense, especially if I connect services from multiple nodes together over Weave.

One solution could be that Weave (maybe with a switch) inserts the defined IP address to the hosts file at start-time so that it is defined before the docker-written one if that is possible.

I tried modifying the metrics for the interfaces so that I could see if ethwe would be preferred, but that was prohibited in a running container. Probably /etc/hosts would still provide the address, though.

[bboreham]

OK, I see the code calling getLocalHost(). I have renamed this issue to more accurately reflect what you want.

To @rade 's point, the Java implementation takes the first address returned after querying all configured name services.

[rade]

@bboreham you wrote

Interestingly [disabling docker networking doesn't work]. Even though that address doesn't work, it is still returned

but I cannot see how that could be the case - when docker networking is disabled then it doesn't even assign an IP. Here's what I'm getting...

root@xps:~# C=$(weave run 10.0.1.3/24 --net=none -t -i ubuntu)
root@xps:~# docker attach $C
root@4f6023ed153a:/# hostname -i
hostname: Name or service not known
root@4f6023ed153a:/# cat /etc/hosts 
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
127.0.0.1       localhost

This obviously doesn't solve the problem, but means that @SirIle's suggestion of inserting the weave IP at the start of /etc/hosts strikes me as a sensible solution. Are there any common circumstances under which that is the wrong thing to do? I'd rather avoid introducing a switch.

Also, what does Java's getLocalHost() return when the hostname resolves to multiple IP addresses? hostname -i happily returns both addresses, in order, but the Java function only has a single InetAddress result. One would hope it returns the first.

[bboreham]

Also, what does Java's getLocalHost() return when the hostname resolves to multiple IP addresses?

It returns the first one, by my reading of the Java source code.

Can't comment on why your experiment gave a different result to mine; maybe Docker has changed.

[rade]

Is there a way to let weave delay the startup of the container until it has created the interface?

See #47

[rade]

As of docker 1.4.1 you can add entries to /etc/hosts with --add-host=name:IP. Alas those entries get appended, which is of no use to resolving this issue.

[qxo]

weave run with " --add-host=name:IP --net=none -h name " will solve the problem!
f --net=none disable docker networking,so we need add reachable dns ip using --dns
weave should add these argument as default :)
eg:

ip=10.2.1.26;weave run  --with-dns $ip/24 --net=none --add-host=a1:$ip -ti -h a1  ubuntu

[rade]

--net=none disables docker networking, i.e. you are left with just weave networking. That may be fine for some users, but it's not something we'd want to have as the default behaviour.

[rade]

Also, if you are running with --net=none then the problem described in this issue doesn't manifest in the first place.

[rade]

/etc/hosts is writable:

$ weave launch && weave launch-dns && weave launch-proxy
47921135b7b4943de2a1f56ff44a2060206e4ee8c7fc136a8e237958b710529a
4f19482849dc1e3166506b7da1aef5f9aebe6615f6c944eb1fae72dedf3bc496
4bc93553a81228a9253e58e48d21aebc582e4587b10ddded1e865b0ff2a94df6
$ eval $(weave proxy-env)
$ docker run --name=foo -ti ubuntu
root@foo:/# hostname -i
172.17.0.63
root@foo:/# echo > /etc/hosts
root@foo:/# hostname -i
10.128.0.2

I also tried prepending an entry and using docker links at the same time, to see whether docker's /etc/hosts updating stomps on the file. It doesn't; it just updated the entry for the link and left my modifications alone. The only gotcha I can see is that because the file is mounted, there is no atomic way to update it.

So, anyway, strawman proposal: make weave run/attach (and thereby also the proxy) prepend an entry to /etc/hosts with the weave IP.

We should probably also deal with dynamic add/remove via weave attach/detach.

[rade]

make weave run/attach (and thereby also the proxy) prepend an entry to /etc/hosts with the weave IP.

The trouble is that we have no easy way to do that from the script since we somehow need to reach into the container's filesystem. docker exec is no good here since it can only execute commands that are present inside the container. The proxy could do it easily, in weavewait (since that is running inside the container). That doesn't help with dynamic attach/detach, but perhaps is still a worthwhile improvement, despite the inconsistency it creates with weave run.

[errordeveloper]

Running Mesos in Docker with Weave & DNS is also problematic due to this, as it looks up it's hostname and registers own cluster membership with the IP address it finds, which turns out to be Docker IP, of course... Just for the record.