default iprange clashes with DigitalOcean Private Networking

[rade]

A user ran into

network 10.128.0.0/10 would overlap with route 10.128.0.0/16
Default -iprange 10.128.0.0/10 overlaps with existing route.
Please pick another range and set it on all hosts.

Further investigation by them revealed that

It looks to be set up for my DigitalOcean's eth1 interface, which is for the (optional) private IP of a droplet. I suspect anyone who has a DigitalOcean droplet with Private Networking enabled may have this route by default.

If confirmed then we may want to consider changing the default iprange to avoid this clash.

That unfortunately is a backward incompatible change. sigh

[errordeveloper]

Yes, it looks like the 10.128.x.x range is the default one for private network on DigitalOcean, as appears in Introducing Private Networking blog post.

[errordeveloper]

I have just created an instance in their London DC 1 and it came up on 10.131.152.68 with netmask 255.255.0.0.

[errordeveloper]

In the default New York DC 3 I get 10.132.201.238 with netmask 255.255.0.0.

[rade]

So it looks like 10.192.0.0/10 would work.

[rade]

So I've been pondering how to make such a change backward compatible. I reckon the best we can sensibly do (that is proportional to the issue), is to stick an advisory in the release notes telling users of the current default iprange that want to perform a rolling upgrade that they should specify the old range explicitly. They can roll out that change prior to the upgrade if they wish. And eventually, at an opportune time for a stop-the-world upgrade, they can remove the option again and thus switch to using the new default range.

@squaremo @bboreham thoughts?

[bboreham]

Agree this will work for users who are affected.
I guess we should be prepared to learn which environments clash with 10.192/10.

[bboreham]

As predicted, we learned that 10.192/10 clashes with GCE.

We could, in the initial consensus stage, collect conflicting ranges from all hosts, then pick a range for IPAM that doesn't conflict with anyone we have heard from. This would not help the "add more hosts slowly" case.

[rade]

We could, in the initial consensus stage, collect conflicting ranges from all hosts

We'd need something more than basic paxos for this, wouldn't we?

[rade]

collect conflicting ranges from all hosts, then pick a range for IPAM that doesn't conflict with anyone we have heard from.

What's the 'picking' algorithm? Say that no private IP ranges are used by any of the nodes. Which IP range would we pick for IPAM? Seems to me that we'd still need a default IP allocation range :) And we should still choose that based on its (un)likeliness of clashing with ranges used by popular platforms/systems, to give us a fighting chance of coping with the 'add more hosts' case.

[bboreham]

If we union the information, it is not necessary to have any consensus. We will have heard from a quorum of peers and we can choose a range based on that information.

The default, in the absence of conflicts, would be as shipped in 1.0, for backward compatibility.

[rade]

The default, in the absence of conflicts, would be as shipped in 1.0, for backward compatibility.

I don't think backward compatibility is a major concern here. As I noted above, we should still choose a range that is unlikely to clash, so that adding more hosts has a good chance of succeeding.

[bboreham]

Do we have any proposals on the table how to identify that range?

[rade]

Do we have any proposals on the table how to identify that range?

Trial and error.