Consul 1.5.3 and 1.6.0 agents defaulting on protocol v2

[BitProcessor]

Overview of the Issue

According to the output of consul -v on a three-node cluster running all identical versions of the consul binary, I expect the agents to talk to each other in protocol v3. However, the active protocol is v2.

For 1.5.3:

# consul -v
Consul v1.5.3
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

For 1.6.0:

# ./consul -v
Consul v1.6.0
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)
# ./consul members
Node                               Address          Status  Type    Build  Protocol  DC         Segment
consul-01.example.com  10.0.2.118:8301  alive   server  1.6.0  2         eu-west-1  <all>
consul-02.example.com  10.0.3.235:8301  alive   server  1.6.0  2         eu-west-1  <all>
consul-03.example.com  10.0.4.78:8301   alive   server  1.6.0  2         eu-west-1  <all>

Also, according to https://www.consul.io/docs/agent/options.html :

-protocol - The Consul protocol version to use. This defaults to the latest version. This should be set only when upgrading. You can view the protocol versions supported by Consul by running consul -v.

Reproduction Steps

Setup a three cluster-nodes consul environment with the following startup command:

/opt/consul/consul agent \
        -node=consul-<id>.example.com \
        -config-dir=/srv/consul/config

and the following config.json:

{
    "server": true,
    "bootstrap_expect": 3,
    "client_addr": "0.0.0.0",
    "datacenter": "eu-west-1",
    "data_dir": "/srv/consul/data",
    "domain": "consul",
    "enable_syslog": true,
    "encrypt": "vE43EkDQtUq0D/9wUDT4bQ==",
    "retry_join": [
        "consul-01.example.com",
        "consul-02.example.com",
        "consul-03.example.com"
    ],
    "ui": true
}

And run (same issue for 1.5.3 and 1.6.0):

# ./consul -v
# ./consul members

Consul info for both Client and Server

Client info

Only 3 agents in agent mode at the moment.

Server info

agent:
	check_monitors = 0
	check_ttls = 0
	checks = 0
	services = 0
build:
	prerelease = 
	revision = a42ded47
	version = 1.5.3
consul:
	acl = disabled
	bootstrap = false
	known_datacenters = 1
	leader = false
	leader_addr = 10.0.2.118:8300
	server = true
raft:
	applied_index = 11222
	commit_index = 11222
	fsm_pending = 0
	last_contact = 38.243693ms
	last_log_index = 11222
	last_log_term = 12
	last_snapshot_index = 0
	last_snapshot_term = 0
	latest_configuration = [{Suffrage:Voter ID:1c7b2b9e-f9a6-3303-643c-884c639887cb Address:10.0.4.78:8300} {Suffrage:Voter ID:4600fc8d-782f-40c3-0d51-7dced4e78957 Address:10.0.3.235:8300} {Suffrage:Voter ID:165d1f53-1f1b-b6bd-7225-72f4019ef5b7 Address:10.0.2.118:8300}]
	latest_configuration_index = 1
	num_peers = 2
	protocol_version = 3
	protocol_version_max = 3
	protocol_version_min = 0
	snapshot_version_max = 1
	snapshot_version_min = 0
	state = Follower
	term = 12
runtime:
	arch = amd64
	cpu_count = 2
	goroutines = 84
	max_procs = 2
	os = linux
	version = go1.12.1
serf_lan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 7
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 30
	members = 3
	query_queue = 0
	query_time = 1
serf_wan:
	coordinate_resets = 0
	encrypted = true
	event_queue = 0
	event_time = 1
	failed = 0
	health_score = 0
	intent_queue = 0
	left = 0
	member_time = 11
	members = 3
	query_queue = 0
	query_time = 1

Operating system and Environment details

root@consul-03:~# cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"

[jameshartig]

I think this is just a display issue. See #1531

[BitProcessor]

I think this is just a display issue. See #1531

Your reference is from version 0.6.0 ; this is 1.6.0 ; all agents are on 1.6.0 so I would guess they'd be speaking v3 witch each other ?

[schristoff]

Closed as answered in #3887

[BitProcessor]

It’s not answered at all. The answer in #3887 says: it’s the default version that is shown. That doesn’t correspond at all with the docs: see my report. If the docs are correct, it should default to v3 and thus be reporting so (if the answer in #3887 is somewhat relevant). So either the docs are wrong, or the output is wrong.

See:

-protocol - The Consul protocol version to use. This defaults to the latest version. This should be set only when upgrading. You can view the protocol versions supported by Consul by running consul -v.

[jameshartig]

Version 1.6.0 says:

Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

So when you call consul members it will show 2 since that's the default, but version 3 will be spoken automatically to compatible agents. I think your misunderstanding is that consul members is not necessarily showing the protocol that the agents are speaking, but instead showing the "default" procotol which is 2. Just because 2 is the default does not mean that they're currently speaking that version. If both agents can speak 3 then they will upgrade from 2 and speak the latest version.

[BitProcessor]

The output: ‘Protocol 2 spoken by default’

The docs say: protocol - The Consul protocol version to use. This defaults to the latest version. This should be set only when upgrading. You can view the protocol versions supported by Consul by running consul -v.

Clearly anyone that does the effort of actually reading what I write can see that this contradicts itself?

[schristoff]

Hey @BitProcessor, apologies about this. Now after properly reading I totally see where the confusion lies, and my eyes have been deceiving me this entire time.
This is a conflicting issue between our docs and the CLI output. I'm going to go ahead and update the docs properly to reflect the information outputted in the CLI so that we don't contribute to further confusion.

[schristoff]

Reopening until PR is merged