Fail to update data in etcd through calicoctl (configuring route reflector) installing the cluster

[marquetemb]

** Description **

Installing the cluster with calico, at the time the route reflector is being configured, an error is shown:

TASK [network_plugin/calico/rr : Calico-rr | Fetch current node object] **********************************************************************************************************************************************************************
Thursday 06 August 2020  06:33:04 +0000 (0:00:00.350)       0:17:30.352 ******* 
changed: [kube-master-1] => {"attempts": 1, "changed": true, "cmd": ["/usr/local/bin/calicoctl.sh", "get", "node", "kube-master-1", "-ojson"], "delta": "0:00:00.061128", "end": "2020-08-06 06:33:04.740774", "rc": 0, "start": "2020-08-06 06:33:04.679646", "stderr": "", "stderr_lines": [], "stdout": "{\n  \"kind\": \"Node\",\n  \"apiVersion\": \"projectcalico.org/v3\",\n  \"metadata\": {\n    \"name\": \"kube-master-1\",\n    \"uid\": \"26fd85da-27b0-42ee-9341-19fe3922017b\",\n    \"resourceVersion\": \"744\",\n    \"creationTimestamp\": \"2020-08-06T06:31:38Z\",\n    \"labels\": {\n      \"beta.kubernetes.io/arch\": \"amd64\",\n      \"beta.kubernetes.io/os\": \"linux\",\n      \"kubernetes.io/arch\": \"amd64\",\n      \"kubernetes.io/hostname\": \"kube-master-1\",\n      \"kubernetes.io/os\": \"linux\",\n      \"node-role.kubernetes.io/master\": \"\"\n    },\n    \"annotations\": {\n      \"projectcalico.org/kube-labels\": \"{\\\"beta.kubernetes.io/arch\\\":\\\"amd64\\\",\\\"beta.kubernetes.io/os\\\":\\\"linux\\\",\\\"kubernetes.io/arch\\\":\\\"amd64\\\",\\\"kubernetes.io/hostname\\\":\\\"kube-master-1\\\",\\\"kubernetes.io/os\\\":\\\"linux\\\",\\\"node-role.kubernetes.io/master\\\":\\\"\\\"}\"\n    }\n  },\n  \"spec\": {\n    \"bgp\": {\n      \"ipv4Address\": \"192.168.121.225/32\",\n      \"ipv4IPIPTunnelAddr\": \"172.16.172.128\"\n    }\n  }\n}", "stdout_lines": ["{", "  \"kind\": \"Node\",", "  \"apiVersion\": \"projectcalico.org/v3\",", "  \"metadata\": {", "    \"name\": \"kube-master-1\",", "    \"uid\": \"26fd85da-27b0-42ee-9341-19fe3922017b\",", "    \"resourceVersion\": \"744\",", "    \"creationTimestamp\": \"2020-08-06T06:31:38Z\",", "    \"labels\": {", "      \"beta.kubernetes.io/arch\": \"amd64\",", "      \"beta.kubernetes.io/os\": \"linux\",", "      \"kubernetes.io/arch\": \"amd64\",", "      \"kubernetes.io/hostname\": \"kube-master-1\",", "      \"kubernetes.io/os\": \"linux\",", "      \"node-role.kubernetes.io/master\": \"\"", "    },", "    \"annotations\": {", "      \"projectcalico.org/kube-labels\": \"{\\\"beta.kubernetes.io/arch\\\":\\\"amd64\\\",\\\"beta.kubernetes.io/os\\\":\\\"linux\\\",\\\"kubernetes.io/arch\\\":\\\"amd64\\\",\\\"kubernetes.io/hostname\\\":\\\"kube-master-1\\\",\\\"kubernetes.io/os\\\":\\\"linux\\\",\\\"node-role.kubernetes.io/master\\\":\\\"\\\"}\"", "    }", "  },", "  \"spec\": {", "    \"bgp\": {", "      \"ipv4Address\": \"192.168.121.225/32\",", "      \"ipv4IPIPTunnelAddr\": \"172.16.172.128\"", "    }", "  }", "}"]}
TASK [network_plugin/calico/rr : Calico-rr | Set route reflector cluster ID] *****************************************************************************************************************************************************************
Thursday 06 August 2020  06:33:04 +0000 (0:00:00.206)       0:17:30.559 ******* 
ok: [kube-master-1] => {"ansible_facts": {"calico_rr_node_patched": {"apiVersion": "projectcalico.org/v3", "kind": "Node", "metadata": {"annotations": {"projectcalico.org/kube-labels": "{\"beta.kubernetes.io/arch\":\"amd64\",\"beta.kubernetes.io/os\":\"linux\",\"kubernetes.io/arch\":\"amd64\",\"kubernetes.io/hostname\":\"kube-master-1\",\"kubernetes.io/os\":\"linux\",\"node-role.kubernetes.io/master\":\"\"}"}, "creationTimestamp": "2020-08-06T06:31:38Z", "labels": {"beta.kubernetes.io/arch": "amd64", "beta.kubernetes.io/os": "linux", "kubernetes.io/arch": "amd64", "kubernetes.io/hostname": "kube-master-1", "kubernetes.io/os": "linux", "node-role.kubernetes.io/master": ""}, "name": "kube-master-1", "resourceVersion": "744", "uid": "26fd85da-27b0-42ee-9341-19fe3922017b"}, "spec": {"bgp": {"ipv4Address": "192.168.121.225/32", "ipv4IPIPTunnelAddr": "172.16.172.128", "routeReflectorClusterID": "1.0.0.1"}}}}, "changed": false}
TASK [network_plugin/calico/rr : Calico-rr | Configure route reflector] **********************************************************************************************************************************************************************
Thursday 06 August 2020  06:33:04 +0000 (0:00:00.049)       0:17:30.608 ******* 
fatal: [kube-master-1]: FAILED! => {"changed": true, "cmd": "/usr/local/bin/calicoctl.sh replace -f-", "delta": "0:00:00.033675", "end": "2020-08-06 06:33:05.013082", "msg": "non-zero return code", "rc": 1, "start": "2020-08-06 06:33:04.979407", "stderr": "Hit error: update conflict: Node(kube-master-1)", "stderr_lines": ["Hit error: update conflict: Node(kube-master-1)"], "stdout": "Partial success: replaced the first 1 out of 1 'Node' resources:", "stdout_lines": ["Partial success: replaced the first 1 out of 1 'Node' resources:"]}

This is due a race condition between getting the calico-node data and the replace of it through calicoctl. The revisionVersion of the data from etcd is outdated when the replace is done, as the data of etcd have been modified and the revisionVersion were increased.

Environment:

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):

Linux 3.10.0-862.9.1.el7.x86_64 x86_64
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

• Version of Ansible (ansible --version):

ansible 2.9.6
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 3.7.7 (default, May 30 2020, 09:56:15) [GCC 8.3.0]

• Version of Python (python --version):
  Python 3.7.7

Kubespray version (commit) (git rev-parse --short HEAD):
01dbc909 -> v2.13.0
Network plugin used:
Calico

Full inventory with variables (ansible -i inventory/sample/inventory.ini all -m debug -a "var=hostvars[inventory_hostname]"):

The hosts of the calico-rr group are the same of the kube-master:

......
    "calico-rr": {
        "hosts": [
            "kube-master-1"
        ]
    },
    "kube-master": {
        "hosts": [
            "kube-master-1"
        ]
    },
.......

[unai-ttxu]

Hi!

Could you review this issue please?

This issue causes some installations to fail when defining calico-rr group and force us to retry the installation.

Thanks.