No arm release for runc v1.0.3

[Payback159]

Environment:

• Cloud provider or hardware configuration:

raspberry pi 4

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):

printf "$(uname -srm)\n$(cat /etc/os-release)\n"
Linux 5.4.0-1048-raspi aarch64
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

• Version of Ansible (ansible --version):

ansible 2.10.15
  config file = /kubespray/ansible.cfg
  configured module search path = ['/kubespray/library']
  ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.6.9 (default, Jan 26 2021, 15:33:00) [GCC 8.4.0]

• Version of Python (python --version):

Python 3.6.9

Kubespray version (commit) (git rev-parse --short HEAD):

92f25bf2

Network plugin used:

calico

Full inventory with variables (ansible -i inventory/sample/inventory.ini all -m debug -a "var=hostvars[inventory_hostname]"):

Command used to invoke ansible:

ansible-playbook -i /inventory/inventory.yaml --private-key /root/.ssh/id_rsa -u ubuntu upgrade-cluster.yml --become

Output of ansible run:

fatal: [himbeere44 -> himbeere44]: FAILED! => {"attempts": 4, "changed": false, "dest": "/tmp/releases/runc", "elapsed": 0, "msg": "Request failed", "response": "HTTP Error 404: Not Found", "status_code": 404, "url": "https://github.com/opencontainers/runc/releases/download/v1.0.3/runc.arm64"}

Anything else do we need to know:

temporarily disabled nolog in download_file step to get the broken link output.

Looks like the hash value was added in PR #8274 but there is no arm release at https://github.com/opencontainers/runc/releases/tag/v1.0.3.

The following value was stored in the runc_checksum under arm64:

v1.0.3: edda26f24f9137f5e6981583dc0e782ea9e3ffee591d4c7a9f9565adc2ed9461

I can create a PR for it too, but wanted to check with you first if that's what you want. The hash value must come from somewhere.

[kjellmoens]

Same problem, no arm release at https://github.com/opencontainers/runc/releases/tag/v1.0.3

[ryota2425-murakami]

Is there any way to use it on arm64?

[Payback159]

At least not a production-ready variant at the moment. I just got a running cluster up and running on my Pis a few minutes ago.

But for that I had to add the hash for runc v1.1.0-rc-1 and containerd-1.6.0-beta.5, because only from these releases an arm64 version is offered upstream.

What I didn't quite understand yet is that the installation under Kubespray v2.17 worked without problems but I couldn't find any arm64 versions upstream runc and containerd for the used versions. You might have to check if something has changed in the download sources between v2.17 and v2.18.

[kjellmoens]

Hi,

I tested the #8391 fix but I still receive the same error. The error message, after disabling no_log: true is

TASK [container-engine/runc : download_file | Download item] ********************************************************************************************************************************************************************************
fatal: [node3 -> 192.168.2.13]: FAILED! => {"attempts": 4, "changed": false, "dest": "/tmp/releases/runc", "elapsed": 0, "msg": "Request failed", "response": "HTTP Error 404: Not Found", "status_code": 404, "url": "https://github.com/opencontainers/runc/releases/download/v1.0.3/runc.arm64"}
fatal: [node2 -> 192.168.2.12]: FAILED! => {"attempts": 4, "changed": false, "dest": "/tmp/releases/runc", "elapsed": 0, "msg": "Request failed", "response": "HTTP Error 404: Not Found", "status_code": 404, "url": "https://github.com/opencontainers/runc/releases/download/v1.0.3/runc.arm64"}
fatal: [node4 -> 192.168.2.14]: FAILED! => {"attempts": 4, "changed": false, "dest": "/tmp/releases/runc", "elapsed": 0, "msg": "Request failed", "response": "HTTP Error 404: Not Found", "status_code": 404, "url": "https://github.com/opencontainers/runc/releases/download/v1.0.3/runc.arm64"}
fatal: [node1 -> 192.168.2.11]: FAILED! => {"attempts": 4, "changed": false, "dest": "/tmp/releases/runc", "elapsed": 0, "msg": "Request failed", "response": "HTTP Error 404: Not Found", "status_code": 404, "url": "https://github.com/opencontainers/runc/releases/download/v1.0.3/runc.arm64"}
fatal: [node5 -> 192.168.2.15]: FAILED! => {"attempts": 4, "changed": false, "dest": "/tmp/releases/runc", "elapsed": 0, "msg": "Request failed", "response": "HTTP Error 404: Not Found", "status_code": 404, "url": "https://github.com/opencontainers/runc/releases/download/v1.0.3/runc.arm64"}

[Payback159]

Yes unfortunately the fix #8391 doesn't quite solve the original problem. As mentioned in the previous comment, I only got the cluster running after I entered runc v1.1.0-rc-1 in the hash values. After that the playbook fails because there is no containerd 1.5.8 for arm64 and so I added the hash for containerd-1.6.0-beta.5 locally.

After that the installation went without problems. Since both are release candidates or beta version, I would not recommend a productive use.

As far as I can see, however, there are no official arm64 binaries of runc and containerd prior to these releases. Therefore the open question from my side, where were the binaries in Kubespray v2.17 obtained from (Because I previously had a running cluster on arm64 running v2.17). I haven't had time to take a closer look for that yet.

[champtar]

@Payback159 we were using docker.com packages but they were slow to update and even skipping containerd version, so we switched to using release from containerd directly

[Payback159]

Thanks @champtar for the clarification. That is, as soon as runc v1.1.0 and containerd 1.6.0 were released, we could think about an arm64 installation. Presumably this will come with Kubespray v2.19 at the earliest or you use master. Am I right?

[champtar]

I don't use k8s with arm(64) but yes once containerd 1.6.0 is released I'm sure It'll land in kubespray pretty fast. If config doesn't change you can just add the hash/change the version in 2.18

[kjellmoens]

Yes unfortunately the fix #8391 doesn't quite solve the original problem. As mentioned in the previous comment, I only got the cluster running after I entered runc v1.1.0-rc-1 in the hash values. After that the playbook fails because there is no containerd 1.5.8 for arm64 and so I added the hash for containerd-1.6.0-beta.5 locally.

Hi, can you tell me which files needs to updated and with which values for containerd ? I got it working for runc but not for containerd

[Payback159]

Hi @kjellmoens, at the moment I can't get to my ansible-roles but in the end it should be enough if you

just add here:
https://github.com/kubernetes-sigs/kubespray/blob/master/roles/download/defaults/main.yml#L393

v1.1.0-rc.1: 6d5225976b51724ea0e693e34af74890f59d60e45cf76608e9562ab9347f1f31

in a new line, plus here
https://github.com/kubernetes-sigs/kubespray/blob/master/roles/download/defaults/main.yml#L467

a new line with:

1.6.0-beta.5: d4285d78aa156eb2f70830faf725450417bb9dde229525efc8c3e43131543853

After that you have to add somewhere in your inventory (for example I added it to the ./inventory/group_vars/all/containerd.yml file and created a new one ./inventory/group_vars/all/runc.yml) following parameters:

containerd_version: 1.6.0-beta.5
runc_version: v1.1.0-rc.1

Hope I could help.

[kjellmoens]

@Payback159 thank you ! It works

[umatare5]

Note: checksum for containerd 1.6.0-rc.1 arm64 is here.

1.6.0-rc.1: bc9f96ec51848dd8467210c94f0d7f3be4d73f0331e15d7c581224c218c9c42f