Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cluster upgrade FAILED #3849

Closed
kisahm opened this issue Dec 7, 2018 · 5 comments
Closed

cluster upgrade FAILED #3849

kisahm opened this issue Dec 7, 2018 · 5 comments

Comments

@kisahm
Copy link

kisahm commented Dec 7, 2018

i tried to upgrade my kubernetes v1.9.2 cluster (deployed with kubespray) to 1.12.3 but the upgrade fails because some certs are missing:

TASK [kubernetes/master : Copy old certs to the kubeadm expected path] ********************************************************************************************************************************************
Friday 07 December 2018  10:01:35 +0000 (0:00:00.470)       1:49:21.355 ******* 
changed: [k8s-0001] => (item={u'dest': u'apiserver.crt', u'src': u'apiserver.pem'})
changed: [k8s-0001] => (item={u'dest': u'apiserver.key', u'src': u'apiserver-key.pem'})
changed: [k8s-0001] => (item={u'dest': u'ca.crt', u'src': u'ca.pem'})
changed: [k8s-0001] => (item={u'dest': u'ca.key', u'src': u'ca-key.pem'})
failed: [k8s-0001] (item={u'dest': u'front-proxy-ca.crt', u'src': u'front-proxy-ca.pem'}) => {"changed": false, "item": {"dest": "front-proxy-ca.crt", "src": "front-proxy-ca.pem"}, "msg": "Source /etc/kubernetes/ssl/front-proxy-ca.pem not found"}
failed: [k8s-0001] (item={u'dest': u'front-proxy-ca.key', u'src': u'front-proxy-ca-key.pem'}) => {"changed": false, "item": {"dest": "front-proxy-ca.key", "src": "front-proxy-ca-key.pem"}, "msg": "Source /etc/kubernetes/ssl/front-proxy-ca-key.pem not found"}
changed: [k8s-0001] => (item={u'dest': u'front-proxy-client.crt', u'src': u'front-proxy-client.pem'})
changed: [k8s-0001] => (item={u'dest': u'front-proxy-client.key', u'src': u'front-proxy-client-key.pem'})
failed: [k8s-0001] (item={u'dest': u'sa.pub', u'src': u'service-account-key.pem'}) => {"changed": false, "item": {"dest": "sa.pub", "src": "service-account-key.pem"}, "msg": "Source /etc/kubernetes/ssl/service-account-key.pem not found"}
failed: [k8s-0001] (item={u'dest': u'sa.key', u'src': u'service-account-key.pem'}) => {"changed": false, "item": {"dest": "sa.key", "src": "service-account-key.pem"}, "msg": "Source /etc/kubernetes/ssl/service-account-key.pem not found"}
changed: [k8s-0001] => (item={u'dest': u'apiserver-kubelet-client.crt', u'src': u'node-k8s-0001.pem'})
changed: [k8s-0001] => (item={u'dest': u'apiserver-kubelet-client.key', u'src': u'node-k8s-0001-key.pem'})

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG

Environment:

  • openstack

Linux 3.10.0-693.21.1.el7.x86_64 x86_64
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

  • Version of Ansible (ansible --version):
    ansible 2.7.4
    config file = /etc/ansible/ansible.cfg
    configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
    ansible python module location = /usr/lib/python2.7/dist-packages/ansible
    executable location = /usr/bin/ansible
    python version = 2.7.12 (default, Nov 12 2018, 14:36:49) [GCC 5.4.0 20160609]

Kubespray version (commit) (git rev-parse --short HEAD): 225f765

Network plugin used: weave

Copy of your inventory file:

[all]
k8s-0001 	 ansible_host=192.168.0.113 ip=192.168.0.113
k8s-0002 	 ansible_host=192.168.0.110 ip=192.168.0.110
k8s-0003 	 ansible_host=192.168.0.103 ip=192.168.0.103
k8s-0005         ansible_host=192.168.0.230 ip=192.168.0.230

[kube-master]
k8s-0001 	 
k8s-0002 	 
k8s-0003

[kube-node]
k8s-0001 	 
k8s-0002 	 
k8s-0003 	 
k8s-0005

[etcd]
k8s-0001 	 
k8s-0002 	 
k8s-0003 	 

[k8s-cluster:children]
kube-node 	 
kube-master 	 

[calico-rr]

[vault]
k8s-0001 	 
k8s-0002 	 
k8s-0003

Command used to invoke ansible: ansible-playbook upgrade-cluster.yml -b -i inventory/mycluster/hosts.ini -e kube_version=v1.12.3
@sdodsley
Copy link

Similar issue upgrading from 1.10.4. The original cluster was built using kubespray from the incubator github. The upgrade is being performed with the latest build from the sigs github.

TASK [kubernetes/master : Copy old certs to the kubeadm expected path] ********************************************************************************************************************************************
Monday 17 December 2018  06:18:38 -0800 (0:00:00.808)       0:47:54.694 *******
changed: [sn1-c07-caas-01] => (item={u'dest': u'apiserver.crt', u'src': u'apiserver.pem'})
changed: [sn1-c07-caas-01] => (item={u'dest': u'apiserver.key', u'src': u'apiserver-key.pem'})
changed: [sn1-c07-caas-01] => (item={u'dest': u'ca.crt', u'src': u'ca.pem'})
changed: [sn1-c07-caas-01] => (item={u'dest': u'ca.key', u'src': u'ca-key.pem'})
changed: [sn1-c07-caas-01] => (item={u'dest': u'front-proxy-ca.crt', u'src': u'front-proxy-ca.pem'})
changed: [sn1-c07-caas-01] => (item={u'dest': u'front-proxy-ca.key', u'src': u'front-proxy-ca-key.pem'})
changed: [sn1-c07-caas-01] => (item={u'dest': u'front-proxy-client.crt', u'src': u'front-proxy-client.pem'})
changed: [sn1-c07-caas-01] => (item={u'dest': u'front-proxy-client.key', u'src': u'front-proxy-client-key.pem'})
changed: [sn1-c07-caas-01] => (item={u'dest': u'sa.pub', u'src': u'service-account-key.pem'})
changed: [sn1-c07-caas-01] => (item={u'dest': u'sa.key', u'src': u'service-account-key.pem'})
failed: [sn1-c07-caas-01] (item={u'dest': u'apiserver-kubelet-client.crt', u'src': u'node-sn1-c07-caas-01.pem'}) => {"changed": false, "item": {"dest": "apiserver-kubelet-client.crt", "src": "node-sn1-c07-caas-01.pem"}, "msg": "Source /etc/kubernetes/ssl/node-sn1-c07-caas-01.pem not found"}
failed: [sn1-c07-caas-01] (item={u'dest': u'apiserver-kubelet-client.key', u'src': u'node-sn1-c07-caas-01-key.pem'}) => {"changed": false, "item": {"dest": "apiserver-kubelet-client.key", "src": "node-sn1-c07-caas-01-key.pem"}, "msg": "Source /etc/kubernetes/ssl/node-sn1-c07-caas-01-key.pem not found"}

@kisahm
Copy link
Author

kisahm commented Dec 18, 2018

does somebody know about a workaroud or fix for this issue?

@Atoms
Copy link
Member

Atoms commented Jan 4, 2019

you should use released versions of kubespray to upgrade (git checkout v2.x.x) and not to skip versions, so if your cluster was installed let's say with 2.5.0 kubespray release, to get latest release you need to execute upgrade-cluster.yml from 2.6.0 then from 2.7.0, and then from 2.8.0 or 2.8.1, we do not support skipping releases

@Atoms Atoms closed this as completed Jan 4, 2019
@caruccio
Copy link
Contributor

caruccio commented May 18, 2020 •
edited
Loading

There are any way to detect the current installer version of the cluster so we can alert users not to upgrade skipping versions? Maybe a configmap or file at the first master?

@giafar
Copy link

giafar commented Oct 9, 2020

Hi all,
it it possibile to upgrade from a tag to a release and then upgrare just release ?
I mean:
tag v2.10.4 --> release 2.11 --> release 2.12 --> release 2.13 --> release 2.14 --> tag v2.14.1

@olevitt olevitt mentioned this issue Jan 17, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@Atoms @caruccio @giafar @sdodsley @kisahm