apt.kubernetes.io should keep old versions

[dawidmalina]

It would be very desired to have all or at least last few releases would be available in the repository.

[Starefossen]

Yes, this hit me today as well....

[jsi-rl]

Is there a apt repo for the archives ? I need to deploy
kubeadm 1.5.0-alpha.2-421-a6bea3d79b8bba-00
kubectl 1.4.4-00
kubelet 1.4.4-01

[ixdy]

cc @luxas @errordeveloper @mikedanese

[dawidmalina]

There is a new release 1.5.2 should we expect that 1.5.1 packages will be deleted?

[marccarre]

I guess the motivation may be to keep most folks on the latest version, which simplifies development, testing, and many other things, but this then means that:

• one cannot rebuild a server configured with an older version, and
• one cannot add new servers to an existing cluster,

without breaking their setup, unless all versions of Kubernetes are backward compatible -- which is probably going to be a bigger pain in the long run.

Ultimately, not being able to set a fixed version therefore leads to configuration drift and snowflake servers, which is quite undesirable for any serious production usage.

Thoughts?

[errordeveloper]

I think we might want to create virtual repos within our repo, e.g.

deb http://apt.kubernetes.io/ kubernetes-xenial-archive main # all packages we ever published
deb http://apt.kubernetes.io/ kubernetes-xenial-latest main # whatever is the latest
deb http://apt.kubernetes.io/ kubernetes-xenial-1.5 main # always latest 1.5.x
deb http://apt.kubernetes.io/ kubernetes-xenial-1.4 main # always latest 1.4.x

[Starefossen]

Yes, please! That would simplify things at our end 😄👍🏼

…

On 23 Jan 2017, at 12:36, Ilya Dmitrichenko ***@***.***> wrote: I think we might want to create virtual repos within our repo, e.g. deb http://apt.kubernetes.io/ kubernetes-xenial-archive main # all packages we ever published deb http://apt.kubernetes.io/ kubernetes-xenial-latest main # whatever is the latest deb http://apt.kubernetes.io/ kubernetes-xenial-1.5 main # always latest 1.5.x deb http://apt.kubernetes.io/ kubernetes-xenial-1.4 main # always latest 1.4.x — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[dawidmalina]

Can you guys tell us when this new and desired feature can be implemented?

[geekofalltrades]

Is the expectation here that users are going to build the debs themselves and host them in their own apt repository?

In my particular use-case, as a security requirement, we isolate certain groups of pods on their own Kubernetes nodes. We have a process dynamically creating and destroying these nodes as needed. If the version of kubernetes in the apt repo changes and the old one goes away, my cluster will break, because my new dynamic worker nodes will start being created with the wrong version of kubelet on them!

[jbeda]

With the switch to rbac on by default for kubeadm for 1.6, we need to allow users to be pinned to an old version.

I'm happy to help make this happen. What needs to be done? @luxas @mikedanese @ixdy @errordeveloper

[mikedanese]

How many point and major releases do we want to keep around? Do we want to keep around multiple point releases of a major release? What if a deb has known security vulnerabilities?

[errordeveloper]

How many point and major releases do we want to keep around? Do we want to keep around multiple point releases of a major release?

I'd say keep everything, but it doesn't have to be all in the same package stream, e.g. we can have an archive channel.

What if a deb has known security vulnerabilities?

Perhaps archive channel would be a good answer to this?

[apsinha]

Maybe two archives one with all the stable major releases (e.g., 1.4.8 or whatever the best and latest point release for 1.4 was) and one with everything?

[jbeda]

Sorry for not responding earlier. This got lost in the noise.

I'm thinking at least 2 major versions. Perhaps 2-3 minor versions for each.

But, at the end of the day, is there a reason not to keep them all? What is the argument against?

[dawidmalina]

I also agree that we should have all in archive chanel and latest stable with stable chanel and next pre release in unstable chanel.

[lukasz-bielinski]

i think this is somehow related #281

[Starefossen]

The reason for having all versions available is the cases where someone pins to a particular version (like me) and then tries to set up new k8s nodes... having it all break since the particular version was no longer available.

[loskultos]

i agree. in automatic provision scenarios of new nodes not having old versions causes real trouble

[dawidmalina]

We've just had this trouble today when adding more nodes:(

[loskultos]

For alle that were in real trouble today like me...

I had luck and could restore the deb files from the apt cache of a three day old node. Installation by dpkg and after that an apt-get update and apt-get -f install to satisfy the broken dependenccies saved me

[jbeda]

@mikedanese -- I thought you had this? Can you restore these?

[mikedanese]

It was not possible to keep the old kubeadm because it was a 1.6.0-alpha which sorted higher then 1.6.0 so it would have taken priority over the stable release. This should not happen again moving forward since an alpha will never been in the stable repo.

[jbeda]

Here is what is currently hosted on the repo.

$ curl -sL https://apt.kubernetes.io/dists/kubernetes-xenial/main/binary-amd64/Packages | grep -E 'Package|Version'
Package: docker-engine
Version: 1.11.2-0~xenial
Package: kubeadm
Version: 1.5.6-00
Package: kubeadm
Version: 1.6.0-00
Package: kubectl
Version: 1.5.1-00
Package: kubectl
Version: 1.5.2-00
Package: kubectl
Version: 1.5.3-00
Package: kubectl
Version: 1.5.6-00
Package: kubectl
Version: 1.6.0-00
Package: kubelet
Version: 1.5.1-00
Package: kubelet
Version: 1.5.2-00
Package: kubelet
Version: 1.5.3-00
Package: kubelet
Version: 1.5.6-00
Package: kubelet
Version: 1.6.0-00
Package: kubernetes-cni
Version: 0.3.0.1-07a8a2-00
Package: kubernetes-cni
Version: 0.5.1-00
Package: rkt
Version: 1.25.0-1

We have the old versions of CNI, kubelet and kubectl. I also see a 1.5.6-00 for kubeadm. Did you hack something up Mike?

In any case, users can download the old kubeadm from https://storage.googleapis.com/kubernetes-release/release/v1.5.6/bin/linux/amd64/kubeadm.

[mikedanese]

Joe, the most recent version of kubeadm in the deb repo (and only version before yesterday) was 1.6.0-alpha.0.2074-a092d8e0f95f52 which I wasn't able to keep around because it sorted higher the 1.6.0. That deb is available from here:

https://apt.k8s.io/pool/kubeadm_1.6.0-alpha.0.2074-a092d8e0f95f52-00_amd64_0206dba536f698b5777c7d210444a8ace18f48e045ab78687327631c6c694f42.deb

I expect that removing this package is what broke people?

[jbeda]

I suspect that is what broke folks. But I'm a bit confused because it looks like there is a 1.5.6 package. Perhaps that is because we built 1.5.6 after we changed up the version numbers in this repo?

https://apt.kubernetes.io/pool/kubeadm_1.5.6-00_amd64_7bb1e4c1e387ce4fa6eab884f8b6d5ff3f491c1a9c404ad4fd4039a31d344ed3.deb

[mikedanese]

@jbeda, I did a 1.5.6 release yesterday since 1.5.6 was also released yesterday. I pushed kubeadm accidentally even though we were not using the release-1.5 branch for kubeadm and noticed because of kubernetes/kubeadm#213, then left it because 1.6.0 is broken. I'm going to remove it (reasons for that in the linked issue) after 1.6.1 kubeadm is released

[geekofalltrades]

wmd@onsite:ansible $ apt-cache madison kubelet
   kubelet |   1.6.0-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubelet |   1.5.6-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubelet |   1.5.3-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubelet |   1.5.2-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages
   kubelet |   1.5.1-00 | http://apt.kubernetes.io kubernetes-xenial/main amd64 Packages

Sweet! Thanks!

[jbeda]

Thanks @mikedanese. I think the best we can do at this point is document this someplace. I'm going to file a docs bug on this.

[fabiand]

@mikedanese to summarize: Generally speaking you would keep older releases i nsome repoditory, just for 1.6.0 it did not work out, because the NVR was broken?