Etcd Upgrade (v2.3.7) /bin/tee: not found

[edulop91]

1. What kops version : Version 1.8.0-beta.1 (git-9b71713)

2. What Kubernetes version are you running?

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:27:35Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.2", GitCommit:"bdaeafa71f6c7c04636251031f93464384d54963", GitTreeState:"clean", BuildDate:"2017-10-24T19:38:10Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

3. What cloud provider are you using? aws

4. What commands did you run? What is the simplest way to reproduce this issue?
   creating a new cluster with etcd version 2.3.7

5. What happened after the commands executed?
   Master etcd logs:

cat containers/etcd-server-ip-xxxxx.ec2.internal_kube-system_etcd-container-xxxxx.log 
{"log":"/bin/sh: /bin/tee: not found\n","stream":"stderr"}

Locally:

docker run -it  gcr.io/google_containers/etcd:2.3.7 sh     
/ # which tee
/usr/bin/tee
/ # /bin/tee
sh: /bin/tee: not found

6. What did you expect to happen?
   Etcd's run command to be compatible with the etcd image hosted on default gcr.

7. Please provide your cluster manifest.

apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  annotations:
  creationTimestamp: xxxxx
spec:
  api:
    loadBalancer:
      type: Internal
  authorization:
    alwaysAllow: {}
  channel: stable
  cloudProvider: aws
  etcdClusters:
  - etcdMembers:
    - encryptedVolume: true
      instanceGroup: master-us-east-1a
      name: us-east-1a
    - encryptedVolume: true
      instanceGroup: master-us-east-1b
      name: us-east-1b
    - encryptedVolume: true
      instanceGroup: master-us-east-1c
      name: us-east-1c
    name: main
    version: 2.3.7
...

9. Anything else do we need to know?
   Attempting to upgrade a kops cluster running etcd 2.2.x to one running 3.x.x. Seems like I need the intermediate 2.3.x etcd version to accomplish this. Is there perhaps an alternate suggested path to perform this upgrade? Is there a mechanism to point the etcd manifest to an alternate etcd image registry?

Thanks!

[chrislovecnm]

At this point we do not have a documented support way to upgrade from etcd 2 to ectd 3. It seems the problem occurred even before that.

/cc @blakebarnett @KashifSaadat @gambol99

Can any of you gurus assist?

[edulop91]

Thanks for the quick response. Was looking through these images a bit more and looks like the etcd gcr.io/google_containers/3.1.10 actually bundles all preceding supported etcd versions. There is also a migrate-if-needed.sh script included (https://github.com/kubernetes/kubernetes/blob/master/cluster/images/etcd/migrate-if-needed.sh).

I assume a combination of these is the intended path forward. Will test further and report back if something comes out of it.

[chrislovecnm]

We need it to work with HA etcd and eventually first out TLS with upgrades as well.

[justinsb]

@edulop91 yes - that script should work, but we really need to get backups working first! We found that the script did not work for HA originally; it should have been fixed, but IMO we need backups first.

[edulop91]

@justinsb sounds good. We were writing our own backup utility and figured it would be simpler to do for etcdv3; it has nicer snapshot facilities.

is there any ongoing work for backups? perhaps we can contribute towards that. our only unusual requirement is we'd need encryption to be not handled by kms.

[chrislovecnm]

Here is the issue for backups #1896

I do not believe any work has been started.

[krogon]

Despite upgrade path of etcd the root question is the possibility to start etcd from container with tag 2.3.7.
This single version has different placement of tee binary. The following PR #4371 should fix that, as now it relies on PATH:

ac25853#diff-f75b2e0d988be04e2c11fbac7c4ac69dR31

[krogon]

@justinsb I think migrate-if-needed.sh has been fixed in terms of HA: kubernetes/kubernetes@e050e7a#diff-def0cdecfe54802401dbf747e6476269

Once bacups has been merged recently seems like we could go back to API migration once again.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[krogon]

#4371 is merged and etcd upgrade plan is documented in roadmap. I think this can be closed.

[geojaz]

good point @krogon

I think the minor issue of tee problems was closed long ago but the larger issue of how to deal with etcd2->3 migrations is still somewhat outstanding. The etcd-manager which is coming as a part of 1.10 will really really close that out. This is covered in #4463 so i'm happy to clear this one.