check function error when enableServiceLinks option set in k8s-1.14.3

[icefed]

Submission type

• Bug report

Environment

Kubernetes 1.14.3

Stolon version

v0.13.0-pg9.6

Additional environment information if useful to understand the bug

Add enableServiceLinks: false option to stolon-* PodSpec, Keeper and Proxy print error and can not work.

Expected behaviour you didn't see

Unexpected behaviour you saw

Stolon-Proxy error output

2019-06-26T08:00:26.228Z INFO cmd/proxy.go:302 check function error {"error": "failed to update proxyInfo: update failed: Pod "stolon-proxy-5f5f8cb9cd-82cmr" is invalid: spec: Forbidden: pod updates may not change fields other than spec.containers[*].image, spec.initContainers[*].image, spec.activeDeadlineSeconds or spec.tolerations (only additions to existing tolerations)\n{"Volumes":[{"Name":"default-token-hs6cs","HostPath":null,"EmptyDir":null,"GCEPersistentDisk":null,"AWSElasticBlockStore":null,"GitRepo":null,"Secret":{"SecretName":"default-token-hs6cs","Items":null,"DefaultMode":420,"Optional":null},"NFS":null,"ISCSI":null,"Glusterfs":null,"PersistentVolumeClaim":null,"RBD":null,"Quobyte":null,"FlexVolume":null,"Cinder":null,"CephFS":null,"Flocker":null,"DownwardAPI":null,"FC":null,"AzureFile":null,"ConfigMap":null,"VsphereVolume":null,"AzureDisk":null,"PhotonPersistentDisk":null,"Projected":null,"PortworxVolume":null,"ScaleIO":null,"StorageOS":null,"CSI":null}],"InitContainers":null,"Containers":[{"Name":"stolon-proxy","Image":"sorintlab/stolon:v0.13.0-pg9.6","Command":["/bin/bash","-ec","exec gosu stolon stolon-proxy\n"],"Args":null,"WorkingDir":"","Ports":[{"Name":"","HostPort":0,"ContainerPort":5432,"Protocol":"TCP","HostIP":""},{"Name":"","HostPort":0,"ContainerPort":8080,"Protocol":"TCP","HostIP":""}],"EnvFrom":null,"Env":[{"Name":"POD_NAME","Value":"","ValueFrom":{"FieldRef":{"APIVersion":"v1","FieldPath":"metadata.name"},"ResourceFieldRef":null,"ConfigMapKeyRef":null,"SecretKeyRef":null}},{"Name":"STPROXY_CLUSTER_NAME","Value":"","ValueFrom":{"FieldRef":{"APIVersion":"v1","FieldPath":"metadata.labels['stolon-cluster']"},"ResourceFieldRef":null,"ConfigMapKeyRef":null,"SecretKeyRef":null}},{"Name":"STPROXY_STORE_BACKEND","Value":"kubernetes","ValueFrom":null},{"Name":"STPROXY_KUBE_RESOURCE_KIND","Value":"configmap","ValueFrom":null},{"Name":"STPROXY_LISTEN_ADDRESS","Value":"0.0.0.0","ValueFrom":null},{"Name":"STPROXY_METRICS_LISTEN_ADDRESS","Value":"0.0.0.0:8080","ValueFrom":null},{"Name":"STPROXY_TCP_KEEPALIVE_IDLE","Value":"600","ValueFrom":null},{"Name":"STPROXY_TCP_KEEPALIVE_COUNT","Value":"8","ValueFrom":null},{"Name":"STPROXY_TCP_KEEPALIVE_INTERVAL","Value":"75","ValueFrom":null}],"Resources":{"Limits":null,"Requests":null},"VolumeMounts":[{"Name":"default-token-hs6cs","ReadOnly":true,"MountPath":"/var/run/secrets/kubernetes.io/serviceaccount","SubPath":"","MountPropagation":null,"SubPathExpr":""}],"VolumeDevices":null,"LivenessProbe":null,"ReadinessProbe":{"Exec":null,"HTTPGet":null,"TCPSocket":{"Port":5432,"Host":""},"InitialDelaySeconds":10,"TimeoutSeconds":5,"PeriodSeconds":10,"SuccessThreshold":1,"FailureThreshold":3},"Lifecycle":null,"TerminationMessagePath":"/dev/termination-log","TerminationMessagePolicy":"File","ImagePullPolicy":"IfNotPresent","SecurityContext":null,"Stdin":false,"StdinOnce":false,"TTY":false}],"RestartPolicy":"Always","TerminationGracePeriodSeconds":30,"ActiveDeadlineSeconds":null,"DNSPolicy":"ClusterFirst","NodeSelector":{"node-role.kubernetes.io/master":""},"ServiceAccountName":"default","AutomountServiceAccountToken":null,"NodeName":"n227","SecurityContext":{"HostNetwork":false,"HostPID":false,"HostIPC":false,"ShareProcessNamespace":null,"SELinuxOptions":null,"RunAsUser":null,"RunAsGroup":null,"RunAsNonRoot":null,"SupplementalGroups":null,"FSGroup":null,"Sysctls":null},"ImagePullSecrets":null,"Hostname":"","Subdomain":"","Affinity":null,"SchedulerName":"default-scheduler","Tolerations":[{"Key":"node.kubernetes.io/not-ready","Operator":"Exists","Value":"","Effect":"NoExecute","TolerationSeconds":300},{"Key":"node.kubernetes.io/unreachable","Operator":"Exists","Value":"","Effect":"NoExecute","TolerationSeconds":300}],"HostAliases":null,"PriorityClassName":"","Priority":0,"DNSConfig":null,"ReadinessGates":null,"RuntimeClassName":null,"EnableServiceLinks":\n\nA: true}\n\nB: false}\n\n"}

Steps to reproduce the problem

[harmjanblok]

Any update on this issue, we seem to have a similar / the same problem.

• Running on kubernetes 1.13.11
• Tested with stolon v0.11 and v0.14
• Applies to all stolon components (proxy, keeper, sentinel)

2019-09-20T14:03:36.637Z        ERROR   cmd/sentinel.go:1857    cannot update sentinel info     {"error": "update failed: Pod \"stolon-sentinel-5d58cc4ffc-hr66w\" is invalid: spec: Forbidden: pod updates may not change fields other than ``spec.containers[*].image`,` `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)

On kubernetes 1.12.8 with stolon v0.11 we didn't see this error.

[sgotti]

@icefed I haven't tried with enableServiceLinks set to false. I'm not sure how this is related. Perhaps an issue in the k8s client version we are using.

@harmjanblok Are you too disabling enableServiceLinks?

[icefed]

Remove enableServiceLinks then deploy stolon is fine, mybe we can update k8s client version and try again.

And enableServiceLinks added sine kubernetes 1.13 kubernetes/kubernetes#68754

[harmjanblok]

No I don't specify anything for enableServiceLinks, pods have assigned the default (enableServiceLinks: true).

[harmjanblok]

In my setup it seems to fail because it want to set ProcMount: null, while it currently uses the default ProcMount: "Default" (ProcMount is inside the pod SecurityContext). I'm still working on a reproducable setup which I can share.

[icefed]

ProcMount added to SecurityContext since kubernetes 1.12
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md#new-features

[sgotti]

@harmjanblok If you haven't changed enableServiceLinks to false your issue is different than the one originally reported. Can you open a new issues with the steps to reproduce it (with the steps to reproduce it in a simple k8s environment like minikube)?

[harmjanblok]

@icefed thanks for the link, I'll investigate further, my issue seems to be caused by a bug introduced with this change: kubernetes/kubernetes#78881
@sgotti yes it seems to be a different issue. Originally thought both might related to the used client-go libraries, hence I commented in this thread.

[harmjanblok]

In harmjanblok@fbfaf92 I've pushed the test setup I was using to reproduce the issue. I've been able to reproduce both issues (enableServiceLinks and procMount), although the procMount was caused by a kubernetes bug.

Steps to reproduce:

cd test-kubernetes
kind create cluster --config kind-config.yml
export KUBECONFIG="$(kind get kubeconfig-path --name="kind")"
kubectl apply -f k8s/
kubectl exec -it deploy/stolon-sentinel -- stolonctl init --cluster-name kube-stolon --store-backend kubernetes --kube-resource-kind configmap
kubectl logs deploy/stolon-sentinel -f
...
kind delete cluster