Disable dynamic creation for admission hooks and update dependencies #1450

andreyvelich · 2021-02-26T01:23:47Z

Fixes: #1405.

This PR introduces new mechanism to get certificate for webhooks.
I updated YAMLs for our webhooks.
I added initContainer to Katib controller which executes cert-generator.sh script.
This script creates CertificateSigningRequest, katib-webhook-cert secret and patches webhooks configurations with appropriate caBundle.
Since we have katib-webhook-cert secret in the manifest, cleanup process should delete everything.

So we don't need to deploy cert-manager for Katib.

@gaocegege @johnugeorge @yanniszark @kuikuikuizzZ @knkski What do you think about this approach ?

Also I updated controller-runtime to v0.8.2 and k8s.io deps to v0.20.4. That requires some changes:

Change some packages location
Change the arguments for client calls (List, Get, etc.)
In the newer Kubernetes versions we can't add owner reference for cluster-scoped objects (e.g. PV) with namespace-scoped object (e.g. Suggestion). Thus, I have to disable owner reference for the PV which is created when Experiment has FromVolume resume policy. For that reason, I added PersistentVolumeReclaimPolicy: Delete for the PV and once PVC is garbage collected, PV should also be deleted.
I removed PyTorch operator from the dependencies because of this problem.

I still need to make some tests and create new image for cert generator.
It would be great if you can start to review this.

/cc @gaocegege @johnugeorge

k8s-ci-robot · 2021-02-26T01:23:58Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [andreyvelich]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

andreyvelich · 2021-02-26T02:00:55Z

I named the ValidatingWebhookConfiguration with katib.kubeflow.org which contains validator.experiment.katib.kubeflow.org webhook.

I named the MutatingWebhookConfiguration with katib.kubeflow.org which contains defaulter.experiment.katib.kubeflow.org and mutator.pod.katib.kubeflow.org webhooks.

Please let me know what do you think about that naming ?

kuikuikuizzZ · 2021-02-26T02:41:01Z

Is the secret.yaml in manifest still needed? https://github.com/kubeflow/katib/blob/master/manifests/v1beta1/katib-controller/secret.yaml @andreyvelich

andreyvelich · 2021-02-26T03:04:43Z

Is the secret.yaml in manifest still needed? https://github.com/kubeflow/katib/blob/master/manifests/v1beta1/katib-controller/secret.yaml @andreyvelich

Basically this secret is needed only to clean-up all resources after removing Katib from the cluster.
We will include it to the kustomization file.

I think users can also use this secret to create their own certs (We might provide this functionality also).

kuikuikuizzZ · 2021-02-26T03:21:55Z

Is the secret.yaml in manifest still needed? https://github.com/kubeflow/katib/blob/master/manifests/v1beta1/katib-controller/secret.yaml @andreyvelich

Basically this secret is needed only to clean-up all resources after removing Katib from the cluster.
We will include it to the kustomization file.

I think users can also use this secret to create their own certs (We might provide this functionality also).

ok, SGTM

andreyvelich · 2021-02-26T15:52:02Z

@tenzen-y @imilos @zuiurs @trog-levrai I tested this change on GKE cluster with v1.19.7-gke.1500 version and the new build is working for me. I didn't receive error that you noticed here: #1395.

It would be great to see how it works on your platforms since you were facing problems with CommonName before.

/cc @gaocegege @johnugeorge

andreyvelich · 2021-02-26T15:57:36Z

Also I switched to certificates.k8s.io/v1 from certificates.k8s.io/v1beta1 since v1beta1 will be not supported soon.

Thus, I had to update signerName: kubernetes.io/kube-apiserver-client in CSR and RBAC according to this doc: https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#kubernetes-signers.

andreyvelich · 2021-02-26T23:26:43Z

I downgraded CertificateSigningRequest to v1beta1 since we are using v1.18 Kubernetes version for our CI.
cc @PatrickXYS.

The test seems to be working.
I removed WIP status, please take a look.

andreyvelich · 2021-02-26T23:27:03Z

/hold for reviews

PatrickXYS · 2021-02-26T23:45:45Z

@andreyvelich https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.19.md#api-change-2

I see, you can use 1.19 EKS for test now, we delivered it a few days ago

PatrickXYS · 2021-02-26T23:47:59Z

Ah looks like I need to make changes in kubeflow/testing https://github.com/kubeflow/testing/blob/master/images/aws-scripts/create-eks-cluster.sh#L29

And build new images, let me know if you need my help @andreyvelich

andreyvelich · 2021-02-26T23:48:20Z

Thank you for this information @PatrickXYS!
I think it's better to leave v1beta1 version for CSR for now.
I believe many of our users are still using k8s < 1.19

PatrickXYS · 2021-02-26T23:50:17Z

@andreyvelich Good assumption, we definitely need to provide backward compatiability. And staying 1.18 should be a good way to go for now

trog-levrai · 2021-02-27T14:44:30Z

I installed katib with kubeflow on a minikube setup for testing only. I'm afraid I'm not sure how to test it, I tried downloading this branch and doing make deploy but I got the following error log:

bash scripts/v1beta1/deploy.sh
++ dirname scripts/v1beta1/deploy.sh
+ SCRIPT_ROOT=scripts/v1beta1/../..
+ cd scripts/v1beta1/../..
+ kubectl get validatingwebhookconfigurations katib-validating-webhook-config
NAME                              WEBHOOKS   AGE
katib-validating-webhook-config   1          9d
+ kubectl delete validatingwebhookconfigurations katib-validating-webhook-config
validatingwebhookconfiguration.admissionregistration.k8s.io "katib-validating-webhook-config" deleted
+ kubectl get mutatingwebhookconfigurations katib-mutating-webhook-config
NAME                            WEBHOOKS   AGE
katib-mutating-webhook-config   2          9d
+ kubectl delete mutatingwebhookconfigurations katib-mutating-webhook-config
mutatingwebhookconfiguration.admissionregistration.k8s.io "katib-mutating-webhook-config" deleted
+ kubectl apply -f manifests/v1beta1
namespace/kubeflow configured
+ kubectl apply -f manifests/v1beta1/katib-controller
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/experiments.kubeflow.org configured
customresourcedefinition.apiextensions.k8s.io/suggestions.kubeflow.org configured
customresourcedefinition.apiextensions.k8s.io/trials.kubeflow.org configured
configmap/katib-config configured
clusterrole.rbac.authorization.k8s.io/katib-controller configured
serviceaccount/katib-controller configured
clusterrolebinding.rbac.authorization.k8s.io/katib-controller configured
Warning: resource secrets/katib-controller is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
secret/katib-controller configured
service/katib-controller configured
configmap/trial-template configured
The Deployment "katib-controller" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"katib-controller"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable
make: *** [Makefile:28: deploy] Error 1

PS not an expert with K8S, sorry if the answer is obvious. I also assume I should also delete the secret I created as a workaround to make sure the default init works properly ?

andreyvelich · 2021-02-27T22:16:42Z

@trog-levrai Thank you for the testing!

First of all, you have to properly uninstall Katib from your minikube clutser.
Run undeploy.sh from the master branch.

Then, try to use my branch to install updated Katib manifests.
Note that you have to change images for the Katib components.
To build them run build.sh script.
After updating the images, run deploy.sh script.

tenzen-y · 2021-02-28T11:23:47Z

@tenzen-y @imilos @zuiurs @trog-levrai I tested this change on GKE cluster with v1.19.7-gke.1500 version and the new build is working for me. I didn't receive error that you noticed here: #1395.

It would be great to see how it works on your platforms since you were facing problems with CommonName before.

/cc @gaocegege @johnugeorge

@andreyvelich

I tested your changed on K8s v1.19.8 created by kubeadm.
Then, I got the following errors.

Pods in kubectlow

$ kubectl get pods -n kubeflow
NAME                                READY   STATUS                  RESTARTS   AGE
katib-controller-5b7d756869-kj44l   0/1     Init:CrashLoopBackOff   5          11m
katib-db-manager-6895b946bc-g9bwr   1/1     Running                 0          11m
katib-mysql-bd448c8b5-pxvbh         1/1     Running                 0          11m

cert-generator's logs

$ kubectl logs -n kubeflow $(kubectl get pods -n kubeflow -l app=katib-controller -ojsonpath='{.items[*].metadata.name}') -c cert-generator
INFO: Creating certs in tmpdir /tmp/tmp.cHLBnl Generating RSA private key, 2048 bit long modulus (2 primes)................................................................+++++..........................................+++++e is 65537 (0x010001)INFO: Creating CSR: katib-controller.kubeflow 
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
NAME                        AGE     SIGNERNAME                            REQUESTOR                                         CONDITION
katib-controller.kubeflow   6m16s   kubernetes.io/kube-apiserver-client   system:serviceaccount:kubeflow:katib-controller   Approved,Failed
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
certificatesigningrequest.certificates.k8s.io "katib-controller.kubeflow" deleted
WARN: Previous CSR was found and removed.
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
certificatesigningrequest.certificates.k8s.io/katib-controller.kubeflow created
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
NAME                        AGE   SIGNERNAME                            REQUESTOR                                         CONDITION
katib-controller.kubeflow   0s    kubernetes.io/kube-apiserver-client   system:serviceaccount:kubeflow:katib-controller   Pending
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
certificatesigningrequest.certificates.k8s.io/katib-controller.kubeflow approved
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
ERROR: After approving csr katib-controller.kubeflow, the signed certificate did not appear on the resource. Giving up after 1 minute.

certificatesigningrequests

$ kubectl get certificatesigningrequests.certificates.k8s.io 
NAME                        AGE   SIGNERNAME                            REQUESTOR                                         CONDITION
katib-controller.kubeflow   9s    kubernetes.io/kube-apiserver-client   system:serviceaccount:kubeflow:katib-controller   Approved,Failed

certificatesigningrequests status

$ kubectl get certificatesigningrequests.certificates.k8s.io katib-controller.kubeflow -ojsonpath='{.status.conditions}' | jq .
[
  {
    "lastTransitionTime": "2021-02-28T10:54:40Z",
    "lastUpdateTime": "2021-02-28T10:54:40Z",
    "message": "This CSR was approved by kubectl certificate approve.",
    "reason": "KubectlApprove",
    "status": "True",
    "type": "Approved"
  },
  {
    "lastTransitionTime": "2021-02-28T10:54:40Z",
    "lastUpdateTime": "2021-02-28T10:54:40Z",
    "message": "invalid usage for client certificate: server auth",
    "reason": "SignerValidationFailure",
    "status": "True",
    "type": "Failed"
  }
]

I think that server auth isn't included in usages in kubernetes.io/kube-apiserver-client.

https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#kubernetes-signers

Permitted key usages - must include ["client auth"]. Must not include key usages beyond ["digital signature", "key encipherment", "client auth"].

  signerName: kubernetes.io/kube-apiserver-client
  usages:
  - digital signature
  - key encipherment
  - server auth

Thanks.

andreyvelich · 2021-02-28T18:15:43Z

@tenzen-y @imilos @zuiurs @trog-levrai I tested this change on GKE cluster with v1.19.7-gke.1500 version and the new build is working for me. I didn't receive error that you noticed here: #1395.
It would be great to see how it works on your platforms since you were facing problems with CommonName before.
/cc @gaocegege @johnugeorge

@andreyvelich

I tested your changed on K8s v1.19.8 created by kubeadm.
Then, I got the following errors.

Pods in kubectlow

$ kubectl get pods -n kubeflow
NAME                                READY   STATUS                  RESTARTS   AGE
katib-controller-5b7d756869-kj44l   0/1     Init:CrashLoopBackOff   5          11m
katib-db-manager-6895b946bc-g9bwr   1/1     Running                 0          11m
katib-mysql-bd448c8b5-pxvbh         1/1     Running                 0          11m

cert-generator's logs

$ kubectl logs -n kubeflow $(kubectl get pods -n kubeflow -l app=katib-controller -ojsonpath='{.items[*].metadata.name}') -c cert-generator
INFO: Creating certs in tmpdir /tmp/tmp.cHLBnl Generating RSA private key, 2048 bit long modulus (2 primes)................................................................+++++..........................................+++++e is 65537 (0x010001)INFO: Creating CSR: katib-controller.kubeflow 
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
NAME                        AGE     SIGNERNAME                            REQUESTOR                                         CONDITION
katib-controller.kubeflow   6m16s   kubernetes.io/kube-apiserver-client   system:serviceaccount:kubeflow:katib-controller   Approved,Failed
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
certificatesigningrequest.certificates.k8s.io "katib-controller.kubeflow" deleted
WARN: Previous CSR was found and removed.
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
certificatesigningrequest.certificates.k8s.io/katib-controller.kubeflow created
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
NAME                        AGE   SIGNERNAME                            REQUESTOR                                         CONDITION
katib-controller.kubeflow   0s    kubernetes.io/kube-apiserver-client   system:serviceaccount:kubeflow:katib-controller   Pending
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
certificatesigningrequest.certificates.k8s.io/katib-controller.kubeflow approved
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
Warning: certificates.k8s.io/v1beta1 CertificateSigningRequest is deprecated in v1.19+, unavailable in v1.22+; use certificates.k8s.io/v1 CertificateSigningRequest
ERROR: After approving csr katib-controller.kubeflow, the signed certificate did not appear on the resource. Giving up after 1 minute.

certificatesigningrequests

$ kubectl get certificatesigningrequests.certificates.k8s.io 
NAME                        AGE   SIGNERNAME                            REQUESTOR                                         CONDITION
katib-controller.kubeflow   9s    kubernetes.io/kube-apiserver-client   system:serviceaccount:kubeflow:katib-controller   Approved,Failed

certificatesigningrequests status

$ kubectl get certificatesigningrequests.certificates.k8s.io katib-controller.kubeflow -ojsonpath='{.status.conditions}' | jq .
[
  {
    "lastTransitionTime": "2021-02-28T10:54:40Z",
    "lastUpdateTime": "2021-02-28T10:54:40Z",
    "message": "This CSR was approved by kubectl certificate approve.",
    "reason": "KubectlApprove",
    "status": "True",
    "type": "Approved"
  },
  {
    "lastTransitionTime": "2021-02-28T10:54:40Z",
    "lastUpdateTime": "2021-02-28T10:54:40Z",
    "message": "invalid usage for client certificate: server auth",
    "reason": "SignerValidationFailure",
    "status": "True",
    "type": "Failed"
  }
]

I think that server auth isn't included in usages in kubernetes.io/kube-apiserver-client.

https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#kubernetes-signers

Permitted key usages - must include ["client auth"]. Must not include key usages beyond ["digital signature", "key encipherment", "client auth"].

  signerName: kubernetes.io/kube-apiserver-client
  usages:
  - digital signature
  - key encipherment
  - server auth

Thanks.

Make sense, thank you for the testing @tenzen-y!
I changed signerName to kubernetes.io/kubelet-serving to make sure that all usages are supported.
Please can you double check on your side if that CSR is working ?

andreyvelich · 2021-02-28T19:05:03Z

/retest

andreyvelich · 2021-02-28T23:59:18Z

/retest

tenzen-y

Make sense, thank you for the testing @tenzen-y!
I changed signerName to kubernetes.io/kubelet-serving to make sure that all usages are supported.
Please can you double check on your side if that CSR is working ?

@andreyvelich
I checked your changes.
Then, I got subject organization is not system:nodes errors.

$ # pods in kubeflow
$ kubectl get pods -n kubeflow 
NAME                                READY   STATUS                  RESTARTS   AGE
katib-controller-5b7d756869-6rcbj   0/1     Init:CrashLoopBackOff   5          11m
katib-db-manager-6895b946bc-jsblq   1/1     Running                 0          11m
katib-mysql-bd448c8b5-szbdt         1/1     Running                 0          11m
$ # certificatesigningrequests status
$ kubectl get certificatesigningrequests.certificates.k8s.io katib-controller.kubeflow -ojsonpath='{.status.conditions}' | jq .
[
  {
    "lastTransitionTime": "2021-03-01T01:42:20Z",
    "lastUpdateTime": "2021-03-01T01:42:20Z",
    "message": "This CSR was approved by kubectl certificate approve.",
    "reason": "KubectlApprove",
    "status": "True",
    "type": "Approved"
  },
  {
    "lastTransitionTime": "2021-03-01T01:42:20Z",
    "lastUpdateTime": "2021-03-01T01:42:20Z",
    "message": "subject organization is not system:nodes",
    "reason": "SignerValidationFailure",
    "status": "True",
    "type": "Failed"
  }
]

It looks like common name and organizations are not correct.

Permitted subjects - organizations are exactly ["system:nodes"], common name starts with "system:node:".

https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/#kubernetes-signers

It correctly worked when I fixed it.

$ # certificatesigningrequests
$ kubectl get certificatesigningrequests.certificates.k8s.io 
NAME                        AGE    SIGNERNAME                      REQUESTOR                                         CONDITION
katib-controller.kubeflow   112s   kubernetes.io/kubelet-serving   system:serviceaccount:kubeflow:katib-controller   Approved,Issued
# certificatesigningrequests status
$ kubectl get certificatesigningrequests.certificates.k8s.io katib-controller.kubeflow -ojsonpath='{.status.conditions}' | jq .
[
  {
    "lastTransitionTime": "2021-03-01T02:17:24Z",
    "lastUpdateTime": "2021-03-01T02:17:24Z",
    "message": "This CSR was approved by kubectl certificate approve.",
    "reason": "KubectlApprove",
    "status": "True",
    "type": "Approved"
  }
]
$ # deploy tpe-example.yaml
$ kubectl apply -f examples/v1beta1/tpe-example.yaml 
experiment.kubeflow.org/tpe-example created
$ kubectl get pods -n kubeflow 
NAME                                READY   STATUS    RESTARTS   AGE
katib-controller-5b7d756869-5gsbv   1/1     Running   0          3m51s
katib-db-manager-6895b946bc-x26wm   1/1     Running   0          3m52s
katib-mysql-bd448c8b5-zhmhj         1/1     Running   0          3m51s
katib-ui-5dbbdc6596-b6bb2           1/1     Running   0          3m40s
tpe-example-5nljhpbf-6prx9          2/2     Running   0          24s
tpe-example-b5wm8n2t-bfrpr          2/2     Running   0          24s
tpe-example-d2mscqgn-fnxvt          2/2     Running   0          24s
tpe-example-tpe-77dcbf7548-msk67    1/1     Running   0          44s

Thanks.

hack/cert-generator.sh

tenzen-y

@andreyvelich

It looks like that would be better to use environment variables in hack/cert-generator.sh instead of fixing the namespace.

manifests/v1beta1/katib-controller/katib-controller.yaml

hack/cert-generator.sh

johnugeorge · 2021-03-06T07:03:46Z

/lgtm

andreyvelich · 2021-03-06T17:28:47Z

Thanks everyone to help on this PR!
/hold cancel

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

Updates Docker image to include changes from #1450, and updates operator to latest version of operator framework.

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

k8s-ci-robot requested review from gaocegege and johnugeorge February 26, 2021 01:23

k8s-ci-robot added the do-not-merge/work-in-progress label Feb 26, 2021

k8s-ci-robot added approved size/XXL labels Feb 26, 2021

andreyvelich mentioned this pull request Feb 26, 2021

fix: update the api to latest kubeflow pipelines for katib sample[ Fixes #467] kubeflow/kfp-tekton#468

Merged

2 tasks

k8s-ci-robot added the do-not-merge/hold label Feb 26, 2021

andreyvelich changed the title ~~[WIP] Disable dynamic creation for admission hooks and update dependencies~~ Disable dynamic creation for admission hooks and update dependencies Feb 26, 2021

k8s-ci-robot removed the do-not-merge/work-in-progress label Feb 26, 2021

tenzen-y requested changes Mar 1, 2021

View reviewed changes

hack/cert-generator.sh Outdated Show resolved Hide resolved

tenzen-y requested changes Mar 1, 2021

View reviewed changes

manifests/v1beta1/katib-controller/katib-controller.yaml Outdated Show resolved Hide resolved

hack/cert-generator.sh Outdated Show resolved Hide resolved

google-oss-robot assigned johnugeorge Mar 6, 2021

google-oss-robot added the lgtm label Mar 6, 2021

google-oss-robot removed the do-not-merge/hold label Mar 6, 2021

google-oss-robot merged commit 12e7f1e into kubeflow:master Mar 6, 2021

knkski added a commit to knkski/katib that referenced this pull request Mar 9, 2021

Update Katib operator and image

cea53fa

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

knkski added a commit to knkski/katib that referenced this pull request Mar 9, 2021

Update Katib operator and image

0ffa111

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

knkski added a commit to knkski/katib that referenced this pull request Mar 9, 2021

Update Katib operator and image

946b3c1

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

knkski added a commit to knkski/katib that referenced this pull request Mar 9, 2021

Update Katib operator and image

e215f42

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

knkski added a commit to knkski/katib that referenced this pull request Mar 9, 2021

Update Katib operator and image

a913906

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

knkski added a commit to knkski/katib that referenced this pull request Mar 10, 2021

Update Katib operator and image

50fd9c4

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

knkski added a commit to knkski/katib that referenced this pull request Mar 10, 2021

Update Katib operator and image

a29ccdf

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

knkski added a commit to knkski/katib that referenced this pull request Mar 10, 2021

Update Katib operator and image

8063869

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

knkski added a commit to knkski/katib that referenced this pull request Mar 10, 2021

Update Katib operator and image

75e6478

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

knkski added a commit to knkski/katib that referenced this pull request Mar 10, 2021

Update Katib operator and image

a82ad67

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

knkski added a commit to knkski/katib that referenced this pull request Mar 10, 2021

Update Katib operator and image

3fa6fbe

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

andreyvelich mentioned this pull request Mar 10, 2021

[Documentation] Katib webhook information #1266

Closed

knkski added a commit to knkski/katib that referenced this pull request Mar 10, 2021

Update Katib operator and image

b17667d

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

knkski added a commit to knkski/katib that referenced this pull request Mar 10, 2021

Update Katib operator and image

8484ddb

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

knkski added a commit to knkski/katib that referenced this pull request Mar 10, 2021

Update Katib operator and image

40a94b0

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

knkski mentioned this pull request Mar 10, 2021

Update Katib operator and image #1465

Merged

knkski added a commit to knkski/katib that referenced this pull request Mar 11, 2021

Update Katib operator and image

e1d8648

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

knkski added a commit to knkski/katib that referenced this pull request Mar 11, 2021

Update Katib operator and image

c9de12d

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

knkski added a commit to knkski/katib that referenced this pull request Mar 11, 2021

Update Katib operator and image

4bc5a0c

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

google-oss-robot pushed a commit that referenced this pull request Mar 12, 2021

Update Katib operator and image (#1465)

badbbdb

Updates Docker image to include changes from #1450, and updates operator to latest version of operator framework.

andreyvelich pushed a commit to andreyvelich/katib that referenced this pull request Mar 12, 2021

Update Katib operator and image (kubeflow#1465)

a282f3e

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

andreyvelich pushed a commit to andreyvelich/katib that referenced this pull request Mar 12, 2021

Update Katib operator and image (kubeflow#1465)

985dee6

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

andreyvelich pushed a commit to andreyvelich/katib that referenced this pull request Mar 12, 2021

Update Katib operator and image (kubeflow#1465)

4179e47

Updates Docker image to include changes from kubeflow#1450, and updates operator to latest version of operator framework.

andreyvelich deleted the issue-1405-disable-dynamic-webhooks branch October 6, 2021 00:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disable dynamic creation for admission hooks and update dependencies #1450

Disable dynamic creation for admission hooks and update dependencies #1450

andreyvelich commented Feb 26, 2021 •

edited

Loading

k8s-ci-robot commented Feb 26, 2021

andreyvelich commented Feb 26, 2021

kuikuikuizzZ commented Feb 26, 2021

andreyvelich commented Feb 26, 2021 •

edited

Loading

kuikuikuizzZ commented Feb 26, 2021

andreyvelich commented Feb 26, 2021

andreyvelich commented Feb 26, 2021

andreyvelich commented Feb 26, 2021

andreyvelich commented Feb 26, 2021

PatrickXYS commented Feb 26, 2021

PatrickXYS commented Feb 26, 2021

andreyvelich commented Feb 26, 2021

PatrickXYS commented Feb 26, 2021

trog-levrai commented Feb 27, 2021 •

edited

Loading

andreyvelich commented Feb 27, 2021

tenzen-y commented Feb 28, 2021

andreyvelich commented Feb 28, 2021

andreyvelich commented Feb 28, 2021

andreyvelich commented Feb 28, 2021

tenzen-y left a comment •

edited

Loading

tenzen-y left a comment

johnugeorge commented Mar 6, 2021

andreyvelich commented Mar 6, 2021

Disable dynamic creation for admission hooks and update dependencies #1450

Disable dynamic creation for admission hooks and update dependencies #1450

Conversation

andreyvelich commented Feb 26, 2021 • edited Loading

k8s-ci-robot commented Feb 26, 2021

andreyvelich commented Feb 26, 2021

kuikuikuizzZ commented Feb 26, 2021

andreyvelich commented Feb 26, 2021 • edited Loading

kuikuikuizzZ commented Feb 26, 2021

andreyvelich commented Feb 26, 2021

andreyvelich commented Feb 26, 2021

andreyvelich commented Feb 26, 2021

andreyvelich commented Feb 26, 2021

PatrickXYS commented Feb 26, 2021

PatrickXYS commented Feb 26, 2021

andreyvelich commented Feb 26, 2021

PatrickXYS commented Feb 26, 2021

trog-levrai commented Feb 27, 2021 • edited Loading

andreyvelich commented Feb 27, 2021

tenzen-y commented Feb 28, 2021

andreyvelich commented Feb 28, 2021

andreyvelich commented Feb 28, 2021

andreyvelich commented Feb 28, 2021

tenzen-y left a comment • edited Loading

Choose a reason for hiding this comment

tenzen-y left a comment

Choose a reason for hiding this comment

johnugeorge commented Mar 6, 2021

andreyvelich commented Mar 6, 2021

andreyvelich commented Feb 26, 2021 •

edited

Loading

andreyvelich commented Feb 26, 2021 •

edited

Loading

trog-levrai commented Feb 27, 2021 •

edited

Loading

tenzen-y left a comment •

edited

Loading