chore(manifests): refactor manifests for kustomize5 compatibility. Part of #10053

[rawc0der]

Description of your changes:

This PR is part of larger initiative #10053 to refactor manifests and addresses the problems of kustomize5 compatibility.

Warnings thrown before this PR:

$ kustomize build apps/pipeline/upstream/env/cert-manager/platform-agnostic-multi-user/  

# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'bases' is deprecated. Please use 'resources' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesStrategicMerge' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
2023/10/12 15:10:41 well-defined vars that were never replaced: kfp-app-name,kfp-app-version

Expected behaviour after refactoring the resources for kustomize5 is to obtain healthy set of manifests without any warnings.

Remaining Warnings (Work in progress):

 Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
# Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically.
2023/10/12 15:50:46 well-defined vars that were never replaced: kfp-app-version,kfp-app-name

Changes

This PR should fix the deprecated fields for the ml pipelines kustomization stack.
The deprecated fields in kustomize5 to refactor:

• bases -> resources
• patchesStrategicMerge -> patches
• patchesJson6902 -> patchces
• vars -> replacements

After couple more commits I am expecting to have also refactored the remaining vars and patchesJson6902 deprecations from the rest of the resources.

Looking for feedback! (@annajung, @midhun1998)

Checklist:

• The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

[google-oss-prow]

Hi @rawc0der. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[zijianjoy]

/ok-to-test

[juliusvonkohout]

@rawc0der it is usually very helpful to join the kfp working group meeting to speed up PR review and discuss open issues and PRs

[juliusvonkohout]

/retest

[juliusvonkohout]

@zijianjoy this is definitely something that we want for Kubeflow 1.9 in the manifests WG.

@rawc0der please fix the integration tests

[rawc0der]

@rawc0der please fix the integration tests

Great, then I'll find the time to do it this week!

[rawc0der]

/retest

[rawc0der]

Integration tests have been fixed. Some changes were required in the install scripts where older version of Kustomize v2.0.9 was used.
Otherwise there are no more patchesJson6902, patchesStrategicMerge or bases references.

More work is needed for replacing the remaining vars. Working on this and should be ready soon.

FYI @juliusvonkohout

[juliusvonkohout]

@rawc0der please join the KFP meeting on January 31 and put your PR on the agenda in the document in advance. This way we might get this merged soon.

[rawc0der]

@rawc0der please join the KFP meeting on January 31 and put your PR on the agenda in the document in advance. This way we might get this merged soon.

Sure, I will try to join the next KFP meeting to move this forward.

[zijianjoy]

Thank you for the contributions! Would you like to also update other places in the repo?

pipelines/manifests/kustomize/hack/presubmit.sh

Line 26 in 05f69b2

KUSTOMIZE_VERSION=3.10.0

(Just to make sure this doesn't break our postsubmit test)

[rawc0der]

Sure, I can do that. Any other pointers?

[zijianjoy]

The testing mechanism is defined in this folder: https://github.com/GoogleCloudPlatform/oss-test-infra/tree/3f8905d98cdbb7005887113ae692b4024b83188f/prow/prowjobs/kubeflow/pipelines, you can take a look and see if anything needs to be updated.

[rimolive]

@rawc0der Did you have the chance to evaluate the tests?

@zijianjoy Is there a way to run the postsubmit test in a local environment?

[rawc0der]

I will try to spend some time today to wrap this up.

[rimolive]

@rawc0der Any updates? Is there something I can help you?

[rawc0der]

hey @zijianjoy this is bumped already.

[rawc0der]

Hi @rimolive, I've run the test locally. No problems when building the templates. Here's the output.

(⎈ |kind-k8s-1.26:default)$ bash test.sh  

+++ dirname test.sh
++ cd .
++ pwd
+ DIR=/Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack
+ MANIFESTS_DIR=/Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/..
+ kubectl version --client=true
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.0", GitCommit:"a866cbe2e5bbaa01cfd5e969aa3e033f3282a8a2", GitTreeState:"clean", BuildDate:"2022-08-23T17:44:59Z", GoVersion:"go1.19", Compiler:"gc", Platform:"darwin/amd64"}
Kustomize Version: v4.5.7
+ kustomize version
v5.0.3
+ kpt version
1.0.0-beta.49
+ kustomization_yamls=("cluster-scoped-resources" "base/installs/generic" "env/dev" "env/gcp" "env/platform-agnostic" "env/platform-agnostic-pns" "env/platform-agnostic-emissary" "env/aws" "env/azure")
+ for path in '"${kustomization_yamls[@]}"'
+ kubectl kustomize /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../cluster-scoped-resources
+ for path in '"${kustomization_yamls[@]}"'
+ kubectl kustomize /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../base/installs/generic
I0312 23:00:59.093164   18920 log.go:198] well-defined vars that were never replaced: kfp-namespace,kfp-app-name,kfp-app-version,kfp-artifact-bucket-name
+ for path in '"${kustomization_yamls[@]}"'
+ kubectl kustomize /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../env/dev
+ for path in '"${kustomization_yamls[@]}"'
+ kubectl kustomize /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../env/gcp
+ for path in '"${kustomization_yamls[@]}"'
+ kubectl kustomize /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../env/platform-agnostic
I0312 23:00:59.997132   18925 log.go:198] well-defined vars that were never replaced: kfp-app-name,kfp-app-version
+ for path in '"${kustomization_yamls[@]}"'
+ kubectl kustomize /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../env/platform-agnostic-pns
I0312 23:01:00.301708   18926 log.go:198] well-defined vars that were never replaced: kfp-app-name,kfp-app-version
+ for path in '"${kustomization_yamls[@]}"'
+ kubectl kustomize /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../env/platform-agnostic-emissary
I0312 23:01:00.590625   18928 log.go:198] well-defined vars that were never replaced: kfp-app-name,kfp-app-version
+ for path in '"${kustomization_yamls[@]}"'
+ kubectl kustomize /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../env/aws
I0312 23:01:00.915687   18929 log.go:198] well-defined vars that were never replaced: kfp-namespace,kfp-app-name,kfp-app-version
+ for path in '"${kustomization_yamls[@]}"'
+ kubectl kustomize /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../env/azure
I0312 23:01:01.184173   18930 log.go:198] well-defined vars that were never replaced: kfp-app-version,kfp-app-name
+ kustomization_yamls_v3=("base/installs/multi-user" "env/platform-agnostic-multi-user" "env/platform-agnostic-multi-user-pns" "env/platform-agnostic-multi-user-emissary")
+ for path in '"${kustomization_yamls_v3[@]}"'
+ kustomize build /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../base/installs/multi-user
# Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
2024/03/12 23:01:01 well-defined vars that were never replaced: kfp-app-name,kfp-app-version,kfp-artifact-bucket-name
+ for path in '"${kustomization_yamls_v3[@]}"'
+ kustomize build /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../env/platform-agnostic-multi-user
# Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
2024/03/12 23:01:01 well-defined vars that were never replaced: kfp-app-name,kfp-app-version
+ for path in '"${kustomization_yamls_v3[@]}"'
+ kustomize build /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../env/platform-agnostic-multi-user-pns
# Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
2024/03/12 23:01:02 well-defined vars that were never replaced: kfp-app-name,kfp-app-version
+ for path in '"${kustomization_yamls_v3[@]}"'
+ kustomize build /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/../env/platform-agnostic-multi-user-emissary
# Warning: 'vars' is deprecated. Please use 'replacements' instead. [EXPERIMENTAL] Run 'kustomize edit fix' to update your Kustomization automatically.
2024/03/12 23:01:02 well-defined vars that were never replaced: kfp-app-name,kfp-app-version
+ kpt pkg tree /Users/alinspinu/Documents/RawDataLabs/GitHub/kubeflow-pipelines/manifests/kustomize/hack/..

(⎈ |kind-k8s-1.26:default)$ echo $?                
0                                                                                                                 

kpt resources output

``` $ kpt pkg tree ../ (⎈ |kind-k8s-1.26:default) .. ├── application │ ├── [application.yaml] Application $(kfp-app-name) │ └── [kustomization.yaml] Kustomization ├── cache │ ├── [cache-deployment.yaml] Deployment cache-server │ ├── [cache-role.yaml] Role kubeflow-pipelines-cache-role │ ├── [cache-rolebinding.yaml] RoleBinding kubeflow-pipelines-cache-binding │ ├── [cache-sa.yaml] ServiceAccount kubeflow-pipelines-cache │ ├── [cache-service.yaml] Service cache-server │ ├── [kustomization.yaml] Kustomization │ └── cache-deployer │ ├── [cache-deployer-deployment.yaml] Deployment cache-deployer-deployment │ ├── [cache-deployer-role.yaml] Role kubeflow-pipelines-cache-deployer-role │ ├── [cache-deployer-rolebinding.yaml] RoleBinding kubeflow-pipelines-cache-deployer-rolebinding │ ├── [kustomization.yaml] Kustomization │ └── cluster-scoped │ ├── [cache-deployer-clusterrole.yaml] ClusterRole kubeflow-pipelines-cache-deployer-clusterrole │ ├── [cache-deployer-clusterrolebinding.yaml] ClusterRoleBinding kubeflow-pipelines-cache-deployer-clusterrolebinding │ ├── [cache-deployer-sa.yaml] ServiceAccount kubeflow-pipelines-cache-deployer-sa │ └── [kustomization.yaml] Kustomization ├── generic │ ├── [kustomization.yaml] Kustomization │ ├── [mysql-secret.yaml] Secret mysql-secret │ ├── [pipeline-install-config.yaml] ConfigMap pipeline-install-config │ └── postgres │ ├── [kustomization.yaml] Kustomization │ ├── [pipeline-install-config.yaml] ConfigMap pipeline-install-config │ └── [postgres-secret-extended.yaml] Secret postgres-secret-extended ├── multi-user │ ├── [istio-authorization-config.yaml] DestinationRule ml-pipeline │ ├── [istio-authorization-config.yaml] DestinationRule ml-pipeline-ui │ ├── [istio-authorization-config.yaml] DestinationRule ml-pipeline-visualizationserver │ ├── [istio-authorization-config.yaml] AuthorizationPolicy kubeflow/ml-pipeline │ ├── [istio-authorization-config.yaml] AuthorizationPolicy kubeflow/ml-pipeline-ui │ ├── [istio-authorization-config.yaml] AuthorizationPolicy kubeflow/ml-pipeline-visualizationserver │ ├── [istio-authorization-config.yaml] AuthorizationPolicy kubeflow/service-cache-server │ ├── [kustomization.yaml] Kustomization │ ├── [view-edit-cluster-roles.yaml] ClusterRole aggregate-to-kubeflow-pipelines-edit │ ├── [view-edit-cluster-roles.yaml] ClusterRole aggregate-to-kubeflow-pipelines-view │ ├── [view-edit-cluster-roles.yaml] ClusterRole kubeflow-pipelines-edit │ ├── [view-edit-cluster-roles.yaml] ClusterRole kubeflow-pipelines-view │ ├── [virtual-service.yaml] VirtualService ml-pipeline-ui │ ├── api-service │ │ ├── [cluster-role-binding.yaml] ClusterRoleBinding ml-pipeline │ │ ├── [cluster-role.yaml] ClusterRole ml-pipeline │ │ ├── [deployment-patch.yaml] Deployment ml-pipeline │ │ └── [kustomization.yaml] Kustomization │ ├── cache │ │ ├── [cluster-role-binding.yaml] ClusterRoleBinding kubeflow-pipelines-cache-binding │ │ ├── [cluster-role.yaml] ClusterRole kubeflow-pipelines-cache-role │ │ ├── [deployment-patch.yaml] Deployment cache-server │ │ └── [kustomization.yaml] Kustomization │ ├── metadata-writer │ │ ├── [cluster-role-binding.yaml] ClusterRoleBinding kubeflow-pipelines-metadata-writer-binding │ │ ├── [cluster-role.yaml] ClusterRole kubeflow-pipelines-metadata-writer-role │ │ ├── [deployment-patch.yaml] Deployment metadata-writer │ │ └── [kustomization.yaml] Kustomization │ ├── persistence-agent │ │ ├── [cluster-role-binding.yaml] ClusterRoleBinding ml-pipeline-persistenceagent-binding │ │ ├── [cluster-role.yaml] ClusterRole ml-pipeline-persistenceagent-role │ │ ├── [deployment-patch.yaml] Deployment ml-pipeline-persistenceagent │ │ └── [kustomization.yaml] Kustomization │ ├── pipelines-profile-controller │ │ ├── [composite-controller.yaml] CompositeController kubeflow-pipelines-profile-controller │ │ ├── [deployment.yaml] Deployment kubeflow-pipelines-profile-controller │ │ ├── [kustomization.yaml] Kustomization │ │ └── [service.yaml] Service kubeflow-pipelines-profile-controller │ ├── pipelines-ui │ │ ├── [cluster-role-binding.yaml] ClusterRoleBinding ml-pipeline-ui │ │ ├── [cluster-role.yaml] ClusterRole ml-pipeline-ui │ │ ├── [configmap-patch.yaml] ConfigMap ml-pipeline-ui-configmap │ │ ├── [deployment-patch.yaml] Deployment ml-pipeline-ui │ │ └── [kustomization.yaml] Kustomization │ ├── scheduled-workflow │ │ ├── [cluster-role-binding.yaml] ClusterRoleBinding ml-pipeline-scheduledworkflow-binding │ │ ├── [cluster-role.yaml] ClusterRole ml-pipeline-scheduledworkflow-role │ │ ├── [deployment-patch.yaml] Deployment ml-pipeline-scheduledworkflow │ │ └── [kustomization.yaml] Kustomization │ └── viewer-controller │ ├── [cluster-role-binding.yaml] ClusterRoleBinding ml-pipeline-viewer-crd-binding │ ├── [cluster-role.yaml] ClusterRole ml-pipeline-viewer-controller-role │ ├── [deployment-patch.yaml] Deployment ml-pipeline-viewer-crd │ └── [kustomization.yaml] Kustomization ├── base │ ├── [kustomization.yaml] Kustomization │ ├── [metadata-envoy-deployment.yaml] Deployment metadata-envoy-deployment │ ├── [metadata-envoy-service.yaml] Service metadata-envoy-service │ ├── [metadata-grpc-configmap.yaml] ConfigMap metadata-grpc-configmap │ ├── [metadata-grpc-deployment.yaml] Deployment metadata-grpc-deployment │ ├── [metadata-grpc-sa.yaml] ServiceAccount metadata-grpc-server │ └── [metadata-grpc-service.yaml] Service metadata-grpc-service ├── istio │ ├── [destination-rule.yaml] DestinationRule metadata-grpc-service │ ├── [istio-authorization-policy.yaml] AuthorizationPolicy metadata-grpc-service │ ├── [kustomization.yaml] Kustomization │ └── [virtual-service.yaml] VirtualService kubeflow/metadata-grpc ├── db │ ├── [kustomization.yaml] Kustomization │ ├── [metadata-db-deployment.yaml] Deployment metadata-db │ ├── [metadata-db-pvc.yaml] PersistentVolumeClaim metadata-mysql │ ├── [metadata-db-service.yaml] Service metadata-db │ └── patches │ └── [metadata-grpc-deployment.yaml] Deployment metadata-grpc-deployment ├── postgres │ ├── [kustomization.yaml] Kustomization │ ├── [metadata-db-deployment.yaml] Deployment metadata-postgres-db │ ├── [metadata-db-pvc.yaml] PersistentVolumeClaim metadata-postgres │ ├── [metadata-db-service.yaml] Service metadata-postgres-db │ └── patches │ └── [metadata-grpc-deployment.yaml] Deployment metadata-grpc-deployment ├── pipeline │ ├── [container-builder-sa.yaml] ServiceAccount kubeflow-pipelines-container-builder │ ├── [kfp-launcher-configmap.yaml] ConfigMap kfp-launcher │ ├── [kustomization.yaml] Kustomization │ ├── [ml-pipeline-apiserver-deployment.yaml] Deployment ml-pipeline │ ├── [ml-pipeline-apiserver-role.yaml] Role ml-pipeline │ ├── [ml-pipeline-apiserver-rolebinding.yaml] RoleBinding ml-pipeline │ ├── [ml-pipeline-apiserver-sa.yaml] ServiceAccount ml-pipeline │ ├── [ml-pipeline-apiserver-service.yaml] Service ml-pipeline │ ├── [ml-pipeline-persistenceagent-deployment.yaml] Deployment ml-pipeline-persistenceagent │ ├── [ml-pipeline-persistenceagent-role.yaml] Role ml-pipeline-persistenceagent-role │ ├── [ml-pipeline-persistenceagent-rolebinding.yaml] RoleBinding ml-pipeline-persistenceagent-binding │ ├── [ml-pipeline-persistenceagent-sa.yaml] ServiceAccount ml-pipeline-persistenceagent │ ├── [ml-pipeline-scheduledworkflow-deployment.yaml] Deployment ml-pipeline-scheduledworkflow │ ├── [ml-pipeline-scheduledworkflow-role.yaml] Role ml-pipeline-scheduledworkflow-role │ ├── [ml-pipeline-scheduledworkflow-rolebinding.yaml] RoleBinding ml-pipeline-scheduledworkflow-binding │ ├── [ml-pipeline-scheduledworkflow-sa.yaml] ServiceAccount ml-pipeline-scheduledworkflow │ ├── [ml-pipeline-ui-configmap.yaml] ConfigMap ml-pipeline-ui-configmap │ ├── [ml-pipeline-ui-deployment.yaml] Deployment ml-pipeline-ui │ ├── [ml-pipeline-ui-role.yaml] Role ml-pipeline-ui │ ├── [ml-pipeline-ui-rolebinding.yaml] RoleBinding ml-pipeline-ui │ ├── [ml-pipeline-ui-sa.yaml] ServiceAccount ml-pipeline-ui │ ├── [ml-pipeline-ui-service.yaml] Service ml-pipeline-ui │ ├── [ml-pipeline-viewer-crd-deployment.yaml] Deployment ml-pipeline-viewer-crd │ ├── [ml-pipeline-viewer-crd-role.yaml] Role ml-pipeline-viewer-controller-role │ ├── [ml-pipeline-viewer-crd-rolebinding.yaml] RoleBinding ml-pipeline-viewer-crd-binding │ ├── [ml-pipeline-viewer-crd-sa.yaml] ServiceAccount ml-pipeline-viewer-crd-service-account │ ├── [ml-pipeline-visualization-deployment.yaml] Deployment ml-pipeline-visualizationserver │ ├── [ml-pipeline-visualization-sa.yaml] ServiceAccount ml-pipeline-visualizationserver │ ├── [ml-pipeline-visualization-service.yaml] Service ml-pipeline-visualizationserver │ ├── [pipeline-runner-role.yaml] Role pipeline-runner │ ├── [pipeline-runner-rolebinding.yaml] RoleBinding pipeline-runner-binding │ ├── [pipeline-runner-sa.yaml] ServiceAccount pipeline-runner │ ├── [viewer-sa.yaml] ServiceAccount kubeflow-pipelines-viewer │ ├── cluster-scoped │ │ ├── [kustomization.yaml] Kustomization │ │ ├── [scheduled-workflow-crd.yaml] CustomResourceDefinition scheduledworkflows.kubeflow.org │ │ └── [viewer-crd.yaml] CustomResourceDefinition viewers.kubeflow.org │ └── metadata-writer │ ├── [kustomization.yaml] Kustomization │ ├── [metadata-writer-deployment.yaml] Deployment metadata-writer │ ├── [metadata-writer-role.yaml] Role kubeflow-pipelines-metadata-writer-role │ ├── [metadata-writer-rolebinding.yaml] RoleBinding kubeflow-pipelines-metadata-writer-binding │ └── [metadata-writer-sa.yaml] ServiceAccount kubeflow-pipelines-metadata-writer ├── cache │ ├── [cache-deployment-patch.yaml] Deployment cache-server │ └── [kustomization.yaml] Kustomization ├── pipeline │ ├── [kustomization.yaml] Kustomization │ └── [ml-pipeline-apiserver-deployment-patch.yaml] Deployment ml-pipeline ├── cluster-scoped-resources │ ├── [kustomization.yaml] Kustomization │ └── [namespace.yaml] Namespace $(kfp-cluster-scoped-namespace) ├── aws │ ├── [aws-configuration-pipeline-patch.yaml] Deployment ml-pipeline │ ├── [aws-configuration-pipeline-ui-patch.yaml] Deployment ml-pipeline-ui │ └── [kustomization.yaml] Kustomization ├── azure │ ├── [kustomization.yaml] Kustomization │ └── minio-azure-gateway │ ├── [kustomization.yaml] Kustomization │ ├── [minio-azure-gateway-deployment.yaml] Deployment minio │ └── [minio-azure-gateway-service.yaml] Service minio-service ├── base │ ├── [cache-cert-issuer.yaml] Issuer kfp-cache-selfsigned-issuer │ ├── [cache-cert.yaml] Certificate kfp-cache-cert │ ├── [cache-webhook-config.yaml] MutatingWebhookConfiguration cache-webhook-kubeflow │ └── [kustomization.yaml] Kustomization ├── cluster-scoped-resources │ └── [kustomization.yaml] Kustomization ├── dev │ ├── [delete-cache-deployer.yaml] Deployment cache-deployer-deployment │ ├── [delete-cache-deployer.yaml] Role kubeflow-pipelines-cache-deployer-role │ ├── [delete-cache-deployer.yaml] RoleBinding kubeflow-pipelines-cache-deployer-rolebinding │ ├── [kustomization.yaml] Kustomization │ └── [namespace.yaml] Namespace $(kfp-dev-namespace) ├── platform-agnostic-multi-user │ ├── [kustomization.yaml] Kustomization │ └── patches │ ├── [delete.clusterrole.cache-deployer.yaml] ClusterRole kubeflow-pipelines-cache-deployer-clusterrole │ ├── [delete.crb.cache-deployer.yaml] ClusterRoleBinding kubeflow-pipelines-cache-deployer-clusterrolebinding │ ├── [delete.deployment.cache-deployer.yaml] Deployment cache-deployer-deployment │ ├── [delete.role.cache-deployer.yaml] Role kubeflow-pipelines-cache-deployer-role │ ├── [delete.rolebinding.cache-deployer.yaml] RoleBinding kubeflow-pipelines-cache-deployer-rolebinding │ └── [delete.sa.cache-deployer.yaml] ServiceAccount kubeflow-pipelines-cache-deployer-sa ├── dev │ ├── [kustomization.yaml] Kustomization │ └── postgresql │ └── [kustomization.yaml] Kustomization ├── gcp │ ├── [gcp-configurations-patch.yaml] Deployment ml-pipeline │ ├── [kustomization.yaml] Kustomization │ ├── cloudsql-proxy │ │ ├── [cloudsql-proxy-deployment.yaml] Deployment cloudsqlproxy │ │ ├── [cloudsql-proxy-sa.yaml] ServiceAccount kubeflow-pipelines-cloudsql-proxy │ │ ├── [kustomization.yaml] Kustomization │ │ └── [mysql-service.yaml] Service mysql │ ├── inverse-proxy │ │ ├── [kustomization.yaml] Kustomization │ │ ├── [proxy-configmap.yaml] ConfigMap inverse-proxy-config │ │ ├── [proxy-deployment.yaml] Deployment proxy-agent │ │ ├── [proxy-role.yaml] Role proxy-agent-runner │ │ ├── [proxy-rolebinding.yaml] RoleBinding proxy-agent-runner │ │ └── [proxy-sa.yaml] ServiceAccount proxy-agent-runner │ └── minio-gcs-gateway │ ├── [kustomization.yaml] Kustomization │ ├── [minio-gcs-gateway-deployment.yaml] Deployment minio │ ├── [minio-gcs-gateway-sa.yaml] ServiceAccount kubeflow-pipelines-minio-gcs-gateway │ └── [minio-gcs-gateway-service.yaml] Service minio-service ├── platform-agnostic │ ├── [kustomization.yaml] Kustomization │ ├── platform-agnostic-emissary │ │ ├── [kustomization.yaml] Kustomization │ │ └── [workflow-controller-configmap-patch.yaml] ConfigMap workflow-controller-configmap │ ├── platform-agnostic-multi-user │ │ ├── [kustomization.yaml] Kustomization │ │ └── platform-agnostic-multi-user-pns │ │ ├── [kustomization.yaml] Kustomization │ │ └── [workflow-controller-configmap-patch.yaml] ConfigMap workflow-controller-configmap │ ├── platform-agnostic-multi-user-emissary │ │ ├── [kustomization.yaml] Kustomization │ │ └── [workflow-controller-configmap-patch.yaml] ConfigMap workflow-controller-configmap │ ├── platform-agnostic-multi-user-legacy │ │ └── [kustomization.yaml] Kustomization │ ├── platform-agnostic-pns │ │ ├── [kustomization.yaml] Kustomization │ │ └── [workflow-controller-configmap-patch.yaml] ConfigMap workflow-controller-configmap │ └── platform-agnostic-postgresql │ └── [kustomization.yaml] Kustomization ├── sample │ ├── [kustomization.yaml] Kustomization │ └── cluster-scoped-resources │ └── [kustomization.yaml] Kustomization ├── application │ ├── [application-controller-deployment.yaml] Deployment controller-manager │ ├── [application-controller-role.yaml] Role application-manager-role │ ├── [application-controller-rolebinding.yaml] RoleBinding application-manager-rolebinding │ ├── [application-controller-sa.yaml] ServiceAccount application │ ├── [application-controller-service.yaml] Service controller-manager-service │ ├── [kustomization.yaml] Kustomization │ └── cluster-scoped │ ├── [application-crd.yaml] CustomResourceDefinition applications.app.k8s.io │ └── [kustomization.yaml] Kustomization ├── Package "argo" │ ├── [Kptfile] Kptfile argo │ ├── base │ │ ├── [kustomization.yaml] Kustomization │ │ ├── [workflow-controller-configmap-patch.yaml] ConfigMap workflow-controller-configmap │ │ └── [workflow-controller-deployment-patch.yaml] Deployment workflow-controller │ ├── cluster │ │ └── [kustomization.yaml] Kustomization │ ├── namespace │ │ ├── [kustomization.yaml] Kustomization │ │ └── cluster-scoped │ │ └── [kustomization.yaml] Kustomization │ ├── Package "manifests" │ │ ├── [Kptfile] Kptfile manifests │ │ ├── base │ │ │ ├── [kustomization.yaml] Kustomization │ │ │ ├── full │ │ │ │ ├── [argoproj.io_clusterworkflowtemplates.yaml] CustomResourceDefinition clusterworkflowtemplates.argoproj.io │ │ │ │ ├── [argoproj.io_cronworkflows.yaml] CustomResourceDefinition cronworkflows.argoproj.io │ │ │ │ ├── [argoproj.io_workfloweventbindings.yaml] CustomResourceDefinition workfloweventbindings.argoproj.io │ │ │ │ ├── [argoproj.io_workflows.yaml] CustomResourceDefinition workflows.argoproj.io │ │ │ │ ├── [argoproj.io_workflowtaskresults.yaml] CustomResourceDefinition workflowtaskresults.argoproj.io │ │ │ │ ├── [argoproj.io_workflowtasksets.yaml] CustomResourceDefinition workflowtasksets.argoproj.io │ │ │ │ ├── [argoproj.io_workflowtemplates.yaml] CustomResourceDefinition workflowtemplates.argoproj.io │ │ │ │ └── [kustomization.yaml] Kustomization │ │ │ └── workflow-controller │ │ │ ├── [kustomization.yaml] Kustomization │ │ │ ├── [workflow-controller-configmap.yaml] ConfigMap workflow-controller-configmap │ │ │ ├── [workflow-controller-deployment.yaml] Deployment workflow-controller │ │ │ ├── [workflow-controller-metrics-service.yaml] Service workflow-controller-metrics │ │ │ ├── [workflow-controller-priorityclass.yaml] PriorityClass workflow-controller │ │ │ └── [workflow-controller-sa.yaml] ServiceAccount argo │ │ ├── cluster-install │ │ │ └── [kustomization.yaml] Kustomization │ │ ├── argo-server-rbac │ │ │ ├── [argo-server-clusterole.yaml] ClusterRole argo-server-cluster-role │ │ │ ├── [argo-server-clusterolebinding.yaml] ClusterRoleBinding argo-server-binding │ │ │ └── [kustomization.yaml] Kustomization │ │ ├── namespace-install │ │ │ └── [kustomization.yaml] Kustomization │ │ ├── argo-server-rbac │ │ │ ├── [argo-server-role.yaml] Role argo-server-role │ │ │ ├── [argo-server-rolebinding.yaml] RoleBinding argo-server-binding │ │ │ └── [kustomization.yaml] Kustomization │ │ ├── base │ │ │ ├── [agent-default-rolebinding.yaml] RoleBinding agent-default │ │ │ ├── [agent-role.yaml] Role agent │ │ │ ├── [argo-server-sso-secret.yaml] Secret argo-server-sso │ │ │ ├── [artifact-repositories-configmap.yaml] ConfigMap artifact-repositories │ │ │ ├── [cluster-workflow-template-rbac.yaml] ClusterRole argo-clusterworkflowtemplate-role │ │ │ ├── [cluster-workflow-template-rbac.yaml] ClusterRoleBinding argo-clusterworkflowtemplate-role-binding │ │ │ ├── [cluster-workflow-template-rbac.yaml] ClusterRole argo-server-clusterworkflowtemplate-role │ │ │ ├── [cluster-workflow-template-rbac.yaml] ClusterRoleBinding argo-server-clusterworkflowtemplate-role-binding │ │ │ ├── [executor-default-rolebinding.yaml] RoleBinding executor-default │ │ │ ├── [kustomization.yaml] Kustomization │ │ │ ├── [memoizer-default-rolebinding.yaml] RoleBinding memoizer-default │ │ │ ├── [memoizer-role.yaml] Role memoizer │ │ │ ├── [pod-manager-default-rolebinding.yaml] RoleBinding pod-manager-default │ │ │ ├── [pod-manager-role.yaml] Role pod-manager │ │ │ ├── [workflow-default-rolebinding.yaml] RoleBinding workflow-default-binding │ │ │ ├── [workflow-manager-default-rolebinding.yaml] RoleBinding workflow-manager-default │ │ │ ├── [workflow-manager-role.yaml] Role workflow-manager │ │ │ ├── emissary │ │ │ │ └── [executor-role.yaml] Role executor │ │ │ ├── pns │ │ │ │ └── [executor-role.yaml] Role executor │ │ │ ├── minio │ │ │ │ ├── [kustomization.yaml] Kustomization │ │ │ │ ├── [minio-deploy.yaml] Deployment minio │ │ │ │ ├── [minio-service.yaml] Service minio │ │ │ │ └── [my-minio-cred-secret.yaml] Secret my-minio-cred │ │ │ ├── overlays │ │ │ │ ├── [argo-server-deployment.yaml] Deployment argo-server │ │ │ │ └── [workflow-controller-configmap.yaml] ConfigMap workflow-controller-configmap │ │ │ ├── prometheus │ │ │ │ ├── [kustomization.yaml] Kustomization │ │ │ │ ├── [prometheus-config-cluster.yaml] ConfigMap prometheus-config │ │ │ │ ├── [prometheus-deployment.yaml] Deployment prometheus │ │ │ │ └── [prometheus-service.yaml] Service prometheus │ │ │ └── webhooks │ │ │ ├── [argo-workflows-webhook-clients-secret.yaml] Secret argo-workflows-webhook-clients │ │ │ ├── [github.com-rolebinding.yaml] RoleBinding github.com │ │ │ ├── [github.com-sa.yaml] ServiceAccount github.com │ │ │ ├── [kustomization.yaml] Kustomization │ │ │ └── [submit-workflow-template-role.yaml] Role submit-workflow-template │ │ ├── docker │ │ │ └── [executor-role.yaml] Role executor │ │ ├── k8sapi │ │ │ └── [executor-role.yaml] Role executor │ │ ├── kubelet │ │ │ ├── [executor-role.yaml] Role executor │ │ │ ├── [kubelet-executor-clusterrole.yaml] ClusterRole kubelet-executor │ │ │ └── [kubelet-executor-default-clusterrolebinding.yaml] ClusterRoleBinding kubelet-executor-default │ │ ├── minimal │ │ │ └── [kustomization.yaml] Kustomization │ │ ├── overlays │ │ │ └── [workflow-controller-configmap.yaml] ConfigMap workflow-controller-configmap │ │ ├── mysql │ │ │ ├── [argo-mysql-config-secret.yaml] Secret argo-mysql-config │ │ │ ├── [kustomization.yaml] Kustomization │ │ │ ├── [mysql-deployment.yaml] Deployment mysql │ │ │ ├── [mysql-service.yaml] Service mysql │ │ │ └── overlays │ │ │ └── [workflow-controller-configmap.yaml] ConfigMap workflow-controller-configmap │ │ ├── postgres │ │ │ ├── [argo-postgres-config-secret.yaml] Secret argo-postgres-config │ │ │ ├── [kustomization.yaml] Kustomization │ │ │ ├── [postgres-deployment.yaml] Deployment postgres │ │ │ ├── [postgres-service.yaml] Service postgres │ │ │ └── overlays │ │ │ └── [workflow-controller-configmap.yaml] ConfigMap workflow-controller-configmap │ │ └── dex │ │ ├── [dev-svc.yaml] Service dex │ │ ├── [dex-cm.yaml] ConfigMap dex │ │ ├── [dex-deploy.yaml] Deployment dex │ │ ├── [dex-rb.yaml] RoleBinding dex │ │ ├── [dex-role.yaml] Role dex │ │ ├── [dex-sa.yaml] ServiceAccount dex │ │ └── [kustomization.yaml] Kustomization │ ├── argo-server │ │ ├── [argo-server-deployment.yaml] Deployment argo-server │ │ ├── [argo-server-sa.yaml] ServiceAccount argo-server │ │ ├── [argo-server-service.yaml] Service argo-server │ │ └── [kustomization.yaml] Kustomization │ ├── crds │ │ ├── [kustomization.yaml] Kustomization │ │ └── minimal │ │ ├── [argoproj.io_clusterworkflowtemplates.yaml] CustomResourceDefinition clusterworkflowtemplates.argoproj.io │ │ ├── [argoproj.io_cronworkflows.yaml] CustomResourceDefinition cronworkflows.argoproj.io │ │ ├── [argoproj.io_workfloweventbindings.yaml] CustomResourceDefinition workfloweventbindings.argoproj.io │ │ ├── [argoproj.io_workflows.yaml] CustomResourceDefinition workflows.argoproj.io │ │ ├── [argoproj.io_workflowtaskresults.yaml] CustomResourceDefinition workflowtaskresults.argoproj.io │ │ ├── [argoproj.io_workflowtasksets.yaml] CustomResourceDefinition workflowtasksets.argoproj.io │ │ ├── [argoproj.io_workflowtemplates.yaml] CustomResourceDefinition workflowtemplates.argoproj.io │ │ └── [kustomization.yaml] Kustomization │ ├── workflow-controller-rbac │ │ ├── [kustomization.yaml] Kustomization │ │ ├── [workflow-aggregate-roles.yaml] ClusterRole argo-aggregate-to-admin │ │ ├── [workflow-aggregate-roles.yaml] ClusterRole argo-aggregate-to-edit │ │ ├── [workflow-aggregate-roles.yaml] ClusterRole argo-aggregate-to-view │ │ ├── [workflow-controller-clusterrole.yaml] ClusterRole argo-cluster-role │ │ ├── [workflow-controller-clusterrolebinding.yaml] ClusterRoleBinding argo-binding │ │ ├── [workflow-controller-role.yaml] Role argo-role │ │ └── [workflow-controller-rolebinding.yaml] RoleBinding argo-binding │ ├── workflow-controller-rbac │ │ ├── [kustomization.yaml] Kustomization │ │ ├── [workflow-controller-role.yaml] Role argo-role │ │ └── [workflow-controller-rolebinding.yaml] RoleBinding argo-binding │ └── sso │ ├── [kustomization.yaml] Kustomization │ └── overlays │ ├── [argo-server-sa.yaml] ServiceAccount argo-server │ └── [workflow-controller-configmap.yaml] ConfigMap workflow-controller-configmap ├── grafana │ ├── [grafana-deployment.yaml] Deployment kubeflow/grafana │ ├── [grafana-role.yaml] Role grafana │ ├── [grafana-rolebinding.yaml] RoleBinding grafana │ ├── [grafana-sa.yaml] ServiceAccount grafana │ ├── [grafana-service.yaml] Service grafana │ └── [kustomization.yaml] Kustomization ├── base │ ├── [cluster-role-binding.yaml] ClusterRoleBinding meta-controller-cluster-role-binding │ ├── [crd.yaml] CustomResourceDefinition compositecontrollers.metacontroller.k8s.io │ ├── [crd.yaml] CustomResourceDefinition controllerrevisions.metacontroller.k8s.io │ ├── [crd.yaml] CustomResourceDefinition decoratorcontrollers.metacontroller.k8s.io │ ├── [kustomization.yaml] Kustomization │ ├── [service-account.yaml] ServiceAccount meta-controller-service │ └── [stateful-set.yaml] StatefulSet metacontroller ├── base │ ├── [kustomization.yaml] Kustomization │ ├── [minio-deployment.yaml] Deployment minio │ ├── [minio-pvc.yaml] PersistentVolumeClaim minio-pvc │ ├── [minio-service.yaml] Service minio-service │ └── [mlpipeline-minio-artifact-secret.yaml] Secret mlpipeline-minio-artifact ├── istio │ ├── [istio-authorization-policy.yaml] AuthorizationPolicy minio-service │ ├── [istio-authorization-policy.yaml] DestinationRule ml-pipeline-minio │ └── [kustomization.yaml] Kustomization ├── base │ ├── [kustomization.yaml] Kustomization │ ├── [mysql-deployment.yaml] Deployment mysql │ ├── [mysql-pv-claim.yaml] PersistentVolumeClaim mysql-pv-claim │ ├── [mysql-service.yaml] Service mysql │ └── [mysql-serviceaccount.yaml] ServiceAccount mysql ├── istio │ ├── [istio-authorization-policy.yaml] DestinationRule ml-pipeline-mysql │ ├── [istio-authorization-policy.yaml] AuthorizationPolicy kubeflow/mysql │ └── [kustomization.yaml] Kustomization ├── base │ ├── [kustomization.yaml] Kustomization │ ├── [pg-deployment.yaml] Deployment postgres │ ├── [pg-pvc.yaml] PersistentVolumeClaim postgres-pvc │ ├── [pg-secret.yaml] Secret postgres-secret │ ├── [pg-service.yaml] Service postgres-service │ └── [pg-serviceaccount.yaml] ServiceAccount postgresql └── prometheus ├── [kustomization.yaml] Kustomization ├── [prometheus-configmap.yaml] ConfigMap prometheus-configmap ├── [prometheus-deployment.yaml] Deployment prometheus ├── [prometheus-role.yaml] Role prometheus ├── [prometheus-rolebinding.yaml] RoleBinding prometheus ├── [prometheus-sa.yaml] ServiceAccount prometheus └── [prometheus-service.yaml] Service prometheus ```

From my side think there are no more modifications (FYI vars are not modified anywhere for consistency).
I am waiting on some review and CI retest. What do you think?

[rawc0der]

/retest

[rimolive]

Looks good to me.

/lgtm

[juliusvonkohout]

/lgtm

[rimolive]

@zijianjoy Can you review/approve?

[rimolive]

@Tomcli Can you review/approve?

[chensun]

Although this is probably safe, but let's wait until I cut a release first, so there's one less unknown (potential breaker).

/hold

[Tomcli]

Can users still use kubectl apply -k .? Maybe it depends on the kubectl version, but if users only can use kustomize, then we probably need to follow up with another PR to update the documents as well.

[chensun]

/unhold
/lgtm

[chensun]

Undo LGTM.

Waiting for @Tomcli to add LGTM once his comment is addressed.

[Tomcli]

Can users still use kubectl apply -k .?

@rawc0der I want to confirm users can still use kubectl apply -k after this PR. This is not a blocker for this PR. I just want to make sure we are going to address the documentations after this PR is merged.

[rimolive]

Can users still use kubectl apply -k .?

@rawc0der I want to confirm users can still use kubectl apply -k after this PR. This is not a blocker for this PR. I just want to make sure we are going to address the documentations after this PR is merged.

The latest kubectl release comes with kustomize 5.0.4, so probably the best is change it to use kustomize build instead of kubectl apply -k. cc @juliusvonkohout

[Tomcli]

/lgtm

I opened another issue to track the doc changes. Otherwise it looks good to me.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chensun, Tomcli

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• manifests/kustomize/OWNERS [chensun]
• test/OWNERS [chensun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[juliusvonkohout]

Can users still use kubectl apply -k .?

@rawc0der I want to confirm users can still use kubectl apply -k after this PR. This is not a blocker for this PR. I just want to make sure we are going to address the documentations after this PR is merged.

The latest kubectl release comes with kustomize 5.0.4, so probably the best is change it to use kustomize build instead of kubectl apply -k. cc @juliusvonkohout

The latest kubectl might soon have kustomize 5.2.1+ Maybe kubectl 1.30 or so. Nevertheless it will probably still work since this PR also worked with 5.0.3 before. But @Tomcli yes we should mention kustomize explicitly and just pipe it into kubectl apply as shown in kubeflow/manifests/readme.md.