Debugging warnings in pss workflow (kubeflow#2866)

* Debugging warnings in pss workflow Signed-off-by: biswajit-9776 <[email protected]> * Shifted order of applying patches to workflow Signed-off-by: biswajit-9776 <[email protected]> * Fixed linting Signed-off-by: biswajit-9776 <[email protected]> * Increased timeout Signed-off-by: biswajit-9776 <[email protected]> * Skip patching dynamic namespaces Signed-off-by: biswajit-9776 <[email protected]> * Remove debugging job Signed-off-by: biswajit-9776 <[email protected]> * Debugging by restarting deployments Signed-off-by: biswajit-9776 <[email protected]> * Removed redundant line from patch Signed-off-by: biswajit-9776 <[email protected]> * Replace restart with wait command Signed-off-by: biswajit-9776 <[email protected]> * Replace wait for all command with individual wait command Signed-off-by: biswajit-9776 <[email protected]> * Added wait command for pods in kubeflow namespace Signed-off-by: biswajit-9776 <[email protected]> * Fixed linting Signed-off-by: biswajit-9776 <[email protected]> * Separated wait commands for separate namespaces Signed-off-by: biswajit-9776 <[email protected]> * Increased timeout to 600s Signed-off-by: biswajit-9776 <[email protected]> * Added wait commands in loop Signed-off-by: biswajit-9776 <[email protected]> * Fixed typo Signed-off-by: biswajit-9776 <[email protected]> * Debugging failing wait commands Signed-off-by: biswajit-9776 <[email protected]> * Added log process in background Signed-off-by: biswajit-9776 <[email protected]> * Fixed type Signed-off-by: biswajit-9776 <[email protected]> * Fixed typo Signed-off-by: biswajit-9776 <[email protected]> * Added describe command Signed-off-by: biswajit-9776 <[email protected]> * Added runAsUser to debug Signed-off-by: biswajit-9776 <[email protected]> * Added UID 1000 to all pods Signed-off-by: biswajit-9776 <[email protected]> * Added single wait command Signed-off-by: biswajit-9776 <[email protected]> * Debugging Signed-off-by: biswajit-9776 <[email protected]> * Removed background process Signed-off-by: biswajit-9776 <[email protected]> * Increased timeout Signed-off-by: biswajit-9776 <[email protected]> * Retesting Signed-off-by: biswajit-9776 <[email protected]> * Removed UID from profile controller Signed-off-by: biswajit-9776 <[email protected]> * Debugging profiles-controller Signed-off-by: biswajit-9776 <[email protected]> * Increased timeout for profiles-controller Signed-off-by: biswajit-9776 <[email protected]> * Pausing patch to profile-controller Signed-off-by: biswajit-9776 <[email protected]> * Fixed indentation Signed-off-by: biswajit-9776 <[email protected]> * Fixed error Signed-off-by: biswajit-9776 <[email protected]> * Fixed error Signed-off-by: biswajit-9776 <[email protected]> * Fixed error Signed-off-by: biswajit-9776 <[email protected]> * Debugging cache-server Signed-off-by: biswajit-9776 <[email protected]> * Debugging cache-server Signed-off-by: biswajit-9776 <[email protected]> * Increased timeout for cache-server Signed-off-by: biswajit-9776 <[email protected]> * Debugging Signed-off-by: biswajit-9776 <[email protected]> * Deleting pods exclusively Signed-off-by: biswajit-9776 <[email protected]> * Added GID Signed-off-by: biswajit-9776 <[email protected]> * Debugging cache-server Signed-off-by: biswajit-9776 <[email protected]> * Sleeping for 300s Signed-off-by: biswajit-9776 <[email protected]> * Updated GID to 1000 Signed-off-by: biswajit-9776 <[email protected]> * Added GID to all pods Signed-off-by: biswajit-9776 <[email protected]> * Checking pods securityContext Signed-off-by: biswajit-9776 <[email protected]> * Debugging Signed-off-by: biswajit-9776 <[email protected]> * Debugging without IDs Signed-off-by: biswajit-9776 <[email protected]> * Debugging Signed-off-by: biswajit-9776 <[email protected]> * Debugging Signed-off-by: biswajit-9776 <[email protected]> * Debugging Signed-off-by: biswajit-9776 <[email protected]> * Added GID Signed-off-by: biswajit-9776 <[email protected]> * Debugging Signed-off-by: biswajit-9776 <[email protected]> * Debugging Signed-off-by: biswajit-9776 <[email protected]> * Debugging Signed-off-by: biswajit-9776 <[email protected]> * Debugging Signed-off-by: biswajit-9776 <[email protected]> * Debugging Signed-off-by: biswajit-9776 <[email protected]> * Debugging Signed-off-by: biswajit-9776 <[email protected]> * Debugging Signed-off-by: biswajit-9776 <[email protected]> * Debugging Signed-off-by: biswajit-9776 <[email protected]> * Debugging Signed-off-by: biswajit-9776 <[email protected]> * Debugging Signed-off-by: biswajit-9776 <[email protected]> * Debugging Signed-off-by: biswajit-9776 <[email protected]> * Debugging Signed-off-by: biswajit-9776 <[email protected]> * Debugging Signed-off-by: biswajit-9776 <[email protected]> * Added wait time for istio-cni test Signed-off-by: biswajit-9776 <[email protected]> * Debugging Signed-off-by: biswajit-9776 <[email protected]> * Increased timeout Signed-off-by: biswajit-9776 <[email protected]> * Added dynamic ns to pss_test Signed-off-by: biswajit-9776 <[email protected]> * Fix lint Signed-off-by: biswajit-9776 <[email protected]> * Configured istio initContainer Signed-off-by: biswajit-9776 <[email protected]> * Fixed lint Signed-off-by: biswajit-9776 <[email protected]> * Added seccompProfile attribute for istio-proxy Signed-off-by: biswajit-9776 <[email protected]> * Adding PSS label to dynamic ns Signed-off-by: biswajit-9776 <[email protected]> * Reordered dynamic ns test Signed-off-by: biswajit-9776 <[email protected]> * Commented the dynamic ns tests Signed-off-by: biswajit-9776 <[email protected]> * Fix lint Signed-off-by: biswajit-9776 <[email protected]> * Removing debugging stuffs Signed-off-by: biswajit-9776 <[email protected]> --------- Signed-off-by: biswajit-9776 <[email protected]>
hansinikarunarathne · Sep 27, 2024 · e3b2aa4 · e3b2aa4
1 parent 944e829
commit e3b2aa4
Show file tree

Hide file tree

Showing 20 changed files with 91 additions and 33 deletions.
diff --git a/.github/workflows/pss_test.yaml b/.github/workflows/pss_test.yaml
@@ -11,7 +11,7 @@ on:
     - common/cert-manager/**
     - common/oauth2-proxy/**
     - common/istio*/**
-    - tests/gh-actions/install_istio_with_ext_auth.sh
+    - tests/gh-actions/install_istio-cni.sh
     - tests/gh-actions/install_multitenancy.sh
 
 jobs:
@@ -27,12 +27,25 @@ jobs:
     - name: Install kubectl
       run: ./tests/gh-actions/install_kubectl.sh
 
-    - name: Install all deployments from static namespaces
+    - name: Install all istio-cni resources and kubeflow namespace
       run: |
         kustomize build common/kubeflow-namespace/base | kubectl apply -f -
         ./tests/gh-actions/install_cert_manager.sh
-        ./tests/gh-actions/install_istio_with_ext_auth.sh
-        kustomize build common/istio-1-22/kubeflow-istio-resources/base | kubectl apply -f -
+        ./tests/gh-actions/install_istio-cni.sh
+        kustomize build common/istio-cni-1-22/kubeflow-istio-resources/base | kubectl apply -f -
+
+    - name: Configure istio init container with seccompProfile attribute
+      run: |
+        kubectl get cm istio-sidecar-injector -n istio-system -o yaml > temporary_patch.yaml
+        sed -i '0,/runAsNonRoot: true/{s//&\n              seccompProfile:\n                type: RuntimeDefault/}' temporary_patch.yaml
+        sed -i '/runAsNonRoot: true/{N; /runAsUser: {{ .ProxyUID | default "1337" }}/a\
+                      seccompProfile:\n                type: RuntimeDefault
+        }' temporary_patch.yaml
+        kubectl apply -f temporary_patch.yaml
+        rm temporary_patch.yaml
+
+    - name: Install all other deployments of static namespaces
+      run: |
         ./tests/gh-actions/install_multi_tenancy.sh
         kustomize build ./common/oauth2-proxy/overlays/m2m-self-signed | kubectl apply -f -
         echo "Waiting for all oauth2-proxy pods to become ready..."
@@ -44,33 +57,6 @@ jobs:
     - name: Install KF Pipelines
       run: ./tests/gh-actions/install_pipelines.sh
 
-    - name: Apply Pod Security Standards baseline levels for static namespaces
-      run: ./tests/gh-actions/enable_baseline_PSS.sh
-
-    - name: Apply Pod Security Standards baseline levels for dynamic namespaces
-      run: |
-        cat << EOF > ./kustomization.yaml
-        apiVersion: kustomize.config.k8s.io/v1beta1
-        kind: Kustomization
-        resources:
-        - apps/profiles/upstream/overlays/kubeflow
-        components:
-        - contrib/security/PSS/dynamic/baseline
-        EOF
-        kubectl apply -k .
-        rm ./kustomization.yaml
-        kubectl -n kubeflow wait --for=condition=Ready pods -l kustomize.component=profiles --timeout 180s
-
-    - name: Unapply applied baseline values
-      run: |
-        NAMESPACES=("istio-system" "auth" "cert-manager" "oauth2-proxy" "kubeflow")
-        for NAMESPACE in "${NAMESPACES[@]}"; do
-          if kubectl get namespace "$NAMESPACE" >/dev/null 2>&1; then
-            kubectl label namespace $NAMESPACE pod-security.kubernetes.io/enforce-
-          fi
-        done
-        sleep 10
-
     - name: Apply patches to clear warnings
       run: |
         DIRECTORY="contrib/security/PSS/patches"
@@ -87,6 +73,19 @@ jobs:
             kubectl patch "$KIND" "$NAME" -n "$NAMESPACE" --patch-file "$file"
           fi
         done
+        sleep 300
+
+    - name: Apply Pod Security Standards baseline levels for static namespaces
+      run: ./tests/gh-actions/enable_baseline_PSS.sh
+
+    - name: Unapply applied baseline labels
+      run: |
+        NAMESPACES=("istio-system" "auth" "cert-manager" "oauth2-proxy" "kubeflow")
+        for NAMESPACE in "${NAMESPACES[@]}"; do
+          if kubectl get namespace "$NAMESPACE" >/dev/null 2>&1; then
+            kubectl label namespace $NAMESPACE pod-security.kubernetes.io/enforce-
+          fi
+        done
 
     - name: Applying Pod Security Standards restricted levels for static namespaces
       run: ./tests/gh-actions/enable_restricted_PSS.sh
diff --git a/contrib/security/PSS/patches/cache-server.yaml b/contrib/security/PSS/patches/cache-server.yaml
@@ -13,6 +13,8 @@ spec:
           seccompProfile:
             type: RuntimeDefault
           runAsNonRoot: true
+          runAsUser: 1000
+          runAsGroup: 0
           capabilities:
             drop:
             - ALL
diff --git a/contrib/security/PSS/patches/kfam.yaml b/contrib/security/PSS/patches/kfam.yaml
@@ -13,6 +13,8 @@ spec:
           seccompProfile:
             type: RuntimeDefault
           runAsNonRoot: true
+          runAsUser: 1000
+          runAsGroup: 0
           capabilities:
             drop:
             - ALL
diff --git a/contrib/security/PSS/patches/kubeflow-pipelines-profile-controller.yaml b/contrib/security/PSS/patches/kubeflow-pipelines-profile-controller.yaml
@@ -13,6 +13,8 @@ spec:
           seccompProfile:
             type: RuntimeDefault
           runAsNonRoot: true
+          runAsUser: 1000
+          runAsGroup: 0
           capabilities:
             drop:
             - ALL
diff --git a/contrib/security/PSS/patches/manager.yaml b/contrib/security/PSS/patches/manager.yaml
@@ -13,6 +13,8 @@ spec:
           seccompProfile:
             type: RuntimeDefault
           runAsNonRoot: true
+          runAsUser: 1000
+          runAsGroup: 0
           capabilities:
             drop:
             - ALL
diff --git a/contrib/security/PSS/patches/metacontroller.yaml b/contrib/security/PSS/patches/metacontroller.yaml
@@ -0,0 +1,13 @@
+apiVersion: apps/v1
+kind: StatefulSet
+metadata:
+  name: metacontroller
+  namespace: kubeflow
+spec:
+  template:
+    spec:
+      containers:
+      - name: metacontroller
+        securityContext:
+          seccompProfile: 
+            type: RuntimeDefault
diff --git a/contrib/security/PSS/patches/metadata-envoy-deployment.yaml b/contrib/security/PSS/patches/metadata-envoy-deployment.yaml
@@ -13,6 +13,8 @@ spec:
           seccompProfile:
             type: RuntimeDefault
           runAsNonRoot: true
+          runAsUser: 1000
+          runAsGroup: 0
           capabilities:
             drop:
             - ALL
diff --git a/contrib/security/PSS/patches/metadata-grpc-deployment.yaml b/contrib/security/PSS/patches/metadata-grpc-deployment.yaml
@@ -13,6 +13,8 @@ spec:
           seccompProfile:
             type: RuntimeDefault
           runAsNonRoot: true
+          runAsUser: 1000
+          runAsGroup: 0
           capabilities:
             drop:
             - ALL
diff --git a/contrib/security/PSS/patches/metadata-writer.yaml b/contrib/security/PSS/patches/metadata-writer.yaml
@@ -13,6 +13,8 @@ spec:
           seccompProfile:
             type: RuntimeDefault
           runAsNonRoot: true
+          runAsUser: 1000
+          runAsGroup: 0
           capabilities:
             drop:
             - ALL
diff --git a/contrib/security/PSS/patches/minio.yaml b/contrib/security/PSS/patches/minio.yaml
@@ -13,6 +13,8 @@ spec:
           seccompProfile:
             type: RuntimeDefault
           runAsNonRoot: true
+          runAsUser: 1000
+          runAsGroup: 0
           capabilities:
             drop:
             - ALL
diff --git a/contrib/security/PSS/patches/ml-pipeline-persistenceagent.yaml b/contrib/security/PSS/patches/ml-pipeline-persistenceagent.yaml
@@ -13,6 +13,8 @@ spec:
           seccompProfile:
             type: RuntimeDefault
           runAsNonRoot: true
+          runAsUser: 1000
+          runAsGroup: 0
           capabilities:
             drop:
             - ALL
diff --git a/contrib/security/PSS/patches/ml-pipeline-scheduledworkflow.yaml b/contrib/security/PSS/patches/ml-pipeline-scheduledworkflow.yaml
@@ -13,6 +13,8 @@ spec:
           seccompProfile:
             type: RuntimeDefault
           runAsNonRoot: true
+          runAsUser: 1000
+          runAsGroup: 0
           capabilities:
             drop:
             - ALL
diff --git a/contrib/security/PSS/patches/ml-pipeline-ui.yaml b/contrib/security/PSS/patches/ml-pipeline-ui.yaml
@@ -13,6 +13,8 @@ spec:
           seccompProfile:
             type: RuntimeDefault
           runAsNonRoot: true
+          runAsUser: 1000
+          runAsGroup: 0
           capabilities:
             drop:
             - ALL
diff --git a/contrib/security/PSS/patches/ml-pipeline-viewer-crd.yaml b/contrib/security/PSS/patches/ml-pipeline-viewer-crd.yaml
@@ -13,6 +13,8 @@ spec:
           seccompProfile:
             type: RuntimeDefault
           runAsNonRoot: true
+          runAsUser: 1000
+          runAsGroup: 0
           capabilities:
             drop:
             - ALL
diff --git a/contrib/security/PSS/patches/ml-pipeline-visualizationserver.yaml b/contrib/security/PSS/patches/ml-pipeline-visualizationserver.yaml
@@ -13,6 +13,8 @@ spec:
           seccompProfile:
             type: RuntimeDefault
           runAsNonRoot: true
+          runAsUser: 1000
+          runAsGroup: 0
           capabilities:
             drop:
             - ALL
diff --git a/contrib/security/PSS/patches/ml-pipeline.yaml b/contrib/security/PSS/patches/ml-pipeline.yaml
@@ -13,6 +13,8 @@ spec:
           seccompProfile:
             type: RuntimeDefault
           runAsNonRoot: true
+          runAsUser: 1000
+          runAsGroup: 0
           capabilities:
             drop:
             - ALL
diff --git a/contrib/security/PSS/patches/mysql.yaml b/contrib/security/PSS/patches/mysql.yaml
@@ -13,6 +13,8 @@ spec:
           seccompProfile:
             type: RuntimeDefault
           runAsNonRoot: true
+          runAsUser: 1000
+          runAsGroup: 0
           capabilities:
             drop:
             - ALL
diff --git a/contrib/security/PSS/patches/oauth2-proxy.yaml b/contrib/security/PSS/patches/oauth2-proxy.yaml
@@ -4,7 +4,6 @@ metadata:
   name: oauth2-proxy
   namespace: oauth2-proxy
 spec:
-  replicas: 2
   template:
     spec:
       containers:

diff --git a/contrib/security/PSS/patches/workflow-controller.yaml b/contrib/security/PSS/patches/workflow-controller.yaml
@@ -0,0 +1,14 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: workflow-controller
+  namespace: kubeflow
+spec:
+  template:
+    spec:
+      containers:
+      - name: workflow-controller
+        securityContext:
+          seccompProfile:
+            type: RuntimeDefault
+          runAsNonRoot: true
diff --git a/tests/gh-actions/install_istio-cni.sh b/tests/gh-actions/install_istio-cni.sh
@@ -4,4 +4,7 @@ echo "Installing Istio-cni ..."
 cd common/istio-cni-1-22
 kustomize build istio-crds/base | kubectl apply -f -
 kustomize build istio-namespace/base | kubectl apply -f -
-kustomize build istio-install/base | kubectl apply -f -
+kustomize build istio-install/base | kubectl apply -f -
+
+echo "Waiting for all Istio Pods to become ready..."
+kubectl wait --for=condition=Ready pods --all -n istio-system --timeout 300s