Staging deployments not being cleaned up

[NimJay]

• Background info:
  • We have a GitHub Action that creates a staging deployment of Online Boutique for each pull-request that alters relevant source code. These staging deployments are hosted on a staging cluster named prs-gke-cluster.
  • A staging deployment for a given pull-request is supposed to be cleaned up by /.github/workflows/cleanup.yaml once the pull-request is closed/merged.
• Problem: Staging deployments aren't being cleaned up (even though pull-requests have been closed/merged).
• Example: On Aug 19 (2024), the following pull-requests were closed/merged but their staging deployments were still running:
  • chore(deps): refresh pip-compile outputs python #2644
  • chore(deps): update gcr.io/k8s-skaffold/skaffold docker tag to v2.13.1 #2652
  • feat: add support for baseUrl #2609
  • chore(deps): update dotnet #2637
  • chore(deps): update kubevious/cli action to v1.0.62 #2638
  • chore(deps): update dotnet to v2.65.0 #2664
  • chore(deps): update docker #2661
  • chore(deps): update dotnet to v2.64.0 #2651
  • chore(deps): update node.js to v20.15.1 #2639
  • fix(deps): update java #2666
  • fix(deps): update java #2684
  • chore(deps): update docker #2649
  • fix(deps): update nodejs #2640
  • fix(deps): update module cloud.google.com/go/secretmanager to v1.13.5 #2663
  • fix(deps): update dependency pino to v9.3.2 #2662
  • chore(deps): refresh pip-compile outputs python #2667
  • fix(deps): update golang #2641
  • fix(deps): update nodejs #2655
  • chore(deps): update java #2643
• This issue uses up CPU/Memory unnecessarily. See our staging autopilot cluster (prs-gke-cluster)'s CPU and memory:

[NimJay]

For now, I've manually deleted those namespaces (for which pull-requests have been closed/merged):

kubectl delete namespace pr2644
kubectl delete namespace pr2652
kubectl delete namespace pr2609
kubectl delete namespace pr2637
kubectl delete namespace pr2638
kubectl delete namespace pr2664
kubectl delete namespace pr2661
kubectl delete namespace pr2651
kubectl delete namespace pr2639
kubectl delete namespace pr2666
kubectl delete namespace pr2684
kubectl delete namespace pr2649
​​kubectl delete namespace pr2640
kubectl delete namespace pr2663
kubectl delete namespace pr2662
kubectl delete namespace pr2667
kubectl delete namespace pr2641
kubectl delete namespace pr2655
kubectl delete namespace pr2643 

Looking into root cause now.

[NimJay]

• Ah, it looks like the "Approve and run" GitHub org-wide policy is preventing cleanup.yaml from running automatically.

• The screenshot above is for this run/invocation of the cleanup.yaml workflow. I will click "Approve and run" now. Update: I clicked "Approve and run" but the workflow failed because I had already manually cleaned up the namespace — see my previous comment. All good.

• CC: @bourgeoisor, just FYI. :) For now, whenever we merge (or close) a pull-request, we may need to manually "Approve an run" the cleanup.yaml workflow. See list of cleanup.yaml workflows "runs" (invocations) here.

[bourgeoisor]

Oh wow good catch!-- And extremely annoying.