Use new GKE cluster for ci-pr.yaml #2418

NimJay · 2024-03-13T18:38:29Z

Background

Currently, the deployment-tests GitHub Action is broken — for other pull-requests (see example error logs).
Specifically, deployment of the adservice (Java microservice) is failing:
The error we're seeing in the GKE cluster:

pod/adservice-746b58f97b-gsxzr: container server terminated with exit code 139 ...
A fatal error has been detected by the Java Runtime Environment: ...
Problematic frame:
      > [adservice-746b58f97b-gsxzr server] # C  [profiler_java_agent.so+0x897522]  std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::assign(char const*)+0xc ...
Can not save log file, dump to screen...
VM state: not at safepoint (not fully initialized) ...
VM Mutex/Monitor currently owned by a thread: None ...

Based on our investigation, this seems like an issue with the Nodes currently used by the "online-boutique-prs" GKE cluster. Reason for conclusion: the images deploy fine in other clusters.

Changes from this PR

In this PR, we:
- Add Terraform for a new autopilot GKE cluster ("prs-gke-cluster"). I have terraform apply-ed.
- Update GitHub Actions to use the new GKE cluster instead of the existing "online-boutique-prs" cluster.
- Add a README.md about using the Terraform.

How to test

We have to make sure the GitHub checks work (in this pull-request). Passed. ✅
We should also check the staging URL. Works. ✅
After merging, we should check that:
- cleanup.yaml successfully cleans up the staging deployment created by this pull-request.
- ci-main.yaml successfully deploys the HEAD commit in main to the new cluster (prs-gke-cluster).

Additional info

Just so we can track the gains from switching to autopilot, I've created a billing report (Jan 1, 2023 to Mar 14, 2024). See Google-internal report at go/online-boutique-ci-billing-report-2023.
I have updated go/sample-app-ci (Google-internal doc).

github-actions · 2024-03-13T20:18:53Z

🚲 PR staged at http://34.133.162.82

bourgeoisor

LGTM. Checks are all passing.

github-actions · 2024-03-14T17:46:28Z

🚲 PR staged at http://34.133.162.82

NimJay · 2024-03-14T18:06:57Z

.github/workflows/ci-main.yaml

-        PR_CLUSTER: "online-boutique-prs"
-        ZONE: "us-central1-c"
+        PR_CLUSTER: "prs-gke-cluster"
+        REGION: "us-central1"


Rationale for change: Since autopilot = regional and standard = zonal.

github-actions · 2024-03-14T18:19:01Z

🚲 PR staged at http://34.133.162.82

NimJay · 2024-03-14T18:33:16Z

.github/terraform/README.md

+    * Ideally, you would see `Apply complete! Resources: 0 added, 0 changed, 0 destroyed.` in the output.
+1. Make your desired changes to the Terraform code.
+1. Apply the Terraform: `terraform apply -var project_id=${PROJECT_ID}`
+    * This time, Terraform will prompt you confirm your changes before applying them.


Future: Ideally, these instructions would be made redundant by a GitHub Action or Cloud Build trigger that terraform applys Terraform merged into the main branch.

Interestingly, I have a similar issue on the GKE samples side: GoogleCloudPlatform/kubernetes-engine-samples#611

github-actions · 2024-03-14T18:35:22Z

🚲 PR staged at http://34.133.162.82

NimJay · 2024-03-14T18:36:10Z

.github/terraform/main.tf

+    bucket = "cicd-terraform-state"
+    prefix = "terraform-state"
+  }
+}


Heads-up: I stored the Terraform state in this Google Cloud Storage (GCS) bucket.

github-actions · 2024-03-14T18:40:34Z

🚲 PR staged at http://34.133.162.82

NimJay · 2024-03-14T18:43:45Z

.github/terraform/README.md

I've put everything inside the .github/terraform folder to match https://github.com/GoogleCloudPlatform/kubernetes-engine-samples.

bourgeoisor

LGTM!

bourgeoisor · 2024-03-14T19:07:53Z

.github/terraform/README.md

+    * Ideally, you would see `Apply complete! Resources: 0 added, 0 changed, 0 destroyed.` in the output.
+1. Make your desired changes to the Terraform code.
+1. Apply the Terraform: `terraform apply -var project_id=${PROJECT_ID}`
+    * This time, Terraform will prompt you confirm your changes before applying them.


Interestingly, I have a similar issue on the GKE samples side: GoogleCloudPlatform/kubernetes-engine-samples#611

bourgeoisor · 2024-03-14T19:09:34Z

.github/terraform/main.tf

@@ -0,0 +1,116 @@
+/**


This is such a great idea. I know we have an internal doc with what infra we need to recreate the CICD environment for Online Boutique, but it's nice to have it all codify and easily-reproducible in a TF script. Thanks!

Use new PR cluster for ci-pr.yaml

dbc3ebb

NimJay changed the title ~~Use new PR cluster for ci-pr.yaml~~ Use new GKE cluster for ci-pr.yaml Mar 13, 2024

Fix region in ci-pr.yaml

3d64b09

bourgeoisor approved these changes Mar 14, 2024

View reviewed changes

Update ci-main.yaml and cleanup.yaml

d7c9906

NimJay commented Mar 14, 2024

View reviewed changes

Add Terraform for prs-gke-cluster

ba9a76c

Add README for CI Terraform

178ab8d

NimJay commented Mar 14, 2024

View reviewed changes

Run terraform fmt

3619a29

NimJay commented Mar 14, 2024

View reviewed changes

NimJay marked this pull request as ready for review March 14, 2024 18:36

NimJay requested review from yoshi-approver and a team as code owners March 14, 2024 18:36

NimJay commented Mar 14, 2024

View reviewed changes

NimJay requested a review from bourgeoisor March 14, 2024 18:47

bourgeoisor approved these changes Mar 14, 2024

View reviewed changes

NimJay merged commit 1ef9480 into main Mar 14, 2024
10 checks passed

NimJay deleted the nimjay-ci branch March 14, 2024 20:59

NimJay mentioned this pull request Apr 24, 2024

Release/v0.10.0 #2507

Merged

5 tasks

bourgeoisor mentioned this pull request Apr 24, 2024

Recreate the release GKE cluster #2509

Open

NimJay mentioned this pull request Jul 3, 2024

Maintain the production GKE cluster using Terraform #2615

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use new GKE cluster for ci-pr.yaml #2418

Use new GKE cluster for ci-pr.yaml #2418

NimJay commented Mar 13, 2024 •

edited

Loading

github-actions bot commented Mar 13, 2024

bourgeoisor left a comment

github-actions bot commented Mar 14, 2024

NimJay Mar 14, 2024 •

edited

Loading

github-actions bot commented Mar 14, 2024

NimJay Mar 14, 2024 •

edited

Loading

bourgeoisor Mar 14, 2024

github-actions bot commented Mar 14, 2024

NimJay Mar 14, 2024 •

edited

Loading

github-actions bot commented Mar 14, 2024

NimJay Mar 14, 2024

bourgeoisor left a comment

bourgeoisor Mar 14, 2024

bourgeoisor Mar 14, 2024

Use new GKE cluster for ci-pr.yaml #2418

Use new GKE cluster for ci-pr.yaml #2418

Conversation

NimJay commented Mar 13, 2024 • edited Loading

Background

Changes from this PR

How to test

Additional info

github-actions bot commented Mar 13, 2024

bourgeoisor left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 14, 2024

NimJay Mar 14, 2024 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Mar 14, 2024

NimJay Mar 14, 2024 • edited Loading

Choose a reason for hiding this comment

bourgeoisor Mar 14, 2024

Choose a reason for hiding this comment

github-actions bot commented Mar 14, 2024

NimJay Mar 14, 2024 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Mar 14, 2024

NimJay Mar 14, 2024

Choose a reason for hiding this comment

bourgeoisor left a comment

Choose a reason for hiding this comment

bourgeoisor Mar 14, 2024

Choose a reason for hiding this comment

bourgeoisor Mar 14, 2024

Choose a reason for hiding this comment

NimJay commented Mar 13, 2024 •

edited

Loading

NimJay Mar 14, 2024 •

edited

Loading

NimJay Mar 14, 2024 •

edited

Loading

NimJay Mar 14, 2024 •

edited

Loading