DontReview Garnitin/add gke load testing/v1 #2225

gargnitingoogle · 2024-07-26T21:27:18Z

Description

Link to the issue in case of a bug fix.

NA

Testing details

Manual - NA
Unit tests - NA
Integration tests - NA

codecov · 2024-07-26T21:32:04Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.36%. Comparing base (3b6804a) to head (01c1a06).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2225      +/-   ##
==========================================
- Coverage   78.38%   78.36%   -0.03%     
==========================================
  Files         107      107              
  Lines       11793    11793              
==========================================
- Hits         9244     9241       -3     
- Misses       2058     2060       +2     
- Partials      491      492       +1

Flag	Coverage Δ
unittests	`78.36% <ø> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

gargnitingoogle · 2024-07-26T21:29:20Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+fi
+
+function validateMachineConfig() {
+  echo "Validiting input parameters ..."


gargnitingoogle · 2024-07-26T21:29:52Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+  local cluster_name=${1}
+  local zone=${2}
+  local node_pool=${3}
+  if [ $(gcloud container node-pools list --cluster=${cluster_name} --zone=${zone} | grep -ow ${node_pool} | wc -l) -gt 0 ] ; then


simplify this with grep -q

gargnitingoogle · 2024-07-26T21:30:59Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+  if ClusterExists ${cluster_name} ; then
+    gcloud container clusters update ${cluster_name} --location=${zone} --workload-pool=${project_id}.svc.id.goog
+  else
+    # gcloud container --project "${project_id}" clusters create ${cluster_name} --zone "${zone}" --cluster-version "${cluster_version}" --workload-pool=${project_id}.svc.id.goog


remove commented code

gargnitingoogle · 2024-07-26T21:32:10Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+  echo "Enabling/disabling csi add-on ..."
+  # By default, disable the managed csi driver.
+  if ${useCustomCsiDriver}; then
+    # gcloud -q container clusters update ${cluster_name} \


re-enable this

gargnitingoogle · 2024-07-26T21:32:29Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+  gcloud container clusters get-credentials ${cluster_name} --location=${zone}
+  kubectl create namespace ${appnamespace}
+  kubectl create serviceaccount ${ksa} --namespace ${appnamespace}
+  for workload_bucket in ${buckets} ; do


add code to get buckets from somewhere.

gargnitingoogle · 2024-07-26T21:35:00Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+}
+
+# validateMachineConfig ${machine_type} ${num_nodes} ${num_ssd}
+# installDependencies


re-enable all these disabled steps.

gargnitingoogle · 2024-07-26T21:36:51Z

perfmetrics/scripts/testing_on_gke/utils/utils.py

+def get_cpu(pod_name: str, start: str, end: str) -> Tuple[float, float]:
+    # for some reason, the mash filter does not always work, so we fetch all the metrics for all the pods and filter later.
+    result = subprocess.run(["mash", "--namespace=cloud_prod", "--output=csv", 
+                             f"Query(Fetch(Raw('cloud.kubernetes.K8sContainer', 'kubernetes.io/container/cpu/core_usage_time'), {{'project': '927584127901'}})| Window(Rate('10m'))| GroupBy(['pod_name', 'container_name'], Max()), TimeInterval('{start}', '{end}'), '5s')"], 


put the project-number back to the original and change it by code during runtime and then revert it back when done during runtime.

gargnitingoogle · 2024-07-26T21:37:31Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+function updateMachineTypeInPodConfigs() {
+  for file in ${gke_testing_dir}/examples/fio/loading-test/values.yaml ${gke_testing_dir}/examples/dlio/unet3d-loading-test/values.yaml ; do
+    test -f ${file}
+    sed -i -E "s/nodeType: [0-9a-z_-]+$/nodeType: ${machine_type}/g" ${file}


add code to revert this back

gargnitingoogle · 2024-07-26T21:37:42Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

+  for file in ${gke_testing_dir}/examples/fio/loading-test/values.yaml ${gke_testing_dir}/examples/dlio/unet3d-loading-test/values.yaml ; do
+    test -f ${file}
+    # sed -i -E "s/mountOptions: [0-9a-zA-Z,\:\"-_]+$/mountOptions: \"${gcsfuse_mount_options}\"/g" ${file}
+    sed -i -E "s/mountOptions:[ \t]*\"?[a-zA-Z0-9,:_-]+\"? *$/mountOptions: \"${gcsfuse_mount_options}\"/g" "${file}"


add code to revert this back when done

gargnitingoogle · 2024-07-27T04:27:38Z

perfmetrics/scripts/testing_on_gke/utils/utils.py

+    return utc_timestamp_string
+
+def standard_timestamp(timestamp: int) -> str:
+    return timestamp.split('.')[0].replace('T', ' ') + " UTC"


insert newline after this line

gargnitingoogle · 2024-07-27T04:28:16Z

perfmetrics/scripts/testing_on_gke/examples/run-gke-tests.sh

@@ -0,0 +1,425 @@
+#!/bin/bash
+


Add a header and a help option at the top.

Option: force_update_gcsfuse_code Default: false This applies when $gcsfuse_src_dir has been passed, or when $src_dir/gcsfuse already exists.

Adds logs for missing sudoless-docker, helpful instruction to install it, and then to re-run the script.

This reverts commit 6fa2ad2. Reverting this because the above commit exposed PII in form of the gcsfuse-internal project IDs and numbers.

The run-gke-tests script should fail if exactly one out of project_id and project_number has been set. It will error out in such as a case.

Add 'Error: ' prefix in all the error logs for easy spotting in log files.

Utilities: 1. Append given tabular data to the given gsheet id and worksheet name. 2. Return url for a gsheet given its ID. 3. Adds unit tests for the above utilities.

add generic utility to append to a gsheet Add utility for gsheet improve gsheet utility export fio output to gsheet encapsulate cpu/memory calculation in fio disable repeat operations for quick testing add dlio output export to gsheet fix a bug in dlio output parsing fix a column-name in fio csv output Revert "disable repeat operations for quick testing" This reverts commit 04bf834. add log of successful addition to gsheet clean-up code changes added some error-checking wip wip fix calls to download_gcs_objects support key-file on gcs in gsheet put back cpu/memory metrics fix couple of logs fix couple of help messages put back accidentally deleted command

Purposes. * Consistent behavior across all machines * Monitoring API has faster runtime than mash. * Monitoring API is supported on GCE VM too.

Adds row with "ERROR" for all values rather than crashing during printing in CSV file.

gargnitingoogle commented Jul 26, 2024

View reviewed changes

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from 2958a3c to ad97f82 Compare July 29, 2024 10:47

gargnitingoogle commented Jul 29, 2024

View reviewed changes

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch 7 times, most recently from ff7ee55 to 478921c Compare August 6, 2024 09:53

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch 9 times, most recently from 1e4353a to b6a0e76 Compare August 16, 2024 14:45

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch 9 times, most recently from b62b517 to d008a84 Compare August 22, 2024 11:53

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from 0daf723 to 6dbb552 Compare August 26, 2024 09:05

gargnitingoogle added 18 commits October 7, 2024 10:10

unique AR image-name for each run

8c5ccc9

handle pod status error

a91b1c6

make use_custom_csi_driver=true by default

681ff47

remove a redundant project_id setting in a gcloud command

7ad8b48

Add option to force-update existing gcsfuse code

aa477a4

Option: force_update_gcsfuse_code Default: false This applies when $gcsfuse_src_dir has been passed, or when $src_dir/gcsfuse already exists.

fix implementation of force_fetch_gcsfuse

f29cca8

improve debug log

279b4af

Add check for missing sudoless-docker

c0d6bd8

Adds logs for missing sudoless-docker, helpful instruction to install it, and then to re-run the script.

add some error checks in run-script

4526bc2

formatting in run-script

9e31426

Revert "Add project id<->number autoconversion"

c91ad6a

This reverts commit 6fa2ad2. Reverting this because the above commit exposed PII in form of the gcsfuse-internal project IDs and numbers.

Tie project_id and project_number together

6ba5f1e

The run-gke-tests script should fail if exactly one out of project_id and project_number has been set. It will error out in such as a case.

Improve error-logs in cli-script

16245b8

Add 'Error: ' prefix in all the error logs for easy spotting in log files.

fix couple of comments

bde8a24

Add gsheet utilities

425e209

Utilities: 1. Append given tabular data to the given gsheet id and worksheet name. 2. Return url for a gsheet given its ID. 3. Adds unit tests for the above utilities.

fix gsheet unit tests

e335c63

address self-review comments

09020e2

address some more self-review comments

290ffb0

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch 2 times, most recently from 01c1a06 to d48baa9 Compare October 7, 2024 10:26

gargnitingoogle added 9 commits October 7, 2024 10:29

address a self-review comment

13aeace

rename some utilities

d8d55e1

address some self-review comments

80a4844

Replace mash with monitoring-api everywhere

4c7ecfc

Purposes. * Consistent behavior across all machines * Monitoring API has faster runtime than mash. * Monitoring API is supported on GCE VM too.

fix some var names

ed644a8

empty commit

ddae281

Add try-catch for exception during parsing

1e975f4

Adds row with "ERROR" for all values rather than crashing during printing in CSV file.

fix log in run-automated

bc8f62a

gargnitingoogle force-pushed the garnitin/add-gke-load-testing/v1 branch from d48baa9 to bc8f62a Compare October 7, 2024 10:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DontReview Garnitin/add gke load testing/v1 #2225

DontReview Garnitin/add gke load testing/v1 #2225

gargnitingoogle commented Jul 26, 2024

codecov bot commented Jul 26, 2024 •

edited

Loading

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 26, 2024

gargnitingoogle Jul 27, 2024

gargnitingoogle Jul 27, 2024

DontReview Garnitin/add gke load testing/v1 #2225

Are you sure you want to change the base?

DontReview Garnitin/add gke load testing/v1 #2225

Conversation

gargnitingoogle commented Jul 26, 2024

Description

Link to the issue in case of a bug fix.

Testing details

codecov bot commented Jul 26, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jul 26, 2024 •

edited

Loading