feat: add volume attachment time & csi grpc execution time metrics #1142

iltyty · 2024-08-23T05:41:20Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot · 2024-08-23T05:41:26Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: iltyty
Once this PR has been reviewed and has the lgtm label, please assign mowangdk for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2024-08-23T05:41:30Z

Hi @iltyty. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

mowangdk · 2024-08-23T05:42:39Z

/ok-to-test

huww98 · 2024-09-24T03:59:19Z

pkg/common/interceptors.go

+	return fullName[i+1:]
+}
+
+func observeExecTime(time int64, method, driverType, volumeId string, err error) {


Suggested change

func observeExecTime(time int64, method, driverType, volumeId string, err error) {

func observeExecTime(time time.Duration, method, driverType, volumeId string, err error) {

Just pass in the Duration to avoid any ambiguous about unit.

huww98 · 2024-09-24T04:00:37Z

pkg/common/interceptors.go

+	case ControllerPublish, ControllerUnpublish, NodeStage, NodeUnstage, NodeUnpublish:
+		start := time.Now()
+		resp, err := handler(ctx, req)
+		execTime := time.Since(start).Milliseconds()


Suggested change

execTime := time.Since(start).Milliseconds()

execTime := time.Since(start).Seconds()

Use seconds as it already returns a float64.

huww98 · 2024-09-24T04:03:01Z

pkg/metric/csi_grpc_exec_time_collector.go

+func NewCsiGrpcExecTimeCollector() (Collector, error) {
+	return &CsiGrpcExecTimeCollector, nil
+}


This is confusing. It is named New but returns an existing object.

huww98 · 2024-09-24T04:04:16Z

pkg/metric/volume_stat_collector.go

+const (
+	VolumeStatsCollectorName = "volume_stats"
+	VolumeStatsLabelType     = "type"
+	VolumeStatsLabelId       = "id"


Remove ID as we discussed. We don't expect unbounded number of label values.

huww98 · 2024-09-24T04:59:05Z

pkg/common/interceptors.go

+	if !strings.Contains(fullName, "/") || i == len(fullName)-1 {
+		return ""
+	}
+	return fullName[i+1:]


I think this is not needed? If "/" is not in fullName, LastIndex will return -1, and fullName[i+1:] should become the fullName.
If i == len(fullName)-1, fullName[i+1:] will become empty string automatically.

huww98 · 2024-09-24T05:03:08Z

pkg/common/interceptors.go

+	return codes.Unknown
+}
+
+func observeVolumeAttachmentTime(curTime int64, req *csi.NodePublishVolumeRequest, driverType string, err error) {


Suggested change

func observeVolumeAttachmentTime(curTime int64, req *csi.NodePublishVolumeRequest, driverType string, err error) {

func observeVolumeAttachmentTime(curTime time.Time, req *csi.NodePublishVolumeRequest, driverType string, err error) {

Again, to avoid ambiguity.

huww98 · 2024-09-24T05:05:11Z

pkg/common/interceptors.go

+
+func observeVolumeAttachmentTime(curTime int64, req *csi.NodePublishVolumeRequest, driverType string, err error) {
+	if client == nil {
+		if client = getK8sClient(); client == nil {


This is not thread safe. And please try to avoid global variable.

huww98 · 2024-09-24T05:08:03Z

pkg/common/interceptors.go

+		log.Warnf("observeVolumeAttachmentTime: empty pod name/namespace: %s, %s", podName, podNamespace)
+		return
+	}
+	pod, err := client.CoreV1().Pods(podNamespace).Get(context.Background(), podName, metav1.GetOptions{})


Additional API server invocation. This can hinder the overall throughput of CSI. We should try to avoid this. Combine it to the runtime class detection?

huww98 · 2024-09-24T05:16:10Z

pkg/common/interceptors.go

+		execTime := time.Since(start).Milliseconds()
+		observeExecTime(execTime, method, driverType, volumeId, err)
+		return resp, err
+	case NodePublish:


How about add this one to a real NodePublishVolume handler? We should avoid string comparisons and prefer strongly typed handler generally.

k8s-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Aug 23, 2024

k8s-ci-robot requested a review from huww98 August 23, 2024 05:41

k8s-ci-robot requested a review from mowangdk August 23, 2024 05:41

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 23, 2024

iltyty marked this pull request as draft August 23, 2024 05:41

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 23, 2024

iltyty force-pushed the feat/volume-new-metrics branch from fcc7310 to 438034b Compare August 29, 2024 08:35

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 29, 2024

iltyty force-pushed the feat/volume-new-metrics branch from 249b43d to 027da89 Compare September 9, 2024 02:11

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 9, 2024

iltyty marked this pull request as ready for review September 9, 2024 02:32

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 9, 2024

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 23, 2024

iltyty added 4 commits September 24, 2024 10:37

feat: add volume attachment time & csi grpc execution time metrics

f364a01

chore: change observe time unit to seconds

b851bdb

refactor: instrument grpc calls with interceptors

f8f48d8

fix: incorrect NodePublish/NodeUnpublish method name

da7c531

huww98 reviewed Sep 24, 2024

View reviewed changes

iltyty added 2 commits September 26, 2024 14:24

feat: change metric types to Counter

1f6701a

test: add interceptor tests

4734f6d

iltyty force-pushed the feat/volume-new-metrics branch from e7fb58d to 4734f6d Compare September 26, 2024 07:15

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add volume attachment time & csi grpc execution time metrics #1142

feat: add volume attachment time & csi grpc execution time metrics #1142

iltyty commented Aug 23, 2024

k8s-ci-robot commented Aug 23, 2024

k8s-ci-robot commented Aug 23, 2024

mowangdk commented Aug 23, 2024

huww98 Sep 24, 2024

huww98 Sep 24, 2024

huww98 Sep 24, 2024

huww98 Sep 24, 2024

huww98 Sep 24, 2024

huww98 Sep 24, 2024

huww98 Sep 24, 2024

huww98 Sep 24, 2024

huww98 Sep 24, 2024

	func observeExecTime(time int64, method, driverType, volumeId string, err error) {
	func observeExecTime(time time.Duration, method, driverType, volumeId string, err error) {

	execTime := time.Since(start).Milliseconds()
	execTime := time.Since(start).Seconds()

	func observeVolumeAttachmentTime(curTime int64, req *csi.NodePublishVolumeRequest, driverType string, err error) {
	func observeVolumeAttachmentTime(curTime time.Time, req *csi.NodePublishVolumeRequest, driverType string, err error) {

feat: add volume attachment time & csi grpc execution time metrics #1142

Are you sure you want to change the base?

feat: add volume attachment time & csi grpc execution time metrics #1142

Conversation

iltyty commented Aug 23, 2024

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot commented Aug 23, 2024

k8s-ci-robot commented Aug 23, 2024

mowangdk commented Aug 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment