Added summary command to kompanion #3180

cheftako · 2024-11-14T19:01:30Z

kompanion summary walks the relevant cluster inspecting cnrm resources. It will sum the count of resources by namespace, type and status.

Change description

Sample usage

$ ./kompanion summary
2024/11/14 18:51:25 Running kompanion summary with kubeconfig:
2024/11/14 18:51:25 Starting worker threads to process Config Connector objects
2024/11/14 18:51:25 Dumping Config Connector summaries
2024/11/14 18:51:33 Namespace: e2e-655cda
2024/11/14 18:51:33 - GroupKind: iamserviceaccounts.iam.cnrm.cloud.google.com (True: 50)
2024/11/14 18:51:33 - GroupKind: storagebuckets.storage.cnrm.cloud.google.com (True: 100)
2024/11/14 18:51:33 - GroupKind: iampartialpolicies.iam.cnrm.cloud.google.com (True: 152)
2024/11/14 18:51:33 - GroupKind: configconnectorcontexts.core.cnrm.cloud.google.com (no status: 1)
2024/11/14 18:51:33 - GroupKind: containerclusters.container.cnrm.cloud.google.com (True: 1)
2024/11/14 18:51:33 - GroupKind: sqldatabases.sql.cnrm.cloud.google.com (True: 45)
2024/11/14 18:51:33 - GroupKind: sqlusers.sql.cnrm.cloud.google.com (True: 45)
2024/11/14 18:51:33 - GroupKind: containernodepools.container.cnrm.cloud.google.com (True: 4)
2024/11/14 18:51:33 - GroupKind: sqlinstances.sql.cnrm.cloud.google.com (True: 9)
...
2024/11/14 18:51:33 Namespace: e2e-0704d0
2024/11/14 18:51:33 - GroupKind: configconnectorcontexts.core.cnrm.cloud.google.com (no status: 1)
2024/11/14 18:51:33 - GroupKind: containerclusters.container.cnrm.cloud.google.com (True: 1)
2024/11/14 18:51:33 - GroupKind: containernodepools.container.cnrm.cloud.google.com (True: 4)
2024/11/14 18:51:33 - GroupKind: storagebuckets.storage.cnrm.cloud.google.com (True: 100)
2024/11/14 18:51:33 - GroupKind: iampartialpolicies.iam.cnrm.cloud.google.com (True: 152)
2024/11/14 18:51:33 - GroupKind: sqlinstances.sql.cnrm.cloud.google.com (True: 9)
2024/11/14 18:51:33 - GroupKind: iamserviceaccounts.iam.cnrm.cloud.google.com (True: 50)
2024/11/14 18:51:33 - GroupKind: sqldatabases.sql.cnrm.cloud.google.com (True: 45)
2024/11/14 18:51:33 - GroupKind: sqlusers.sql.cnrm.cloud.google.com (True: 45)
2024/11/14 18:51:33 Totals:
2024/11/14 18:51:33 - GroupKind: storagebuckets.storage.cnrm.cloud.google.com (True: 2200)
2024/11/14 18:51:33 - GroupKind: containerclusters.container.cnrm.cloud.google.com (False: 26, True: 23)
2024/11/14 18:51:33 - GroupKind: iampolicymembers.iam.cnrm.cloud.google.com (True: 69)
2024/11/14 18:51:33 - GroupKind: sqlusers.sql.cnrm.cloud.google.com (False: 80, True: 990)
2024/11/14 18:51:33 - GroupKind: iamserviceaccounts.iam.cnrm.cloud.google.com (True: 1123)
2024/11/14 18:51:33 - GroupKind: sqldatabases.sql.cnrm.cloud.google.com (False: 49, True: 1021)
2024/11/14 18:51:33 - GroupKind: iampartialpolicies.iam.cnrm.cloud.google.com (True: 3480, False: 58)
2024/11/14 18:51:33 - GroupKind: configconnectorcontexts.core.cnrm.cloud.google.com (no status: 50)
2024/11/14 18:51:33 - GroupKind: containernodepools.container.cnrm.cloud.google.com (True: 92, False: 104)
2024/11/14 18:51:33 - GroupKind: sqlinstances.sql.cnrm.cloud.google.com (False: 16, True: 198)

Fixes #

Tests you have done

Run make ready-pr to ensure this PR is ready for review.
Perform necessary E2E testing for changed resources.

acpana

Overall LGTM! Just some questions.

/lgtm

experiments/kompanion/pkg/utils/utils.go

acpana · 2024-11-15T08:06:11Z

experiments/kompanion/pkg/utils/utils.go

I like this! Can we create a pkg/discovery instead of utils?

We could but unless we anticipate adding a lot more than just 1 method for discovery this doesn't seem like its own package. Might make sense to move it from utils.go to discovery.go.

acpana · 2024-11-15T08:07:15Z

experiments/kompanion/cmd/summary/summary.go

+	return val, nil
+}
+
+func shouldExclude(name string, excludes []string, includes []string) bool {


hehe we can probably factor some of these out but non blocking

acpana · 2024-11-15T08:10:00Z

experiments/kompanion/cmd/summary/summary.go

+	return cmd
+}
+
+type Result struct {


I think this is fine for now but adding some types around the Map[gk, Map[status, count]]] might make this easier to read.

Is it possible to represent this as a 3-uple instead? [<NS,GK,status> => count] ? would this help simplify increment?

Its possible but that then makes other things harder. If you want to count objects in a namespace you now have to walk the entire thrupple map and filter then accrue, rather than just walking the given namespace map and accruing.

acpana · 2024-11-15T08:13:54Z

experiments/kompanion/cmd/summary/options.go

+const (
+	// flag names.
+	kubeconfigFlag       = "kubeconfig"
+	reportNamePrefixFlag = "report-prefix"
+
+	targetNamespacesFlag = "target-namespaces"
+	ignoreNamespacesFlag = "exclude-namespaces"
+
+	targetObjectsFlag = "target-objects"
+
+	workerRoutinesFlag = "worker-routines"
+)


might be nice to make some of these common options; can be added as a follow on!

acpana · 2024-11-15T08:17:50Z

experiments/kompanion/cmd/summary/summary.go

+			if err != nil {
+				log.Printf("Got error retrieving status of type gvr %s, %v", gvr, err)
+			}
+			gk := gvr.Resource + "." + gvr.Group


wait I am confused -- is this actually the group kind then ?

Group and Kind should uniquely identifies a type of resource. Version identifies which (of several) access mechanisms you want to use to get that resource. If we include the gvr we will can end up multi-counting the resource. We should probably actually add code in utils.GetResources() to defend against that. I don't believe we are seeing instances of that today because we generally remove the v1alphaX when we add the v1betaY. Also we don't currently have any V1s. However we probably should add some protections to GetResources().

acpana · 2024-11-15T08:23:31Z

experiments/kompanion/cmd/summary/summary.go

+	gkMap[status] = count + 1
+}
+
+func (r *Result) Print() {


I think this format is fine but it may be nice to support tabulation, json or other formats in the future.

+1. Even just a CSV format. See if its popular and if it is how users would like to consumer the results.

kompanion summary walks the relevant cluster inspecting cnrm resources. It will sum the count of resources by namespace, type and status. Fix date on the copyright.

yuwenma · 2024-11-18T17:12:20Z

/approve

google-oss-prow · 2024-11-18T17:12:32Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: yuwenma

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~experiments/OWNERS~~ [yuwenma]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

yuwenma · 2024-11-18T17:14:33Z

/lgtm

Cool. Thanks!

cheftako requested review from acpana and justinsb November 14, 2024 19:01

acpana reviewed Nov 15, 2024

View reviewed changes

google-oss-prow bot assigned acpana Nov 15, 2024

google-oss-prow bot added the lgtm label Nov 15, 2024

Added summary command to kompanion

66afa04

kompanion summary walks the relevant cluster inspecting cnrm resources. It will sum the count of resources by namespace, type and status. Fix date on the copyright.

cheftako force-pushed the kompanion branch from 4895664 to 66afa04 Compare November 17, 2024 20:52

google-oss-prow bot removed the lgtm label Nov 17, 2024

google-oss-prow bot added the approved label Nov 18, 2024

google-oss-prow bot assigned yuwenma Nov 18, 2024

google-oss-prow bot added the lgtm label Nov 18, 2024

google-oss-prow bot merged commit c86a410 into GoogleCloudPlatform:master Nov 18, 2024
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added summary command to kompanion #3180

Added summary command to kompanion #3180

cheftako commented Nov 14, 2024

acpana left a comment

acpana Nov 15, 2024

cheftako Nov 16, 2024

acpana Nov 15, 2024

cheftako Nov 16, 2024

acpana Nov 15, 2024

acpana Nov 15, 2024

cheftako Nov 16, 2024

acpana Nov 15, 2024

cheftako Nov 16, 2024

acpana Nov 15, 2024

cheftako Nov 16, 2024

acpana Nov 15, 2024

cheftako Nov 16, 2024

yuwenma commented Nov 18, 2024

google-oss-prow bot commented Nov 18, 2024

yuwenma commented Nov 18, 2024

Added summary command to kompanion #3180

Added summary command to kompanion #3180

Conversation

cheftako commented Nov 14, 2024

Change description

Tests you have done

acpana left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuwenma commented Nov 18, 2024

google-oss-prow bot commented Nov 18, 2024

yuwenma commented Nov 18, 2024