cip-auditor: alerts are noisy

[spiffxp]

Tracking issue for the fact that CIP auditor alerts are noisy.

The alerting was manually created by thockin a while ago (could not find an issue to link at a glance, maybe someone else knows). Then it was manually disabled because the alerts were perceived as noisy.

The alerting is managed via click-ops, ideally it could be done with gcloud: #1624

IAM policies to allow viewing of incidents and alerts without granting admin access to k8s-artifacts prod would be helpful. What's the appropriate group to grant this access to?

/cc @listx
FYI @tylerferrara
Making this tracking issue since I'm about to make a manual IAM change and I want to document it somewhere

[spiffxp]

/priority important-longterm
/wg k8s-infra
/sig release
/area artifacts
/area release-eng
/area infra/monitoring
/milestone v1.22

[spiffxp]

FWIW here is the policy generating the alert in question

# $ gcloud alpha monitoring policies list --project=kubernetes-public
---
combiner: OR
conditions:
- conditionThreshold:
    aggregations:
    - alignmentPeriod: 60s
      crossSeriesReducer: REDUCE_MAX
      perSeriesAligner: ALIGN_RATE
    comparison: COMPARISON_GT
    duration: 0s
    filter: metric.type="logging.googleapis.com/user/cip-auditor-alert"
    trigger:
      count: 1
  displayName: logging/user/cip-auditor-alert
  name: projects/kubernetes-public/alertPolicies/12089518252001301876/conditions/12089518252001300531
creationRecord:
  mutateTime: '2020-03-31T00:27:12.391234443Z'
  mutatedBy: thockin@REDACTED
displayName: Image promoter alert
documentation:
  content: The image promoter has logged something that we consider an alert.
  mimeType: text/markdown
enabled: true
mutationRecord:
  mutateTime: '2020-03-31T01:45:19.410085124Z'
  mutatedBy: thockin@REDACTED
name: projects/kubernetes-public/alertPolicies/12089518252001301876
notificationChannels:
- projects/kubernetes-public/notificationChannels/7630148271419930225
- projects/kubernetes-public/notificationChannels/17367851054639804370
- projects/kubernetes-public/notificationChannels/2533614711603005061
- projects/kubernetes-public/notificationChannels/7846745591716920888

[tylerferrara]

/assign

[spiffxp]

members of k8s-infra-gcp-auditors should be able to do what I just did above:

k8s.io/infra/gcp/roles/audit.viewer.yaml

Lines 882 to 883 in 15f7d7c

	- monitoring.alertPolicies.get
	- monitoring.alertPolicies.list

[spiffxp]

This is what IAM is for the project in question right now: https://github.com/kubernetes/k8s.io/blob/main/audit/projects/kubernetes-public/iam.json

I could be convinced that k8s-infra-cluster-admins should get something like roles/monitoring.editor or higher on this project, as the nearest conveniently existing group that has editor/admin type roles on this project

I could also be convinced there should be a group dedicated to CIP related infrastructure that gets more granular IAM permissions

[spiffxp]

@tylerferrara is unable to view incidents because of lack of roles/monitoring.viewer, ref: https://cloud.google.com/monitoring/alerts/troubleshooting-alerts#no-permission

The only existing group that has this is gke-security-groups:

k8s.io/audit/projects/kubernetes-public/iam.json

Lines 117 to 122 in 15f7d7c

	"members": [
	"group:[email protected]",
	"serviceAccount:[email protected]",
	"serviceAccount:k8s-infra-monitoring-viewer@kubernetes-public.iam.gserviceaccount.com"
	],
	"role": "roles/monitoring.viewer"

Which is populated entirely by app-specific rbac groups:

k8s.io/groups/groups.yaml

Lines 172 to 194 in 15f7d7c

	# Every RBAC group should be added here.
	- email-id: [email protected]
	name: gke-security-groups
	description: |-
	Security Groups for GKE clusters
	settings:
	ReconcileMembers: "true"
	WhoCanViewMembership: "ALL_MEMBERS_CAN_VIEW" # needed for RBAC
	members:
	- [email protected]
	- [email protected]
	- [email protected]
	- [email protected]
	- [email protected]
	- [email protected]
	- [email protected]
	- [email protected]
	- [email protected]
	- [email protected]
	- [email protected]
	- [email protected]
	- [email protected]
	- [email protected]

Rather than untangle the right place to put them, since I'm about to go AFK, I'm going to add @tylerferrara manually

$ gcloud projects add-iam-policy-binding kubernetes-public --role=roles/monitoring.viewer --member=user:[email protected]
Updated IAM policy for project [kubernetes-public].

[tylerferrara]

Incidents are now viewable!

With respect to the auditor logs, tracking down these incident triggers requires looking at stack traces. I'm still unable to view anything from the "Traces" service in GCP for the k8s-artifacts-prod project.

[spiffxp]

#2365 added the necessary cloudtrace read-only permissions to audit.viewer which is granted org-wide to k8s-infra-gcp-auditors@, which @tylerferrara is already a member of.

[tylerferrara]

The PR: #2366 has stopped the audit crashes, which was causing the alerts. However, this issue should remain open until the following investigation has been resolved (CIP issue: #353).

[spiffxp]

/milestone v1.23

[spiffxp]

/remove-priority important-longterm
/priority important-soon

[spiffxp]

@kubernetes/release-engineering. I am not sure where we stand on this anymore. Has the fix been:

• identified?
• implemented?
• deployed?

[ameukam]

/lifecycle frozen
/milestone clear