Add test for testrgrid alert emails on release-blocking jobs

[hasheddan]

Adds a check that release-blocking jobs have testgrid alert emails. Logs
violations for now, but should be updated to error on violations when
all jobs pass.

Signed-off-by: hasheddan [email protected]

xref: kubernetes/sig-release#1216 #19844 (comment)

/assign @justaugustus @cpanato

[hasheddan]

/cc @kubernetes/ci-signal

[RobertKielty]

/lgtm

[hasheddan]

/assign @spiffxp

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chases2, hasheddan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• config/OWNERS [chases2]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[cpanato]

Thanks for this @hasheddan

[spiffxp]

Sorry I missed this. We should revert this, because there is already a test that does this, and catches more corner cases. It is possible to configure test grid alerts outside of prowjob files, so just checking prow job annotations is insufficient.

Instead, we should build on this test:

test-infra/config/tests/testgrids/config_test.go

Lines 506 to 544 in 2293860

	func TestReleaseBlockingJobsMustHaveTestgridDescriptions(t *testing.T) {
	// TODO(spiffxp): start with master, enforce for all release branches
	re := regexp.MustCompile("^sig-release-master-(blocking|informing)$")
	for _, dashboard := range cfg.Dashboards {
	if !re.MatchString(dashboard.Name) {
	continue
	}
	suffix := re.FindStringSubmatch(dashboard.Name)[1]
	for _, dashboardtab := range dashboard.DashboardTab {
	intro := fmt.Sprintf("dashboard_tab %v/%v is release-%v", dashboard.Name, dashboardtab.Name, suffix)
	if dashboardtab.Name == "" {
	t.Errorf("%v: - Must have a name", intro)
	}
	if dashboardtab.TestGroupName == "" {
	t.Errorf("%v: - Must have a test_group_name", intro)
	}
	if dashboardtab.Description == "" {
	t.Errorf("%v: - Must have a description", intro)
	}
	// TODO(spiffxp): enforce for informing as well
	if suffix == "informing" {
	if !strings.HasPrefix(dashboardtab.Description, "OWNER: ") {
	t.Logf("NOTICE: %v: - Must have a description that starts with OWNER: ", intro)
	}
	if dashboardtab.AlertOptions == nil {
	t.Logf("NOTICE: %v: - Must have alert_options (ensure informing dashboard is listed first in testgrid-dashboards)", intro)
	} else if dashboardtab.AlertOptions.AlertMailToAddresses == "" {
	t.Logf("NOTICE: %v: - Must have alert_options.alert_mail_to_addresses", intro)
	}
	} else {
	if dashboardtab.AlertOptions == nil {
	t.Errorf("%v: - Must have alert_options (ensure blocking dashboard is listed first in testgrid-dashboards)", intro)
	} else if dashboardtab.AlertOptions.AlertMailToAddresses == "" {
	t.Errorf("%v: - Must have alert_options.alert_mail_to_addresses", intro)
	}
	}
	}
	}
	}

It requires a testgrid.proto file to be generated (using test grid files and prow job files as source), and then looks at the test grid options. If you invoke this with bazel, that proto file is generated automatically (see #18599 (comment))

[hasheddan]

@spiffxp thanks for the info and apologies for missing this existing test. I'll revert and then update the existing 👍