added valImage to monitor image vulnerabilities #1175

yt3liu · 2019-07-23T17:42:30Z

What this PR does, why we need it:

generalized the pubsub client in monitoring
moved test-infra monitoring specific stuff to alert.go
added valImage to listen for image vulnerabilities message, send an email for all messages received

This PR does NOT create an issue in the relevant repo. This is due to when testing in my own gcp project, I didn't see anything that gives us repo specific information. Thus, for now, only send an email when receives the image vulnerability message.

Sample Email Message:

Message Data: {"name":"projects/<project-name>/occurrences/<uuid>","kind":"VULNERABILITY","notificationTime":"2019-07-23T16:54:30.902457Z"}

Pubsub Message: {
	"ID": "626966346998275",
	"Data": "<base64encoded content of the above message data>",
	"Attributes": null,
	"PublishTime": "2019-07-23T16:54:31.025Z"
}

Raw Message: &{ID:xxxxxxxxxxxxxxxxxx Data:[...] Attributes:map[] ackID:xxxxxxxxxx PublishTime:2019-07-23 16:54:31.025 +0000 UTC receiveTime:{wall:13782682632966708414 ext:29034076097 loc:0xdd5f40} size:0 calledDone:false doneFunc:0x8115e0}

Which issue(s) this PR fixes:
Part of knative/infra#136

Special notes to reviewers:

User-visible changes in this PR:

knative-prow-robot

@yt3liu: 0 warnings.

In response to this:

What this PR does, why we need it:

generalized the pubsub client in monitoring

moved test-infra monitoring specific stuff to alert.go

added valImage to listen for image vulnerabilities message, send an email for all messages received

This PR does NOT create an issue in the relevant repo. This is due to when testing in my own gcp project, I didn't see anything that gives us repo specific information. Thus, for now, only send an email when receives the image vulnerability message.

Sample Email Message:
Message Data: {"name":"projects/<project-name>/occurrences/<uuid>","kind":"VULNERABILITY","notificationTime":"2019-07-23T16:54:30.902457Z"}

Pubsub Message: {
  "ID": "626966346998275",
  "Data": "<base64encoded content of the above message data>",
  "Attributes": null,
  "PublishTime": "2019-07-23T16:54:31.025Z"
}

Raw Message: &{ID:xxxxxxxxxxxxxxxxxx Data:[...] Attributes:map[] ackID:xxxxxxxxxx PublishTime:2019-07-23 16:54:31.025 +0000 UTC receiveTime:{wall:13782682632966708414 ext:29034076097 loc:0xdd5f40} size:0 calledDone:false doneFunc:0x8115e0}
Which issue(s) this PR fixes:
Part of https://github.com/knative/test-infra/issues/678

Special notes to reviewers:

User-visible changes in this PR:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

tools/monitoring/main.go

tools/monitoring/valimage/valImage.go

tools/validate-image/main.go

tools/validate-image/valImage.go

chaodaiG · 2019-09-13T14:13:08Z

/approve

knative-prow-robot · 2019-09-13T14:13:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: chaodaiG, yt3liu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [chaodaiG]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

srinivashegde86 · 2019-09-16T19:05:11Z

images/validate-image/Dockerfile

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM golang:1.12.1


FROM golang:latest?

Didn't know this was an option. Will try this out :)

srinivashegde86 · 2019-09-16T19:07:44Z

tools/validate-image/main.go

+
+	mailConfig, err := mail.NewMailConfig(*mailAddrSF, *mailPassSF)
+	if err != nil {
+		log.Fatal(err)


Suggested change

log.Fatal(err)

log.Fatalf("Failed to create mail config: %v", err)

srinivashegde86 · 2019-09-16T19:08:49Z

tools/validate-image/main.go

+
+func validateImage(w http.ResponseWriter, r *http.Request) {
+	log.Printf("Serving request: %s", r.URL.Path)
+


remove blank line

srinivashegde86 · 2019-09-16T19:11:47Z

tools/validate-image/valImage.go

+	go func() {
+		err := subClient.Receive(context.Background(), func(ctx context.Context, msg *pubsub.Message) {
+			log.Printf("Message: %v\n", string(msg.Data))
+			log.Printf("Pubsub Message: %v\n", msg)


whats the point of logging both? This will log the data as well?

The one logged with string(msg.Data) prints data in a human-readable format, while printing the message directly from msg prints the message as base64encoded string.

I'd like to keep the pubsub message for debugging purposes in case there's additional attributes that gets added to the image vulnerabilities message.

# log.Printf("Message: %v\n", string(msg.Data)) Message Data: {"name":"projects/joyceyu-test/occurrences/ce3b5a33-05ed-40c2-80f2-821d38426171","kind":"DISCOVERY","notificationTime":"2019-09-12T18:04:36.630196Z"} # log.Printf("Pubsub Message: %v\n", msg) Pubsub Message: { "ID": "719306958870977", "Data": "eyJuYW1lIjoicHJvamVjdHMvam95Y2V5dS10ZXN0L29jY3VycmVuY2VzL2NlM2I1YTMzLTA1ZWQtNDBjMi04MGYyLTgyMWQzODQyNjE3MSIsImtpbmQiOiJESVNDT1ZFUlkiLCJub3RpZmljYXRpb25UaW1lIjoiMjAxOS0wOS0xMlQxODowNDozNi42MzAxOTZaIn0=", "Attributes": null, "PublishTime": "2019-09-12T18:04:37.058Z" }

srinivashegde86 · 2019-09-16T19:12:35Z

tools/validate-image/valImage.go

+
+	return &Client{
+		subClients: subClients,
+		mailClient: mconfig,


we should validate mConfig != nil

srinivashegde86 · 2019-09-16T19:13:17Z

tools/validate-image/valImage.go

+			log.Printf("Pubsub Message: %v\n", msg)
+
+			if time.Now().Sub(lastSent) > alertFreq {
+				err := c.mailClient.Send(recipients, "Image Vulnerabilities Detected", toMailContent(msg))


combine if
if err := ...; err != nil {}

srinivashegde86 · 2019-09-16T19:16:07Z

tools/validate-image/valImage.go

+}
+
+func toMailContent(msg *pubsub.Message) string {
+	c := fmt.Sprintf("Message Data: %v\n", string(msg.Data))


Why are we logging message and then the data from the message as well?

From the alert receiver's perspective, the pubsub message is probably not useful. I'll update the mail content to only print out the human readable message.

srinivashegde86 · 2019-09-16T19:16:33Z

tools/validate-image/valImage.go

+}
+
+// NewValidateImageClient initialize all the resources for monitoring image vulnerabilities
+func NewValidateImageClient(mconfig *mail.Config) (*Client, error) {


Add unit test

chaodaiG · 2019-11-04T16:21:34Z

/uncc
add me back if needed

duglin · 2020-01-28T17:34:44Z

@yt3liu what's the status of this?

knative-housekeeping-robot · 2020-04-28T00:02:34Z

Issues go stale after 90 days of inactivity.
Mark the issue as fresh by adding the comment /remove-lifecycle stale.
Stale issues rot after an additional 30 days of inactivity and eventually close.
If this issue is safe to close now please do so by adding the comment /close.

Send feedback to Knative Productivity Slack channel or file an issue in knative/test-infra.

/lifecycle stale

knative-housekeeping-robot · 2020-04-28T12:01:38Z

Issues go stale after 90 days of inactivity.
Mark the issue as fresh by adding the comment /remove-lifecycle stale.
Stale issues rot after an additional 30 days of inactivity and eventually close.
If this issue is safe to close now please do so by adding the comment /close.

Send feedback to Knative Productivity Slack channel or file an issue in knative/test-infra.

/lifecycle stale

* genreconciler:nonNamespaced * Adding a force-kinds flag to the reconciler generator to allow us to generate reconcilers for non-owned types. * force-kinds --> force-genreconciler-kinds * some minor nits with the generators for non-knative types needed to be removed. * remove whitespace Co-authored-by: Nacho Cano <[email protected]>

knative-prow-robot reviewed Jul 23, 2019

View reviewed changes

knative-prow-robot requested review from chaodaiG and steuhs July 23, 2019 17:42

knative-prow-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 23, 2019

srinivashegde86 reviewed Jul 23, 2019

View reviewed changes

tools/monitoring/main.go Outdated Show resolved Hide resolved

srinivashegde86 reviewed Jul 23, 2019

View reviewed changes

tools/monitoring/valimage/valImage.go Outdated Show resolved Hide resolved

srinivashegde86 reviewed Jul 23, 2019

View reviewed changes

tools/monitoring/valimage/valImage.go Outdated Show resolved Hide resolved

yt3liu added 2 commits September 5, 2019 09:57

added valImage to monitor image vulnerabilities

db34047

moved the validateimage files to a separate tools directory

595333b

yt3liu force-pushed the val-image branch from 246e38a to 595333b Compare September 12, 2019 20:39

chaodaiG reviewed Sep 12, 2019

View reviewed changes

tools/validate-image/main.go Outdated Show resolved Hide resolved

tools/validate-image/valImage.go Outdated Show resolved Hide resolved

changed alert email to [email protected]

683b6f0

knative-prow-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 13, 2019

srinivashegde86 suggested changes Sep 16, 2019

View reviewed changes

adrcunha mentioned this pull request Jan 28, 2020

Process for reporting issues with our images knative/infra#136

Open

knative-prow-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 28, 2020

yt3liu closed this May 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

added valImage to monitor image vulnerabilities #1175

added valImage to monitor image vulnerabilities #1175

yt3liu commented Jul 23, 2019

knative-prow-robot left a comment

chaodaiG commented Sep 13, 2019

knative-prow-robot commented Sep 13, 2019

srinivashegde86 Sep 16, 2019

yt3liu Sep 16, 2019

srinivashegde86 Sep 16, 2019

srinivashegde86 Sep 16, 2019

srinivashegde86 Sep 16, 2019

yt3liu Sep 16, 2019

srinivashegde86 Sep 16, 2019

srinivashegde86 Sep 16, 2019

srinivashegde86 Sep 16, 2019

yt3liu Sep 16, 2019

srinivashegde86 Sep 16, 2019

chaodaiG commented Nov 4, 2019

duglin commented Jan 28, 2020

knative-housekeeping-robot commented Apr 28, 2020

knative-housekeeping-robot commented Apr 28, 2020

	log.Fatal(err)
	log.Fatalf("Failed to create mail config: %v", err)


		func validateImage(w http.ResponseWriter, r *http.Request) {
		log.Printf("Serving request: %s", r.URL.Path)

added valImage to monitor image vulnerabilities #1175

added valImage to monitor image vulnerabilities #1175

Conversation

yt3liu commented Jul 23, 2019

knative-prow-robot left a comment

Choose a reason for hiding this comment

chaodaiG commented Sep 13, 2019

knative-prow-robot commented Sep 13, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chaodaiG commented Nov 4, 2019

duglin commented Jan 28, 2020

knative-housekeeping-robot commented Apr 28, 2020

knative-housekeeping-robot commented Apr 28, 2020