Create single status comment and correctly dismiss reviews #2171

gregfurman · 2024-01-22T20:20:37Z

Overview

This PR allows for a single status comment to be repeatedly created and updated, illustrating an overview state of the minder evaluation. This summary comment is made in conjunction with the minder review comment.

Summary of changes

New interface for comments that includes formatting, more info, and emojis
Creates or updates a single summary comment atop a PR that is edited whenever a new evaluation is run
Finds correct prior minder review from GitHub API and (when applicable) dismisses it and creates a new review
Include separate internal/engine/eval/vulncheck/report.go that handles review and status comment templating
Checks if review or comment is from user stacklokbot -- preventing a user from abusing magic comment checks

To do

Include automatic resolving of review comments after dismissing (see Include GitHub v4 GraphQL API #2170)
Potentially add comparison information between summary comments to show changes will affect main (see Include comparison between vulnerability evaluations in minder summary #2149)

Example comments

Example Summary comment

Minder Vulnerability Report ⚠️

Minder found vulnerable dependencies in this PR. Either push an updated version or accept the proposed changes. Note that accepting the changes will include Minder as a co-author of this PR.

Vulnerability scan of 27d6810b:

📊 View Full Review

🐞 vulnerabilities: 1

🛠 fixes: 1

Package	Version	#Vulnerabilities	#Fixes	Patch
mongodb	0.5.0	1	1	0.6.0

📬 Have feedback? Share it here.

Example Review Comment

Summary of vulnerabilities found

Minder found the following vulnerabilities in this PR:

Ecosystem	Name	Version	Vulnerability ID	Summary	Introduced	Fixed
npm	mongodb	0.5.0	mongodb			0.6.0

📬 Have feedback on the report? Share it here.

JAORMX · 2024-01-23T06:30:10Z

I accidentally marked it off draft since I was reviewing from my phone.

jhrozek

This is great, thanks for working on improving the UX and fixing the bug! I left a couple of comments inline.

jhrozek · 2024-01-23T08:26:19Z

internal/engine/eval/vulncheck/report.go

+const (
+	reviewBodyMagicComment = "<!-- minder: pr-review-body -->"
+	statusBodyMagicComment = "<!-- minder: pr-status-body -->"
+	separatorComment       = "<!--[SEP]-->"


separatorComment seems to be unused.

Separator is meant for adding metadata to the body of the review and allowing us to easily parse it. So we can include data from a previous review as a comment in JSON form and use it to provide a comparison to the current review. i.e the below will provide info on the previous status report:

 <h2>Minder Vulnerability Report ⚠️</h2> ...

Probably should have made this more clear. It was also mentioned it here #2149 😄

yes, but the constant separatorComment is still unused, the "string constant is used verbatim in minderTemplateString`.

jhrozek · 2024-01-23T08:31:30Z

internal/engine/eval/vulncheck/review.go

-Previous Minder review was dismissed because the PR was updated.
-`
-	vulnFoundWithNoPatch = "Vulnerability found, but no patched version exists yet."
+	stacklokBotUser     = "stacklokbot"


Hmm, I don't think we want to hardcode a bot username here. Later the code does:

isMinder := review.GetUser().GetLogin() == stacklokBotUser && strings.HasPrefix(review.GetBody(), reviewBodyMagicComment)

Could we instead use:

isMinder := review.GetUser().GetLogin() == ghCli.GetAuthenticatedUser() && strings.HasPrefix(review.GetBody(), reviewBodyMagicComment)

Haven't looked into that function. Do we expect ghCli.GetAuthenticatedUser() to return stacklokbot or the user who triggered the pipeline?

It would return the user who enrolled that particular org or personal account, so in the case of stacklok's deployment, it would be stacklokbot.

jhrozek · 2024-01-23T08:33:30Z

internal/engine/eval/vulncheck/review.go

-	for _, r := range reviews {
-		if strings.HasPrefix(r.GetBody(), reviewBodyMagicComment) && r.GetState() != "DISMISSED" {
-			ra.minderReview = r
+	for i := len(reviews) - 1; i >= 0; i-- {


This is done because one of the calls returns reviews in reverse order, right? Could you add a comment here so that we remember the reason later?

Yes exactly. Very fair point. Will add a comment.

jhrozek · 2024-01-23T08:34:57Z

internal/engine/eval/vulncheck/review.go

+}
+
+func (ra *reviewPrHandler) findPreviousStatusComment(ctx context.Context) error {
+	comments, err := ra.cli.ListComments(ctx, ra.pr.RepoOwner, ra.pr.RepoName, int(ra.pr.Number),


oh so ListComments allows you to set the sorting, but ListReviews does not?

Unfortunately this part of the GitHub API is a bit inconsistent
See https://docs.github.com/en/rest/pulls/reviews?apiVersion=2022-11-28#list-reviews-for-a-pull-request versus https://docs.github.com/en/rest/pulls/comments?apiVersion=2022-11-28#list-review-comments-in-a-repository

Ah, I think this is the crux of the issues I was seeing. So the ra.cli.ListComments method wraps c.client.PullRequests.ListComments, its result is what the code iterates over and saves into ra.minderStatusReport. But, the comments are created with ra.cli.CreateComment which uses c.client.Issues.CreateComment - it's a different API. The Issues one is for what's usually referred to as comments, but the PullRequest API is for iterating over review comments, those that are submitted either inline or as the review summary.

I know this work has been going on for some time, I wonder if there's a reason those two got mixed up?

jhrozek · 2024-01-23T08:36:16Z

internal/engine/eval/vulncheck/report.go

+	fixHtmlEmoji     = "&#128736;"
+	warningHtmlEmoji = "&#9888;&#65039;"
+	successHtmlEmoji = "&#9989;"
+	reviewHtmlEmoji  = "&#128202;"


Love the bling!

jhrozek · 2024-01-24T09:24:33Z

Thanks for handling the comments!

I've tested the PR and there seems to be a case that we don't handle correctly - when I ran minder as myself (meaning the same user who submitted the PR was reviewing) and pushed a new version of a patch (I started with vulnerable python-requests=2.19.0 and as an update I tried to push 2.20.0 which still contain CVEs) I saw:

{"level":"debug","profile":"acme-github-profile-pr-vuln-check","ruleType":"pr_vulnerability_check","eval_status":"error","projectId":"ded12968-7cc4-456e-bc1d-75f28fd2bac9","repositoryId":"40a157bc-2af1-44f6-a458-701aa415bcce","exception.message":"failed to submit pr action: could not dismiss previous review: could not dismiss review: error dismissing review 1840
858980 for PR jakubtestorg/bad-python/73: PUT https://api.github.com/repos/jakubtestorg/bad-python/pulls/73/reviews/1840858980/dismissals: 422 Validation Failed [{Resource: Field: Code: Message:Can not dismiss a commented pull request review}]","Timestamp":1706087836440010000,"message":"result - evaluation"}
{"level":"info","action":"remediate","action_status":"skipped","Timestamp":1706087836440025000,"message":"result - action"}

and I didn't see any more updates from minder..

It seems like we regressed handling dismissing of comment reviews in this patch?

gregfurman · 2024-01-24T09:49:18Z

Hi @jhrozek

That is strange. So to clarify, one was previously able to dismiss reviews made by the same own author as the PR?

Here is what I think is happening, and am not sure if these changes introduce this issue. The newReviewPrHandler function in review.go stipulates the following:

	// if the user wants minder to request changes on a pull request, they need to
	// be different identities
	var failStatus *string
	if pr.AuthorId == cliUser.GetID() {
		failStatus = github.String("COMMENT")
		logger.Debug().Msg("author is the same as the authenticated user, can only comment")
	} else {
		failStatus = github.String("REQUEST_CHANGES")
		logger.Debug().Msg("author is different than the authenticated user, can request changes")
	}

So in the case of your PR, jhrozek is both the author and the de-facto reviewer. This response is therefore to be expected since the author (you) is triggering minder reviews -- which are of type COMMENT as opposed to REQUESTED_CHANGES. Since comment reviews cannot be dismissed this error is thrown (see MorrisonCole/pr-lint-action#155 (comment)).

Also, are you sure you are using this branch version of minder? I don't see any tables being generated in the review comment for the PR you are testing on.

jhrozek · 2024-01-24T11:04:09Z

Hi @jhrozek

That is strange. So to clarify, one was previously able to dismiss reviews made by the same own author as the PR?

I don't think it would dismiss the review, just create another comment. Here is a PR where I started with .20 and then went to .19 and saw a subsequent comment. This one was running main. Before that I tested with another PR running your branch where I pushed an in-between version, but saw no inline comment. I /think/ the summary was updated, but the formatting looks a bit off, too

Here is what I think is happening, and am not sure if these changes introduce this issue. The newReviewPrHandler function in review.go stipulates the following:
	// if the user wants minder to request changes on a pull request, they need to
	// be different identities
	var failStatus *string
	if pr.AuthorId == cliUser.GetID() {
		failStatus = github.String("COMMENT")
		logger.Debug().Msg("author is the same as the authenticated user, can only comment")
	} else {
		failStatus = github.String("REQUEST_CHANGES")
		logger.Debug().Msg("author is different than the authenticated user, can request changes")
	}
So in the case of your PR, jhrozek is both the author and the de-facto reviewer. This response is therefore to be expected since the author (you) is triggering minder reviews -- which are of type COMMENT as opposed to REQUESTED_CHANGES. Since comment reviews cannot be dismissed this error is thrown (see MorrisonCole/pr-lint-action#155 (comment)).

I think this is pretty much what is happening. I'll try to poke either later today or tomorrow again to understand better what's causing the issue.

Also, are you sure you are using this branch version of minder? I don't see any tables being generated in the review comment for the PR you are testing on.

jhrozek

Thank you for your patience, I apologise it took me so long to get back to this review. I think the primary issue is the mixup of the Issues and PullReques APIs, otherwise the code looks quite good to me.

Some minor comments inline.

jhrozek · 2024-02-02T13:07:59Z

internal/engine/eval/vulncheck/report.go

+const (
+	reviewBodyMagicComment = "<!-- minder: pr-review-body -->"
+	statusBodyMagicComment = "<!-- minder: pr-status-body -->"
+	separatorComment       = "<!--[SEP]-->"


yes, but the constant separatorComment is still unused, the "string constant is used verbatim in minderTemplateString`.

jhrozek · 2024-02-02T13:18:05Z

internal/engine/eval/vulncheck/review.go

+}
+
+func (ra *reviewPrHandler) findPreviousStatusComment(ctx context.Context) error {
+	comments, err := ra.cli.ListComments(ctx, ra.pr.RepoOwner, ra.pr.RepoName, int(ra.pr.Number),


Ah, I think this is the crux of the issues I was seeing. So the ra.cli.ListComments method wraps c.client.PullRequests.ListComments, its result is what the code iterates over and saves into ra.minderStatusReport. But, the comments are created with ra.cli.CreateComment which uses c.client.Issues.CreateComment - it's a different API. The Issues one is for what's usually referred to as comments, but the PullRequest API is for iterating over review comments, those that are submitted either inline or as the review summary.

I know this work has been going on for some time, I wonder if there's a reason those two got mixed up?

jhrozek · 2024-02-02T13:19:24Z

internal/engine/eval/vulncheck/report.go

+			<th>Patch</th>
+	  	</tr>
+	</thead>
+	<tbody>


For reason I really don't understand the <tbody> tag is not rendered correctly for me, see this example.

I wonder if we need the thead and tbody tags at all, can't we get away with just tr?

As far as I understand, using the Issues API is not actually an issue since every PR is technically an issue (according to the GitHub docs).

See https://docs.github.com/en/rest/issues/comments?apiVersion=2022-11-28#about-issue-and-pull-request-comments:

You can use the REST API to create and manage comments on issues and pull requests. Every pull request is an issue, but not every issue is a pull request. For this reason, "shared" actions for both features, like managing assignees, labels, and milestones, are provided within the Issues endpoints. To manage pull request review comments, see "Pull request review comments."

With that said, I'll change it to the PullRequest API for correctness.

In order to properly test this, the account making the comments (the author) cannot be the same as the account submitting a review (reviewer) because of the constraints outlined in #2171 (comment).

I'm also going to remove the section of the status comment that links the latest review -- since it's unnecessarily complicating things now that the status comment is required to be posted prior to the review comment.

jhrozek · 2024-02-02T13:21:58Z

internal/engine/eval/vulncheck/review.go

+	}
+
+	if err := ra.dismissReview(ctx); err != nil {
+		return fmt.Errorf("could not dismiss previous review: %w", err)


Note that previously error on dismiss review was not fatal, this was on purpose so that minder wouldn't error out trying to dismiss its own review running as the same use. I wonder if we could add this to body of dismissReview:

+ if ra.pr.AuthorId == ra.authorizedUser.GetID() { + ra.logger.Debug().Msg("author is the same as the authenticated user, can't dismiss") + return nil + } +

this is pretty much the same as ignoring the error, except we don't actually attempt the call so we save a token..

coveralls · 2024-02-22T11:32:21Z

coverage: 38.111% (+0.6%) from 37.465%
when pulling 2cf3dc8 on gregfurman:1862-single-review
into f846a56 on stacklok:main.

jhrozek · 2024-02-22T11:35:14Z

Hi,

I have two apologies - first for the delay, last week Stacklok met in person and not much development got done and so we're slow in PR reviews.

Second, re-reading this PR and our chats on Discord, we must have confused you and sent you down the path of implementing something that's too complex. So I pushed another patch atop yours (we squash commits so you'd still be attributed for this PR!) that I think simplifies the flow in the sense that the inline reviews are still retained, but Minder replies with inline comments only and the whole summary including the CVE table is kept in the single summary comment you introduced.

Would this approach work for you?

gregfurman · 2024-02-23T15:53:08Z

Hi,

I have two apologies - first for the delay, last week Stacklok met in person and not much development got done and so we're slow in PR reviews.

Second, re-reading this PR and our chats on Discord, we must have confused you and sent you down the path of implementing something that's too complex. So I pushed another patch atop yours (we squash commits so you'd still be attributed for this PR!) that I think simplifies the flow in the sense that the inline reviews are still retained, but Minder replies with inline comments only and the whole summary including the CVE table is kept in the single summary comment you introduced.

Would this approach work for you?

Hi. No worries. Happy to go with the above approach and have these changes merged in 😄

evankanderson · 2024-03-05T21:37:22Z

internal/engine/eval/vulncheck/report.go

+	TrackedDependencies []dependencyVulnerabilities
+}
+
+func (r *vulnSummaryReport) render() (string, error) {


Not to block this, but it feels odd to assemble this from three templates, rather than one larger template that takes r.TrackedDependencies as the input, rather than looping outside the template over the tableVulnerabilitiesRows repeatedly.

(Also, the header template doesn't seem to have any template stuff in it, so it could just be treated like the footer.)

evankanderson · 2024-03-05T22:00:32Z

internal/engine/eval/vulncheck/review.go

@@ -305,52 +256,93 @@ func (ra *reviewPrHandler) setStatus() {
 	ra.logger.Debug().Str("status", *ra.status).Msg("will set review status")
 }

-func (ra *reviewPrHandler) findPreviousReview(ctx context.Context) error {
-	reviews, err := ra.cli.ListReviews(ctx, ra.pr.RepoOwner, ra.pr.RepoName, int(ra.pr.Number), nil)
+func (ra *reviewPrHandler) findPreviousStatusComment(ctx context.Context) error {


It would be nice if this function didn't pass the found review as a struct field, but again, I'm not going to block this review on existing code styles.

evankanderson · 2024-03-05T22:02:37Z

internal/engine/eval/vulncheck/review.go

+	return mci
+}
+
+func (ra *reviewPrHandler) submitReview(ctx context.Context, mci magicCommentInfo) (int64, error) {


Future comment -- it would make sense to include a comment indicating what the returned int64 is. In this case, it appears to be the issue / PR number.

Obsoleted by further changes.

evankanderson · 2024-03-05T22:06:00Z

Thanks very much! Unfortunately, it looks like there are a couple of conflicts, so I can't merge. @jhrozek or @gregfurman if you could resolve the conflicts, I'm happy to merge this despite the small nits.

This commit builds atop the previous one and moves further so that the message in the review reply (not to be confused with inline replies) are empty and all the feedback is kept in the single summary comment. Because we still need to dismiss the reviews, but now we don't have a review text body where we used to keep the magic comment with metadata, we keep it in the summary comment instead. We also keep the review ID there because we can't be sure which is the minder review now in case the same identity is used for a human identity and minder.

jhrozek · 2024-03-06T20:33:00Z

Thanks very much! Unfortunately, it looks like there are a couple of conflicts, so I can't merge. @jhrozek or @gregfurman if you could resolve the conflicts, I'm happy to merge this despite the small nits.

Thank you for the review @evankanderson. If you don't mind I will address the comments you had as part of #1203 (which is getting more and more pressing as we have a fair bit of code duplication now between the reply machinery for trusty and OSV)

JAORMX marked this pull request as ready for review January 23, 2024 06:29

JAORMX requested a review from a team as a code owner January 23, 2024 06:29

JAORMX marked this pull request as draft January 23, 2024 07:24

jhrozek reviewed Jan 23, 2024

View reviewed changes

gregfurman force-pushed the 1862-single-review branch 3 times, most recently from be86330 to 2a33343 Compare January 23, 2024 12:59

gregfurman marked this pull request as ready for review January 23, 2024 13:00

gregfurman requested a review from jhrozek January 24, 2024 08:37

jhrozek previously requested changes Feb 2, 2024

View reviewed changes

evankanderson added the waiting-for-feedback label Feb 6, 2024

teodor-yanev mentioned this pull request Feb 8, 2024

Add Homoglyphs detection in Minder #2312

Merged

gregfurman force-pushed the 1862-single-review branch 3 times, most recently from 8a73286 to efdee43 Compare February 13, 2024 15:14

gregfurman requested a review from jhrozek February 13, 2024 15:23

jhrozek force-pushed the 1862-single-review branch from efdee43 to 7799097 Compare February 22, 2024 11:28

jhrozek removed the waiting-for-feedback label Feb 22, 2024

jhrozek force-pushed the 1862-single-review branch from 7799097 to 2cf3dc8 Compare February 23, 2024 19:36

evankanderson previously approved these changes Mar 5, 2024

View reviewed changes

Create single status comment and correctly dismiss reviews

f997abe

jhrozek added 3 commits March 6, 2024 21:26

Don't print review dismissed comment as a code block

98d1dbe

Remove unused ReviewUrl struct member

f213b73

jhrozek dismissed evankanderson’s stale review via f213b73 March 6, 2024 20:28

jhrozek force-pushed the 1862-single-review branch from 2cf3dc8 to f213b73 Compare March 6, 2024 20:28

jhrozek mentioned this pull request Mar 6, 2024

Split the PR actions of the vulncheck evaluator into a proper remediate action #1203

Open

evankanderson approved these changes Mar 6, 2024

View reviewed changes

evankanderson merged commit 9d2df9e into mindersec:main Mar 6, 2024
20 checks passed

evankanderson mentioned this pull request Jul 16, 2024

Include comparison between vulnerability evaluations in minder summary #2149

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create single status comment and correctly dismiss reviews #2171

Create single status comment and correctly dismiss reviews #2171

gregfurman commented Jan 22, 2024

JAORMX commented Jan 23, 2024

jhrozek left a comment

jhrozek Jan 23, 2024

gregfurman Jan 23, 2024

jhrozek Feb 2, 2024

jhrozek Jan 23, 2024

gregfurman Jan 23, 2024

jhrozek Jan 23, 2024

gregfurman Jan 23, 2024

jhrozek Jan 23, 2024

gregfurman Jan 23, 2024

jhrozek Jan 23, 2024

gregfurman Jan 23, 2024

jhrozek Feb 2, 2024

jhrozek Jan 23, 2024

jhrozek commented Jan 24, 2024

gregfurman commented Jan 24, 2024 •

edited

Loading

jhrozek commented Jan 24, 2024

jhrozek left a comment

jhrozek Feb 2, 2024

jhrozek Feb 2, 2024

jhrozek Feb 2, 2024

gregfurman Feb 12, 2024 •

edited

Loading

gregfurman Feb 12, 2024

jhrozek Feb 2, 2024

coveralls commented Feb 22, 2024 •

edited

Loading

jhrozek commented Feb 22, 2024

gregfurman commented Feb 23, 2024

evankanderson Mar 5, 2024

evankanderson Mar 5, 2024

evankanderson Mar 5, 2024

evankanderson commented Mar 5, 2024

jhrozek commented Mar 6, 2024

Create single status comment and correctly dismiss reviews #2171

Create single status comment and correctly dismiss reviews #2171

Conversation

gregfurman commented Jan 22, 2024

Overview

Summary of changes

To do

Example comments

Example Summary comment

Minder Vulnerability Report ⚠️

Vulnerability scan of 27d6810b:

Example Review Comment

Summary of vulnerabilities found

JAORMX commented Jan 23, 2024

jhrozek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jhrozek commented Jan 24, 2024

gregfurman commented Jan 24, 2024 • edited Loading

jhrozek commented Jan 24, 2024

jhrozek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gregfurman Feb 12, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Feb 22, 2024 • edited Loading

jhrozek commented Feb 22, 2024

gregfurman commented Feb 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evankanderson commented Mar 5, 2024

jhrozek commented Mar 6, 2024

Vulnerability scan of `27d6810b:`

gregfurman commented Jan 24, 2024 •

edited

Loading

gregfurman Feb 12, 2024 •

edited

Loading

coveralls commented Feb 22, 2024 •

edited

Loading