Minder to avoid commenting each time there's a change on a PR

[rdimitrov]

Please describe the enhancement

Each time there's a push to a PR triggers a new evaluation which results in another comment being added on the PR from minder. That can easily spam the PR and annoy people, i.e. #1806.

Solution Proposal

I'll give codecov as an example, but there are other tools that use this approach too.

The idea is to have only one comment that is just being edited each time there's a change.

Here's an example of this in a PR:

• chore: split actions to tests and linting rdimitrov/go-tuf-metadata#89 (comment)
• You can see all of the edits that have been done each time the author of the PR made a change
• Another thing we can reuse is the overview presented in it. In our case we can do a similar table view and have something like - minder found X vulnerabilities, suggested remediations for Y of them, etc.

Describe alternatives you've considered

No response

Additional context

No response

Acceptance Criteria

No response

[gregfurman]

Hi @rdimitrov.

Currently, when Minder posts a review with vulnerabilities on GitHub (example: #1806 (review)), it adds one comment for each vulnerability and posts all of them alongside the review POST. However, the single-review approach will require comments to be deleted, edited, or added with each new PR (assuming vulnerabilities change).

The issue is that the GitHub API doesn't support batch deleting or adding comments to an existing PR review, requiring one request per comment (DELETE/POST/PUT) running the risk of triggering rate limiting.

Assuming suggestion comments are not entirely necessary, I think a some good alternative approaches could be:

• Delete the previous review when a new one is made -- however this will not pin the review at the top of the PR
• Create a vulnerability table and post that within the review body (and hope this doesn't exceed the body character limit)
• Link to a full external report, with all suggestions, similar to Codecov and provide a summary in the review body

However, if rate limiting is not going to be an issue then I'm happy to go with the 1 request per comment change approach. Else perhaps just deleting the previous review and posting a new one is the best approach.

[jhrozek]

Given that finding CVEs in dependencies in a relatively rare thing I wouldn't stress too much about depleting the API tokens honestly. As long as the happy path (checking if dependencies have CVEs) don't trigger that many API calls, I would be OK to make the experience for the rarer use-case nicer.

The alternative of deleting a previous review sounds honestly also good to me if API rate limiting is of concern.

My 2c.

[gregfurman]

Given that finding CVEs in dependencies in a relatively rare thing I wouldn't stress too much about depleting the API tokens honestly. As long as the happy path (checking if dependencies have CVEs) don't trigger that many API calls, I would be OK to make the experience for the rarer use-case nicer.

Alright will keep going with this approach then.

The alternative of deleting a previous review sounds honestly also good to me if API rate limiting is of concern.

Only consideration here is it runs the risk of being a bit spammy if one has email notifications enabled -- since each new review will send another notification. Not a huge deal though but could maybe get a bit annoying having 2 new emails/notifications for every new update.

[rdimitrov]

The motivation behind the issue is mostly to not clutter a PR that is in active development but also to avoid spamming all subscribed people since each review translates to another email.

I think a nice balance can be to fix the summary so it is a single comment(instead of multiple new ones) which sits on top (assuming Minder commented first when the PR was created) and just edit it each time there's a change. This will help reduce the amount of clutter and spam.

• If there are no new issues found, there will be no reviews and no new comments -> so no spam.
• Otherwise if Minder finds an issue, it should still perform a review and request changes on it. The summary can include a table list of found vulnerabilities.
• If there was a previous review by Minder and now it is fixed, discard the "requested changes" review and edit the summary to match that it's all good now.

@jhrozek @gregfurman - what do you think?

[gregfurman]

Makes sense. So to clarify, the comments I am referring to are those that reference where in the diff a vulnerable dependency is imported. You are suggesting such comments be removed in favour of a summary table that will reside in the review body?

So instead of a review with attached comments like in #1806 (review), there will be a table that includes this information within the review body without any review comments?

[jhrozek]

Makes sense. So to clarify, the comments I am referring to are those that reference where in the diff a vulnerable dependency is imported. You are suggesting such comments be removed in favour of a summary table that will reside in the review body?

So instead of a review with attached comments like in #1806 (review), there will be a table that includes this information within the review body without any review comments?

If I understand @rdimitrov correctly, he's referring to the happy path where no CVEs are found in a patch that brings new dependencies. I still think if inline comments are nice.

[rdimitrov]

Apologies for the confusion. Indeed, I was referring to the happy path of no issues. Otherwise we should still do a review with inline comments.

I'm thinking of the following use cases (do chime in if something sounds odd):

1. Minder found issues

• We always perform a review with inline comments referring to the potential issues and set the review status to "requested changes".
• The result should be a single summary comment saying minder found issues (possibly list those in a table) and a review with "requested changes" status and inline comments.

2. Minder found no issues (so called happy path)

• We don't perform a review, but also discard any previous reviews being done from Minder (if feasible also resolving any opened discussions caused by the inline comments)
• The result should be a single summary comment that should say all is good without any reviews from Minder and/or unresolved discussions.

In both cases there should be only one comment from Minder in the PR. When evaluating for the 1st time we'll create the comment, but on next passes we'll just edit it to reflect the desired state (if needed) and either perform/discard a review.

[gregfurman]

1. Minder found issues

• We always perform a review with inline comments referring to the potential issues and set the review status to "requested changes".
• The result should be a single summary comment saying minder found issues (possibly list those in a table) and a review with "requested changes" status and inline comments.

Alright so if I am understanding this correctly, you are suggesting:

• A comment exists at the top of the PR that is edited to reflect a summary of the latest eval.
• If vulnerabilities are found, Minder will also post a review with inline comments and some other information regarding the vulnerability report
• If no vulnerabilities are found and a review exists, remove/dismiss the review

So one should expect (in the case of a vulnerabilities being found) a single comment at the top of the PR to be edited and a separate review to be posted with inline comments. In the happy path, just expect the comment on top of the PR to be edited,

---

Think I am getting a bit confused about the difference between a comment and a review in this context. Github unfortunately does not allow for new inline comments to be added to a review that has been posted -- one can only modify the review body (which is also technically a comment) and the text of the comments within the review.

[rdimitrov]

Alright so if I am understanding this correctly, you are suggesting:

• A comment exists at the top of the PR that is edited to reflect a summary of the latest eval.
• If vulnerabilities are found, Minder will also post a review with inline comments and some other information regarding the vulnerability report
• If no vulnerabilities are found and a review exists, remove/dismiss the review

*and mark as resolved all the diff inline suggestions. The reason is Minder should not falsely block the PR in case the setting where a PR cannot be merged with unresolved discussions is enabled for a given repository.

So one should expect (in the case of a vulnerabilities being found) a single comment at the top of the PR to be edited and a separate review to be posted with inline comments. In the happy path, just expect the comment on top of the PR to be edited,

That's correct! 👍

Think I am getting a bit confused about the difference between a comment and a review in this context. Github unfortunately does not allow for new inline comments to be added to a review that has been posted -- one can only modify the review body (which is also technically a comment) and the text of the comments within the review.

No, you're right, it's a bit awkward because indeed each review is essentially another comment 😃

By the way, feel free to ping me on Discord, I'll be happy to have a chat with you about this 👍 🙏

[gregfurman]

What are the thoughts on the below formatting for the pinned comment that gets edited? These were generated from the unit tests I've created and run locally.

There is a dummy review referred to here but that should link to the actual minder review posted.

Have also added some metadata to the body (in comment form) so that one can get the previous results of a scan and calculate how results differ from one commit to another (have not added this functionality in yet but the metadata is there).

Vulnerabilities found example comment:

Minder Vulnerability Report ⚠️

Minder found vulnerable dependencies in this PR. Either push an updated version or accept the proposed changes. Note that accepting the changes will include Minder as a co-author of this PR.

Vulnerability scan of 27d6810b:

📊 View Full Review

• 🐞 vulnerabilities: 1
• 🛠 fixes: 1

Package	Version	#Vulnerabilities	#Fixes	Patch Exists
mongodb	0.5.0	1	1	✅

---

No vulnerabilties found example comment:

Minder Vulnerability Report ✅

Minder analyzed this PR and found no vulnerable dependencies.

Vulnerability scan of 27d6810b:

• 🐞 vulnerabilities: 0
• 🛠 fixes: 0

[rdimitrov]

@gregfurman - Oh, this looks nice! 🚀

These will be a summary of all found packages, right?

🐞 vulnerabilities: 1
🛠 fixes: 1

Also, I wonder if we should also add a get in touch part like in the screenshot for Codecov? For example: Have feedback on the report? Share it here.

[gregfurman]

These will be a summary of all found packages, right?

🐞 vulnerabilities: 1
🛠 fixes: 1

Yeah, this summary just shows the total count of vulnerable dependencies and total count of suggested/available remediations. The table will go into more detail about how many vulnerabilities were found in each dependency. Happy to add more info here if it's unclear -- was trying to keep things simple and understandable 😄

Also, I wonder if we should also add a get in touch part like in the screenshot for Codecov? For example: Have feedback on the report? Share it here.

That is a great idea. Perhaps it would be useful to provide another template to provide feedback? Labelling the issue as Feedback could help too. One can link it as follows with a template:

Share it here

[rdimitrov]

That is a great idea. Perhaps it would be useful to provide another template to provide feedback? Labelling the issue as Feedback could help too. One can link it as follows with a template:

Share it here

I think the mail should be fine for now since we already refer to it from other things like the security advisories we alert through 👍 We can easily revisit if we want to converge on something else 👍