feat(policies): Add granular policy_sets

[pseudomorph]

what

This change updates policy checks to be more granular.
Namely:

• When specifying multiple policy_sets, owners of each policy set are considered independently.
• Top-level owners are able to give approval for all policy sets.
• Multiple approvals can be configured to be required to pass policy checks. i.e. a policy set can require 2 approvals to pass.
• Policies to approve can be specified by: project, dir, workspace, or an individual poilicyset.
• Output from conftest will be placed in the working dir of the project, so it can be used in custom workflow steps. i.e for use to generate metrics to send to a logging platform of choice.

Implementation details:

PolicySet configuration allows for the specifying of owners at the global level and on a per policy set level. Additionally, a new parameter ReviewCount, which specifies how many approvals are required to override policy checks, has been added on the same levels. These are both considered when determining if a policy set is approved or not.

Some changes were made to pass around or expose the additional context needed to manage state between policy_approval commands. The BoltDB/Redis backends have been updated to store information about each policy set on a per-project level.

Since top-level policy set owners can approve all defined policy sets, an additional --policy-set argument has been added to the comment command parser, which allows for targeted policy approvals. Along with this change, policies can be approved on a per plan, workspace, or dir level, using the same args as all other Atlantis commands.

Having discreetly approvable policy sets changes the possible error/success dynamic that the markdown renderer used to rely on.

For example:

A PR might have two failing policy sets. Someone comes along to approve and is authorized for one but not the other. In this case, we want to bubble up both the successful approval one policy set (or at least information about the approval state) along with an error indicating the authorization issue on the other.

This would also be the case where a user is authorized for all policy sets, but the policy set requires more than one approval.

To account for this, I've updated the markdown renderer to render all policies before error or failure, and then if there is error or failure, the rendered content is passed into those templates as additional context.

why

Currently, when multiple policy sets are specified, only one set of owners is considered for all, and only one approval is sufficient for overriding policy checks, where multiple might be desired.

tests

Updated existing tests, and added some new ones.

references

• closes Feature request: per-policyset approver requirements #2982
• closes Feature Request: Multiple approvers for conftest policies #2024
• closes Policy checks as a sets for different owners #1800
• closes Bug: Running apply twice bypasses policies #3138

[pseudomorph]

Right now I'm thinking of re-factoring contest_client.go to pass a list of structs, instead of text output, which can be interpreted by policy_check_runner.go to generate the output there, and also update pull status with additional info about how many approvals a given policyset requires/has received.

[pseudomorph]

I've verified that the core functionality is set up with local testing. It should just be minor tweaking and polishing from here. I'll get to starting on tests and documentation tonight/tomorrow.

[pseudomorph]

I was thinking more about sticky policy approvals. Would something like storing the commit ID in the Atlantis DB and preserving reviews if the commit IDs match between policy check runs make sense? This behavior would, of course, be gated behind a parameter which defaults to false.

[pseudomorph]

I'll open a separate issue to encapsulate the above problem and a few other enhancements to policies to take care of once this is merged in.

[pseudomorph]

Is there a way to re-trigger tests/checks?

[pseudomorph]

Seems like the build is stuck.

[jamengual]

Seems like the build is stuck.

I have no idea, I guess github could be having issues?

[pseudomorph]

The PR was in a mergeable state the other day, but something funny is going on now. I'm not quite sure how to resolve.

[pseudomorph]

@nitrocode @jamengual @GenPage - Apologies for the ping. Any chance we can get this merged in this week? It'd be great to get back on an upstream Atlantis image.

[GenPage]

@pseudomorph no worries, I will take a look today. I'm also fixing some of the workflows.

[GenPage]

One final review, everything looks good. It touches a lot of the codebase but I don't see any issues right off the bat. Thank you for the documentation update with pictures. Please make sure to test this with the dev image tags before we release.

[pseudomorph]

Thank you so much! I'll get us on the dev image in the next week.