apply_requirements are ignored when policy check is enabled and passes

[edbighead]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

Atlantis allows apply when mergeable is not satisfied (Github).

Reproduction Steps

• Set the following in your repos.yaml config

repos:
  - id: github.com/xxx/yyy
    apply_requirements: ["mergeable"]
    allowed_overrides: [workflow]
    allow_custom_workflows: false
policies:
  owners:
    users:
      - your-user
  policy_sets:
    - name: common-policies
      path: /home/atlantis/policies
      source: local

• Add some dummy policies
• Add CODEOWNERS file to your github repo and set Require review from Code Owners branch protection
• Open a PR and let a non-CODEOWNER approve PR
• Comment atlantis apply

Even though mergeable condition is not being satisfied and Github API is returning "mergeable_state": "blocked" from get-a-pull-request, the apply goes through.

Potentially caused by policies_passed in apply_requirements.

Merge check is happening here:

atlantis/server/events/apply_requirement_handler.go

Lines 31 to 34 in 7199ea4

	case raw.MergeableApplyRequirement:
	if !ctx.PullReqStatus.Mergeable {
	return "Pull request must be mergeable before running apply.", nil
	}

Mergeability status is fetched from Github API here:

atlantis/server/events/vcs/github_client.go

Lines 299 to 319 in 7199ea4

	// PullIsMergeable returns true if the pull request is mergeable.
	func (g *GithubClient) PullIsMergeable(repo models.Repo, pull models.PullRequest) (bool, error) {
	githubPR, err := g.GetPullRequest(repo, pull.Num)
	if err != nil {
	return false, errors.Wrap(err, "getting pull request")
	}
	state := githubPR.GetMergeableState()
	// We map our mergeable check to when the GitHub merge button is clickable.
	// This corresponds to the following states:
	// clean: No conflicts, all requirements satisfied.
	// Merging is allowed (green box).
	// unstable: Failing/pending commit status that is not part of the required
	// status checks. Merging is allowed (yellow box).
	// has_hooks: GitHub Enterprise only, if a repo has custom pre-receive
	// hooks. Merging is allowed (green box).
	// See: https://github.com/octokit/octokit.net/issues/1763
	if state != "clean" && state != "unstable" && state != "has_hooks" {
	return false, nil
	}
	return true, nil
	}

Logs

Obfuscated org and repo names.
Automerge is set to true, apply goes through regardless of mergeable status. Waiting on code owner review from ... log entry tells indeed that this PR is not mergeable.

{"level":"debug","ts":"2022-04-05T14:18:54.388Z","caller":"vcs/github_client.go:277","msg":"GET /repos/xxx/yyy/pulls/4025/reviews","json":{}}
{"level":"debug","ts":"2022-04-05T14:18:55.774Z","caller":"events/project_command_builder.go:547","msg":"Merging config for project at dir: \"terraform/prod/s3/images\" workspace: \"default\"","json":{"repo":"xxx/yyy","pull":"4025"}}
{"level":"debug","ts":"2022-04-05T14:18:55.775Z","caller":"valid/global_cfg.go:231","msg":"MergeProjectCfg started","json":{"repo":"xxx/yyy","pull":"4025"}}
{"level":"debug","ts":"2022-04-05T14:18:55.775Z","caller":"valid/global_cfg.go:471","msg":"setting apply_requirements: [mergeable,policies_passed] from repos[2], id: github.com/xxx/yyy","json":{"repo":"xxx/yyy","pull":"4025"}}
{"level":"debug","ts":"2022-04-05T14:18:55.775Z","caller":"valid/global_cfg.go:471","msg":"setting workflow: \"encrypted\" from repos[2], id: github.com/xxx/yyy","json":{"repo":"xxx/yyy","pull":"4025"}}
{"level":"debug","ts":"2022-04-05T14:18:55.775Z","caller":"valid/global_cfg.go:471","msg":"setting allowed_overrides: [workflow] from repos[2], id: github.com/xxx/yyy","json":{"repo":"xxx/yyy","pull":"4025"}}
{"level":"debug","ts":"2022-04-05T14:18:55.775Z","caller":"valid/global_cfg.go:471","msg":"setting allow_custom_workflows: false from repos[2], id: github.com/xxx/yyy","json":{"repo":"xxx/yyy","pull":"4025"}}
{"level":"debug","ts":"2022-04-05T14:18:55.775Z","caller":"valid/global_cfg.go:471","msg":"setting delete_source_branch_on_merge: false from default server config","json":{"repo":"xxx/yyy","pull":"4025"}}
{"level":"debug","ts":"2022-04-05T14:18:55.775Z","caller":"valid/global_cfg.go:279","msg":"MergeProjectCfg completed","json":{"repo":"xxx/yyy","pull":"4025"}}
{"level":"debug","ts":"2022-04-05T14:18:55.775Z","caller":"valid/global_cfg.go:282","msg":"final settings: apply_requirements: [mergeable,policies_passed], workflow: encrypted","json":{"repo":"xxx/yyy","pull":"4025"}}
{"level":"debug","ts":"2022-04-05T14:18:55.775Z","caller":"events/project_command_context_builder.go:110","msg":" are enabled","json":{"repo":"xxx/yyy","pull":"4025"}}
{"level":"debug","ts":"2022-04-05T14:18:55.775Z","caller":"events/project_command_context_builder.go:53","msg":"Building project command context for apply","json":{"repo":"xxx/yyy","pull":"4025"}}
{"level":"info","ts":"2022-04-05T14:18:56.139Z","caller":"runtime/apply_step_runner.go:38","msg":"starting apply","json":{"repo":"xxx/yyy","pull":"4025"}}
{"level":"debug","ts":"2022-04-05T14:18:56.140Z","caller":"terraform/terraform_client.go:407","msg":"starting \"/usr/local/bin/terraform0.12.31 apply -input=false \\\"/home/atlantis/.atlantis/repos/xxx/yyy/4025/default/terraform/prod/s3/images/tf-prod-s3-images-default.tfplan\\\"\" in \"/home/atlantis/.atlantis/repos/xxx/yyy/4025/default/terraform/prod/s3/images\"","json":{"repo":"xxx/yyy","pull":"4025"}}
{"level":"info","ts":"2022-04-05T14:18:59.932Z","caller":"terraform/terraform_client.go:467","msg":"successfully ran \"/usr/local/bin/terraform0.12.31 apply -input=false \\\"/home/atlantis/.atlantis/repos/xxx/yyy/4025/default/terraform/prod/s3/images/tf-prod-s3-images-default.tfplan\\\"\" in \"/home/atlantis/.atlantis/repos/xxx/yyy/4025/default/terraform/prod/s3/images\"","json":{"repo":"xxx/yyy","pull":"4025"}}
{"level":"info","ts":"2022-04-05T14:18:59.932Z","caller":"runtime/apply_step_runner.go:57","msg":"apply successful, deleting planfile","json":{"repo":"xxx/yyy","pull":"4025"}}
{"level":"debug","ts":"2022-04-05T14:19:00.337Z","caller":"vcs/github_client.go:209","msg":"GET /repos/xxx/yyy/issues/4025/comments","json":{}}
{"level":"debug","ts":"2022-04-05T14:19:00.586Z","caller":"vcs/github_client.go:196","msg":"POST /repos/xxx/yyy/issues/4025/comments","json":{}}
{"level":"debug","ts":"2022-04-05T14:19:01.487Z","caller":"events/db_updater.go:24","msg":"updating DB with pull results","json":{"repo":"xxx/yyy","pull":"4025"}}
{"level":"debug","ts":"2022-04-05T14:19:01.941Z","caller":"vcs/github_client.go:196","msg":"POST /repos/xxx/yyy/issues/4025/comments","json":{}}
{"level":"debug","ts":"2022-04-05T14:19:01.949Z","caller":"server/middleware.go:44","msg":"POST /events – from 172.1.1.1:21382","json":{}}
{"level":"debug","ts":"2022-04-05T14:19:01.949Z","caller":"events/events_controller.go:96","msg":"handling GitHub post","json":{}}
{"level":"debug","ts":"2022-04-05T14:19:01.949Z","caller":"events/events_controller.go:146","msg":"request valid","json":{}}
{"level":"debug","ts":"2022-04-05T14:19:01.950Z","caller":"events/events_controller.go:152","msg":"handling as comment event","json":{}}
{"level":"debug","ts":"2022-04-05T14:19:01.950Z","caller":"logging/simple_logger.go:155","msg":"Ignoring non-command comment: \"Ran Apply for project: `tf-prod-s3-image...\"","json":{}}
{"level":"debug","ts":"2022-04-05T14:19:01.950Z","caller":"server/middleware.go:70","msg":"POST /events – respond HTTP 200","json":{}}
{"level":"info","ts":"2022-04-05T14:19:02.895Z","caller":"events/automerger.go:31","msg":"automerging pull request","json":{"repo":"xxx/yyy","pull":"4025"}}
{"level":"debug","ts":"2022-04-05T14:19:02.895Z","caller":"vcs/github_client.go:403","msg":"GET /repos/xxx/yyy","json":{}}
{"level":"debug","ts":"2022-04-05T14:19:03.206Z","caller":"vcs/github_client.go:426","msg":"PUT /repos/github.User{Login:\"xxx\", ID:6919294, NodeID:\"qqqqq=\", AvatarURL:\"https://avatars.githubusercontent.com/u/xxxxx?v=4\", HTMLURL:\"https://github.com/xxx\", GravatarID:\"\", Type:\"Organization\", SiteAdmin:false, URL:\"https://api.github.com/users/xxx\", EventsURL:\"https://api.github.com/users/xxx/events{/privacy}\", FollowingURL:\"https://api.github.com/users/xxx/following{/other_user}\", FollowersURL:\"https://api.github.com/users/xxx/followers\", GistsURL:\"https://api.github.com/users/xxx/gists{/gist_id}\", OrganizationsURL:\"https://api.github.com/users/xxx/orgs\", ReceivedEventsURL:\"https://api.github.com/users/xxx/received_events\", ReposURL:\"https://api.github.com/users/xxx/repos\", StarredURL:\"https://api.github.com/users/xxx/starred{/owner}{/repo}\", SubscriptionsURL:\"https://api.github.com/users/xxx/subscriptions\"}/0xc00082d380/pulls/4025/merge","json":{}}
{"level":"debug","ts":"2022-04-05T14:19:03.248Z","caller":"server/middleware.go:44","msg":"POST /events – from 172.1.1.1:54102","json":{}}
{"level":"debug","ts":"2022-04-05T14:19:03.248Z","caller":"events/events_controller.go:96","msg":"handling GitHub post","json":{}}
{"level":"debug","ts":"2022-04-05T14:19:03.248Z","caller":"events/events_controller.go:146","msg":"request valid","json":{}}
{"level":"debug","ts":"2022-04-05T14:19:03.248Z","caller":"events/events_controller.go:152","msg":"handling as comment event","json":{}}
{"level":"debug","ts":"2022-04-05T14:19:03.248Z","caller":"logging/simple_logger.go:155","msg":"Ignoring non-command comment: \"Automatically merging because all plans ...\"","json":{}}
{"level":"debug","ts":"2022-04-05T14:19:03.248Z","caller":"server/middleware.go:70","msg":"POST /events – respond HTTP 200","json":{}}
{"level":"error","ts":"2022-04-05T14:19:03.904Z","caller":"events/automerger.go:37","msg":"automerging failed: merging pull request: PUT https://api.github.com/repos/xxx/yyy/pulls/4025/merge: 405 Waiting on code owner review from xxx/platform. []","json":{"repo":"xxx/yyy","pull":"4025"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*AutoMerger).automerge\n\tgithub.com/runatlantis/atlantis/server/events/automerger.go:37\ngithub.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:163\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:259"}
{"level":"debug","ts":"2022-04-05T14:19:03.904Z","caller":"vcs/github_client.go:196","msg":"POST /repos/xxx/yyy/issues/4025/comments","json":{}}

Environment details

Atlantis v0.19.2

Additional Context

I was thoroughly reading the docs to get the answer but no luck. Spotted the error in docs #2170

[edbighead]

when apply_requirements is set to ["approved","mergeable"], the approved check is honored.

[grimm26]

I just noticed this happening and came to check if anyone else was having this issue. Same thing is happening to me, but I am not using policies. I can atlantis apply with no approvals and PR still in not mergable state.

This feels like #1895 all over again

[grimm26]

I wonder if #2053 is causing this? I need to start rolling back versions and seeing when atlantis starts working correctly again.

[grimm26]

I rolled back to 0.19.1 and could still apply a PR with no approvals, not mergable. Thinking that the culprit is #2053, I skipped trying 0.19.0 and rolled back to 0.18.5 and things work again, it doesn't let me apply a non-mergable PR with no approvals.

FYI, I'm using Github Enterprise.

[edbighead]

@grimm26 thanks for confirming!
The revert was merged a while ago but not released yet #2173, theoretically the latest dev image should work

[edbighead]

tested on latest 7199ea4, works as expected