Gitlab_client: Apply doesn't work on a branch cut from another until after making a new commit

[jukie]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

This is an unlikely edge case but I hit it nonetheless. I'm PoC'ing Atlantis at work and once I had a branch I was comfortable with I wanted to cut a new branch/PR so the output was easier to read. I could've also just deleted all the prior comments but this was faster. Upon doing so and opening the second PR without including any new changes, apply fails after a successful plan.
I suspect this has something to do with the commit/branch lookup logic in the gitlab client.

Reproduction Steps

1. Checkout repo and default branch. I'll be using "main" in this example.
2. Checkout new branch: git checkout -b coolStuffA
3. Make a terraform change and commit
4. Push branch and open Merge Request for main
5. Run atlantis plan
6. Close Merge Request
7. On local you should still have coolStuffA checked out. From here create a new branch: git checkout -b coolStuffB
8. Push branch coolStuffB without any new changes: git push origin HEAD
9. Open merge request for coolStuffB into main
10. Run atlantis plan
11. Run atlantis apply - this should produce runtime error: invalid memory address or nil pointer dereference

Logs

{"level":"info","ts":"2022-09-01T20:43:03.317Z","caller":"vcs/gitlab_client.go:105","msg":"determined GitLab is running version 15.2.2","json":{}}
{"level":"info","ts":"2022-09-01T20:43:03.469Z","caller":"server/server.go:869","msg":"Atlantis started - listening on port 4141","json":{}}
{"level":"info","ts":"2022-09-01T20:43:03.469Z","caller":"scheduled/executor_service.go:46","msg":"Scheduled Executor Service started","json":{}}
{"level":"info","ts":"2022-09-01T20:43:55.247Z","caller":"events/events_controller.go:533","msg":"parsed comment as command=\"plan\" verbose=false dir=\"\" workspace=\"\" project=\"\" flags=\"\"","json":{}}
{"level":"info","ts":"2022-09-01T20:43:55.579Z","caller":"events/project_command_builder.go:307","msg":"found no atlantis.yaml file","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:43:55.579Z","caller":"events/project_finder.go:57","msg":"filtered modified files to 4 .tf or terragrunt.hcl files: [aws/example-account/us-west-2/ebs-demo/ebs.tf aws/example-account/us-west-2/ebs-demo/locals.tf aws/example-account/us-west-2/ebs-demo/main.tf aws/example-account/aws-global-commons.tf]","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:43:55.580Z","caller":"events/project_finder.go:78","msg":"there are 2 modified project(s) at path(s): aws/example-account/us-west-2/ebs-demo, aws/example-account","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:43:55.580Z","caller":"events/project_command_builder.go:312","msg":"automatically determined that there were 2 projects modified in this pull request: [repofullname=jukie/atlantis-demo path=aws/example-account/us-west-2/ebs-demo repofullname=jukie/atlantis-demo path=aws/example-account]","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:43:55.584Z","caller":"events/project_command_context_builder.go:294","msg":"cannot determine which version to use from terraform configuration, detected 0 possibilities.","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:43:55.584Z","caller":"events/project_command_context_builder.go:294","msg":"cannot determine which version to use from terraform configuration, detected 0 possibilities.","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:43:55.704Z","caller":"events/project_locker.go:80","msg":"acquired lock with id \"jukie/atlantis-demo/aws/example-account/us-west-2/ebs-demo/default\"","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:43:59.915Z","caller":"models/shell_command_runner.go:156","msg":"successfully ran \"/usr/local/bin/terraform init -input=false -upgrade\" in \"/atlantis-data/repos/jukie/atlantis-demo/6/default/aws/example-account/us-west-2/ebs-demo\"","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:43:59.988Z","caller":"terraform/terraform_client.go:314","msg":"successfully ran \"/usr/local/bin/terraform workspace show\" in \"/atlantis-data/repos/jukie/atlantis-demo/6/default/aws/example-account/us-west-2/ebs-demo\"","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:44:08.463Z","caller":"models/shell_command_runner.go:156","msg":"successfully ran \"/usr/local/bin/terraform plan -input=false -refresh -out \\\"/atlantis-data/repos/jukie/atlantis-demo/6/default/aws/example-account/us-west-2/ebs-demo/default.tfplan\\\"\" in \"/atlantis-data/repos/jukie/atlantis-demo/6/default/aws/example-account/us-west-2/ebs-demo\"","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:44:08.566Z","caller":"events/instrumented_project_command_runner.go:53","msg":"plan success. output available at: https://gitlab.redacted.org/jukie/atlantis-demo/-/merge_requests/6","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:44:08.749Z","caller":"events/project_locker.go:80","msg":"acquired lock with id \"jukie/atlantis-demo/aws/example-account/default\"","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:44:08.778Z","caller":"models/shell_command_runner.go:156","msg":"successfully ran \"/usr/local/bin/terraform init -input=false -upgrade\" in \"/atlantis-data/repos/jukie/atlantis-demo/6/default/aws/example-account\"","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:44:08.879Z","caller":"terraform/terraform_client.go:314","msg":"successfully ran \"/usr/local/bin/terraform workspace show\" in \"/atlantis-data/repos/jukie/atlantis-demo/6/default/aws/example-account\"","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:44:08.977Z","caller":"models/shell_command_runner.go:156","msg":"successfully ran \"/usr/local/bin/terraform plan -input=false -refresh -out \\\"/atlantis-data/repos/jukie/atlantis-demo/6/default/aws/example-account/default.tfplan\\\"\" in \"/atlantis-data/repos/jukie/atlantis-demo/6/default/aws/example-account\"","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:44:09.088Z","caller":"events/instrumented_project_command_runner.go:53","msg":"plan success. output available at: https://gitlab.redacted.org/jukie/atlantis-demo/-/merge_requests/6","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:44:09.395Z","caller":"events/plan_command_runner.go:215","msg":"Running policy check for command=\"plan\" verbose=false dir=\"\" workspace=\"\" project=\"\" flags=\"\"","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:44:09.395Z","caller":"events/policy_check_command_runner.go:39","msg":"no projects to run policy_check in","json":{"repo":"jukie/atlantis-demo","pull":"6"}}
{"level":"info","ts":"2022-09-01T20:44:36.781Z","caller":"events/events_controller.go:533","msg":"parsed comment as command=\"apply\" verbose=false dir=\"\" workspace=\"\" project=\"\" flags=\"\"","json":{}}
{"level":"warn","ts":"2022-09-01T20:44:36.948Z","caller":"events/apply_command_runner.go:101","msg":"unable to update commit status: POST https://gitlab.redacted.org/api/v4/projects/jukie/atlantis-demo/statuses/ae8fdf627b54a1fba8b916e01e01504c6f0e6865: 400 {message: Cannot transition status via :run from :running (Reason(s): Status cannot transition via \"run\")}","json":{"repo":"jukie/atlantis-demo","pull":"6"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:101\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:296"}
{"level":"error","ts":"2022-09-01T20:44:37.189Z","caller":"events/command_runner.go:427","msg":"PANIC: runtime error: invalid memory address or nil pointer dereference\nruntime/panic.go:221 (0x44a075)\nruntime/signal_unix.go:735 (0x44a045)\ngithub.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:209 (0xa5371a)\ngithub.com/runatlantis/atlantis/server/events/vcs/proxy.go:72 (0xa58824)\ngithub.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28 (0xa590c4)\ngithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:109 (0xc33dd5)\ngithub.com/runatlantis/atlantis/server/events/command_runner.go:296 (0xc38703)\nruntime/asm_amd64.s:1581 (0x4651a0)\n","json":{"repo":"jukie/atlantis-demo","pull":"6"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).logPanics\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:427\nruntime.gopanic\n\truntime/panic.go:1047\nruntime.panicmem\n\truntime/panic.go:221\nruntime.sigpanic\n\truntime/signal_unix.go:735\ngithub.com/runatlantis/atlantis/server/events/vcs.(*GitlabClient).PullIsMergeable\n\tgithub.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:209\ngithub.com/runatlantis/atlantis/server/events/vcs.(*ClientProxy).PullIsMergeable\n\tgithub.com/runatlantis/atlantis/server/events/vcs/proxy.go:72\ngithub.com/runatlantis/atlantis/server/events/vcs.(*pullReqStatusFetcher).FetchPullStatus\n\tgithub.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28\ngithub.com/runatlantis/atlantis/server/events.(*ApplyCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/apply_command_runner.go:109\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunCommentCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:296"}

Stacktrace

runtime error: invalid memory address or nil pointer dereference
runtime/panic.go:221 (0x44a146)
runtime/signal_unix.go:735 (0x44a116)
github.com/runatlantis/atlantis/server/events/vcs/gitlab_client.go:209 (0xa655ba)
github.com/runatlantis/atlantis/server/events/vcs/proxy.go:72 (0xa6a5a4)
github.com/runatlantis/atlantis/server/events/vcs/pull_status_fetcher.go:28 (0xa6ada4)
github.com/runatlantis/atlantis/server/events/apply_command_runner.go:109 (0xc32f15)
github.com/runatlantis/atlantis/server/events/command_runner.go:296 (0xc37843)
runtime/asm_amd64.s:1581 (0x465140)

Environment details

Self hosted gitlab using helm chart in EKS

Additional Context

[jukie]

Issue is in mr.HeadPipeline.SHA at:

atlantis/server/events/vcs/gitlab_client.go

Line 209 in 71233ef

statuses, _, err := g.Client.Commits.GetCommitStatuses(mr.ProjectID, mr.HeadPipeline.SHA, nil)

[jukie]

@jamengual now I'm wondering if this is a valid issue for Atlantis or if I should open another one at https://github.com/xanzy/go-gitlab?
https://pkg.go.dev/github.com/xanzy/[email protected]?utm_source=gopls#MergeRequest.HeadPipeline

[jamengual]

we can leave this open and you can file in go-gitlab too and then report back here.

[jukie]

After digging into it a bit more the issue is here and how Atlantis is using the client. In my example mr.HeadPipeline is nil so that just needs to be accounted for.
Since no new commits have been pushed to branch-B, that MR doesn't have any directly associated pipelines. However, traversing to the pipelines from MR/branch-A will show a reference to MR/branch-b.