Atlantis auto merges MR although getting apply error

[dorian-tsorfy]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

After pushing changes to my MR, I tried to run atlantis apply -d <state> while the Autoplan is running, and I got these errors from Atlantis -

Plan Error

deleting dir '/atlantis-data/repos/terraform/general/8279/default' before cloning: unlinkat /atlantis-data/repos/terraform/general/8279/default/.git: directory not empty

In the following comment -

Apply Error

`dir "states/dorian-atlantis-test" does not exist`

And eventually -

Automatically merging because all plans have been successfully applied.

If the Auto plan was still running when I ran the apply, I would expect Atlantis to fail the apply command on the reason there is another Atlantis command running. Moreover, even if the apply command took place and got error, I would expect Atlantis not to merge the MR.

I saw some other issues regarding this behavior (#2785 for reference), however I opened another one because I think it's somehow different.

Something important to say - Our Terraform state repository is big with around ~500 states so even shallow cloning takes time to run.

Reproduction Steps

• Upload new MR with a state change. Wait for the plan to complete
• Upload a simple change to the state so the auto-plan will occur again. Right after commiting the changes and pushing to the branch, run in the MR - atlantis apply -d <your_state>

Logs

Logs

atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:51:45.214Z","caller":"vcs/gitlab_client.go:120","msg":"Getting modified files for GitLab merge request 8281","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:51:45.381Z","caller":"vcs/gitlab_client.go:141","msg":"GET projects/terraform%2Fgeneral/merge_requests/8281/changes returned: 200","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:51:45.381Z","caller":"events/project_command_builder.go:338","msg":"1 files were modified in this pull request. Modified files: [states/dorian-atlantis-test/bucket.tf]","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:51:45.381Z","caller":"events/project_command_builder.go:396","msg":"got workspace lock","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:51:45.381Z","caller":"events/working_dir.go:106","msg":"clone directory '/atlantis-data/repos/terraform/general/8281/default' already exists, checking if it's at the right commit","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"warn","ts":"2024-09-09T11:51:45.390Z","caller":"events/working_dir.go:121","msg":"will re-clone repo, could not determine if was at correct commit: git rev-parse HEAD: exit status 128: fatal: not a git repository (or any parent up to mount point /)\nStopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).\n","json":{"repo":"terraform/general","pull":"8281"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*FileWorkspace).Clone\n\tgithub.com/runatlantis/atlantis/server/events/working_dir.go:121\ngithub.com/runatlantis/atlantis/server/events.(*DefaultProjectCommandBuilder).buildAllCommandsByCfg\n\tgithub.com/runatlantis/atlantis/server/events/project_command_builder.go:399\ngithub.com/runatlantis/atlantis/server/events.(*DefaultProjectCommandBuilder).BuildAutoplanCommands\n\tgithub.com/runatlantis/atlantis/server/events/project_command_builder.go:256\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandBuilder).BuildAutoplanCommands.func1\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_builder.go:29\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandBuilder).buildAndEmitStats\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_builder.go:71\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandBuilder).BuildAutoplanCommands\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_builder.go:26\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).runAutoplan\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:86\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:305\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunAutoplanCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:223"}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:03.581Z","caller":"server/middleware.go:45","msg":"GET /healthz – from 10.10.138.157:53058","json":{}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:03.581Z","caller":"server/middleware.go:72","msg":"GET /healthz – respond HTTP 200","json":{}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:03.581Z","caller":"server/middleware.go:45","msg":"GET /healthz – from 10.10.138.157:53056","json":{}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:03.581Z","caller":"server/middleware.go:72","msg":"GET /healthz – respond HTTP 200","json":{}}
atlantis-production-2 atlantis {"level":"info","ts":"2024-09-09T11:52:44.020Z","caller":"events/working_dir.go:235","msg":"creating dir '/atlantis-data/repos/terraform/general/8281/default'","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"error","ts":"2024-09-09T11:52:44.820Z","caller":"events/instrumented_project_command_builder.go:75","msg":"Error building auto plan commands: running git clone --depth=1 --branch test-automerge-bug-2 --single-branch https://tbot:<redacted>@gitlab.io/terraform/general.git /atlantis-data/repos/terraform/general/8281/default: Cloning into '/atlantis-data/repos/terraform/general/8281/default'...\nwarning: Could not find remote branch test-automerge-bug-2 to clone.\nfatal: Remote branch test-automerge-bug-2 not found in upstream origin\n: exit status 128","json":{},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandBuilder).buildAndEmitStats\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_builder.go:75\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandBuilder).BuildAutoplanCommands\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_builder.go:26\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).runAutoplan\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:86\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:305\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunAutoplanCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:223"}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:44.820Z","caller":"vcs/gitlab_client.go:398","msg":"Updating GitLab commit status for 'atlantis/plan' to 'failed'","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:44.820Z","caller":"vcs/gitlab_client.go:504","msg":"Getting GitLab merge request 8281","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:44.983Z","caller":"vcs/gitlab_client.go:507","msg":"GET /projects/terraform/general/merge_requests/8281 returned: 200","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:44.983Z","caller":"vcs/gitlab_client.go:426","msg":"Head pipeline found for merge request 8281, source 'push'. refTarget 'test-automerge-bug-2'","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:45.074Z","caller":"vcs/gitlab_client.go:468","msg":"POST /projects/terraform/general/statuses/e02ad5ce4fd5d6eee5d8cc0ce6eb306d5ef12ca6 returned: 201","json":{"repo":"terraform/general","pull":"8281","attempt":1,"max_attempts":10,"repo":"terraform/general","commit":"e02ad5ce4fd5d6eee5d8cc0ce6eb306d5ef12ca6","state":"failed"}}
atlantis-production-2 atlantis {"level":"error","ts":"2024-09-09T11:52:45.074Z","caller":"events/pull_updater.go:18","msg":"running git clone --depth=1 --branch test-automerge-bug-2 --single-branch https://tbot:<redacted>@gitlab.io/terraform/general.git /atlantis-data/repos/terraform/general/8281/default: Cloning into '/atlantis-data/repos/terraform/general/8281/default'...\nwarning: Could not find remote branch test-automerge-bug-2 to clone.\nfatal: Remote branch test-automerge-bug-2 not found in upstream origin\n: exit status 128","json":{"repo":"terraform/general","pull":"8281"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*PullUpdater).updatePull\n\tgithub.com/runatlantis/atlantis/server/events/pull_updater.go:18\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).runAutoplan\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:91\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).Run\n\tgithub.com/runatlantis/atlantis/server/events/plan_command_runner.go:305\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunAutoplanCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:223"}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:45.074Z","caller":"vcs/gitlab_client.go:175","msg":"Creating comment on GitLab merge request 8281","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:45.203Z","caller":"vcs/gitlab_client.go:184","msg":"POST /projects/terraform/general/merge_requests/8281/notes returned: 201","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:45.203Z","caller":"events/post_workflow_hooks_command_runner.go:53","msg":"post-hooks configured, running...","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:45.203Z","caller":"events/post_workflow_hooks_command_runner.go:59","msg":"got workspace lock","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"debug","ts":"2024-09-09T11:52:45.206Z","caller":"events/working_dir.go:106","msg":"clone directory '/atlantis-data/repos/terraform/general/8281/default' already exists, checking if it's at the right commit","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"warn","ts":"2024-09-09T11:52:45.209Z","caller":"events/working_dir.go:121","msg":"will re-clone repo, could not determine if was at correct commit: git rev-parse HEAD: exit status 128: fatal: not a git repository (or any parent up to mount point /)\nStopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).\n","json":{"repo":"terraform/general","pull":"8281"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*FileWorkspace).Clone\n\tgithub.com/runatlantis/atlantis/server/events/working_dir.go:121\ngithub.com/runatlantis/atlantis/server/events.(*DefaultPostWorkflowHooksCommandRunner).RunPostHooks\n\tgithub.com/runatlantis/atlantis/server/events/post_workflow_hooks_command_runner.go:62\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunAutoplanCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:225"}
atlantis-production-2 atlantis {"level":"info","ts":"2024-09-09T11:52:45.212Z","caller":"events/working_dir.go:235","msg":"creating dir '/atlantis-data/repos/terraform/general/8281/default'","json":{"repo":"terraform/general","pull":"8281"}}
atlantis-production-2 atlantis {"level":"error","ts":"2024-09-09T11:52:46.010Z","caller":"events/command_runner.go:228","msg":"Error running post-workflow hooks running git clone --depth=1 --branch test-automerge-bug-2 --single-branch https://tbot:<redacted>@gitlab.io/terraform/general.git /atlantis-data/repos/terraform/general/8281/default: Cloning into '/atlantis-data/repos/terraform/general/8281/default'...\nwarning: Could not find remote branch test-automerge-bug-2 to clone.\nfatal: Remote branch test-automerge-bug-2 not found in upstream origin\n: exit status 128.","json":{"repo":"terraform/general","pull":"8281"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunAutoplanCommand\n\tgithub.com/runatlantis/atlantis/server/events/command_runner.go:228"}

Environment details

• Atlantis version: atlantis v0.28.5 (commit: 92d10ec) (build date: 2024-07-17T17:07:30.148Z)
• Deployment method: helm