parallel planning causes signal: killed errors

[nitrocode]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

Parallel planning causes signal killed errors

Reproduction Steps

1. Have 8 terraform directories
2. Set parallel_plan: true
3. Run atlantis plan

Logs

Logs

{"level":"error","ts":"2023-02-07T19:06:14.906Z","caller":"events/instrumented_project_command_runner.go:72","msg":"Error running plan operation: running \"/usr/local/bin/terraform plan -input=false -refresh -out \\\"/atlantis/repos/org/repo/54/default/global/cell-use1-az2/default.tfplan\\\"\" in \"/atlantis/repos/org/repo/54/default/global/cell-use1-az2\": signal: killed\n\n1 test, 1 passed, 0 warnings, 0 failures, 0 exceptions\n\nterraform$ATLANTIS_TERRAFORM_VERSION --version\nTerraform v1.3.7\non linux_amd64\n+ provider registry.terraform.io/gavinbunney/kubectl v1.14.0\n+ provider registry.terraform.io/hashicorp/aws v4.53.0\n+ provider registry.terraform.io/hashicorp/cloudinit v2.2.0\n+ provider registry.terraform.io/hashicorp/helm v2.8.0\n+ provider registry.terraform.io/hashicorp/kubernetes v2.17.0\n+ provider registry.terraform.io/hashicorp/null v3.2.1\n+ provider registry.terraform.io/hashicorp/random v3.4.3\n+ provider registry.terraform.io/hashicorp/time v0.9.1\n+ provider registry.terraform.io/hashicorp/tls v4.0.4\n\ntfsec --version\nv1.28.1\n\n\nNo problems detected!\n\n\n\n","json":{"repo":"org/repo", "pull":"54"},"stacktrace":"github.com/runatlantis/atlantis/server/events.RunAndEmitStats\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:72\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandRunner).Plan\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:37\ngithub.com/runatlantis/atlantis/server/events.runProjectCmdsParallel.func1\n\tgithub.com/runatlantis/atlantis/server/events/project_command_pool_executor.go:29"}

github.com/runatlantis/atlantis/server/events.RunAndEmitStats
	github.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:72
github.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandRunner).Plan
	github.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:37
github.com/runatlantis/atlantis/server/events.runProjectCmdsParallel.func1
	github.com/runatlantis/atlantis/server/events/project_command_pool_executor.go:29

atlantis/server/events/instrumented_project_command_runner.go

Lines 70 to 74 in ec54d83

	if result.Error != nil {
	executionError.Inc(1)
	logger.Err("Error running %s operation: %s", commandName, result.Error.Error())
	return result
	}

Environment details

• Atlantis version: v0.22.3
• Deployment method: eks
• Atlantis flags:

yaml config https://github.com/nitrocode/atlantis-terraform-module/blob/d6d2bf74dff3c30b52232f1edd57dca0251be750/repos.yaml#L66

$ env | grep -i atlantis
ATLANTIS_REPO_ALLOWLIST=github.com/org/*
ATLANTIS_WRITE_GIT_CREDS=true
ATLANTIS_PORT=4141
ATLANTIS_DATA_DIR=/atlantis
HOSTNAME=atlantis
ATLANTIS_AUTOMERGE=1
HOME=/home/atlantis
ATLANTIS_ALLOW_COMMANDS=all
ATLANTIS_GH_ALLOW_MERGEABLE_BYPASS_APPLY=true
ATLANTIS_ENABLE_DIFF_MARKDOWN_FORMAT=true
ATLANTIS_CHECKOUT_STRATEGY=merge
ATLANTIS_LOG_LEVEL=debug
ATLANTIS_AUTOPLAN_MODULES=true
ATLANTIS_ENABLE_REGEXP_CMD=true
ATLANTIS_HIDE_PREV_PLAN_COMMENTS=true
ATLANTIS_CONFIG=/home/atlantis/atlantis.yaml

Additional Context

• Possibly related Transient errors during parallel plan from .terraform.lock: plugins are not consistent #2412
• Possibly related feat: Add stats support for basic operations #2147 (released in v0.19.3)

workaround

Tried

• setting parallel_plan: false resolved the issue.

I have not tried

• downgrading to v0.19.2 (before PR 2147)
• commenting out line 72

atlantis/server/events/instrumented_project_command_runner.go

Line 72 in ec54d83

logger.Err("Error running %s operation: %s", commandName, result.Error.Error())

[nitrocode]

Turns out this issue was related to not enough memory and cpu.

[nicholastulach]

Turns out this issue was related to not enough memory and cpu.

How did you solve this? I am seeing this on my atlantis plans now too.