panic, segmentation violation, and crash with Terraform 0.7.8

[brikis98]

Terraform Version

Terraform v0.7.8

Affected Resource(s)

I can't tell what causes the crash. The Terraform code I'm running uses the following resources:

• aws_instance
• aws_iam_role
• aws_iam_instance_profile
• aws_security_group
• aws_security_group_rule
• aws_eip
• aws_route53_record
• aws_iam_policy
• aws_iam_policy_document
• aws_iam_policy_attachment
• terraform_remote_state
• template_file

Terraform Configuration Files

It's a fairly large amount of code and I can't tell which part is causing the problem. The last log line suggests it had just finished creating the EIP for the EC2 Instance, so perhaps that's the problem. Here are the snippets for those two resources:

resource "aws_instance" "instance" {
  ami = "${var.ami}"
  instance_type = "${var.instance_type}"
  iam_instance_profile = "${aws_iam_instance_profile.instance.name}"
  key_name = "${var.keypair_name}"
  vpc_security_group_ids = ["${aws_security_group.instance.id}"]
  subnet_id = "${var.subnet_id}"
  user_data = "${var.user_data}"
  tags {
    Name = "${var.name}"
  }
}

resource "aws_eip" "instance" {
  count = "${signum(var.attach_eip)}"

  instance = "${aws_instance.instance.id}"
  vpc = true

  # https://github.com/hashicorp/terraform/issues/1815
  provisioner "local-exec" {
    command = "echo 'Sleeping 15 seconds to work around EIP propagation bug in Terraform' && sleep 15"
  }
}

Perhaps it's something to do with using count or local-exec in the aws_eip resource?

Panic Output

The full crash log is here: https://gist.github.com/brikis98/634699f731539a04e1ff7ec0e8b54d0b

module.bastion.aws_eip.instance: Provisioning with 'local-exec'...
module.bastion.aws_eip.instance (local-exec): Executing: /bin/sh -c "echo 'Sleeping 15 seconds to work around EIP propagation bug in Terraform' && sleep 15"
module.bastion.aws_eip.instance (local-exec): Sleeping 15 seconds to work around EIP propagation bug in Terraform
module.bastion.aws_eip.instance: Still creating... (10s elapsed)
module.bastion.aws_eip.instance: Creation complete
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xf1f24]

goroutine 1609 [running]:
panic(0x25e3540, 0xc42000e090)
	/opt/go/src/runtime/panic.go:500 +0x1a1
github.com/hashicorp/terraform/terraform.(*EvalApplyProvisioners).apply(0xc420505170, 0x431cc20, 0xc4208b2a90, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/terraform/terraform/eval_apply.go:278 +0x5b4
github.com/hashicorp/terraform/terraform.(*EvalApplyProvisioners).Eval(0xc420505170, 0x431cc20, 0xc4208b2a90, 0x2, 0x2, 0xc420261600, 0xc)
	/opt/gopath/src/github.com/hashicorp/terraform/terraform/eval_apply.go:179 +0x151
github.com/hashicorp/terraform/terraform.EvalRaw(0x4302480, 0xc420505170, 0x431cc20, 0xc4208b2a90, 0x29882a0, 0x0, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/terraform/terraform/eval.go:53 +0x17f
github.com/hashicorp/terraform/terraform.(*EvalSequence).Eval(0xc42025d6e0, 0x431cc20, 0xc4208b2a90, 0x2, 0x2, 0xc420279130, 0xc)
	/opt/gopath/src/github.com/hashicorp/terraform/terraform/eval_sequence.go:10 +0x6c
github.com/hashicorp/terraform/terraform.EvalRaw(0x4302d80, 0xc42025d6e0, 0x431cc20, 0xc4208b2a90, 0x4302d80, 0xc42025d6e0, 0xc420505260, 0x24)
	/opt/gopath/src/github.com/hashicorp/terraform/terraform/eval.go:53 +0x17f
github.com/hashicorp/terraform/terraform.Eval(0x4302d80, 0xc42025d6e0, 0x431cc20, 0xc4208b2a90, 0xc42025d6e0, 0x4302d80, 0xc42025d6e0, 0xc420965b30)
	/opt/gopath/src/github.com/hashicorp/terraform/terraform/eval.go:34 +0x53
github.com/hashicorp/terraform/terraform.(*Graph).walk.func1(0x2a939e0, 0xc4203b8128, 0x0, 0x0)
	/opt/gopath/src/github.com/hashicorp/terraform/terraform/graph.go:249 +0xa0f
github.com/hashicorp/terraform/dag.(*AcyclicGraph).Walk.func3(0xc4205cf240, 0xc4207be280, 0xc4205cf250, 0xc4207d0de0, 0xc4205cf260, 0x2a939e0, 0xc4203b8128, 0xc42078bda0, 0xc42089cd20)
	/opt/gopath/src/github.com/hashicorp/terraform/dag/dag.go:238 +0x251
created by github.com/hashicorp/terraform/dag.(*AcyclicGraph).Walk
	/opt/gopath/src/github.com/hashicorp/terraform/dag/dag.go:247 +0x618

Expected Behavior

terraform apply should have succeeded.

Actual Behavior

It crashed.

Steps to Reproduce

Run terraform apply

Important Factoids

This code worked fine before upgrading to Terraform 0.7.8

[brikis98]

I just downgraded to Terraform 0.7.7 and the same code worked fine.

[w1lnx]

Same for us as well. terraform plan on 0.7.8 with a non-trivial config works fine. terraform apply on the same plan segfaults.

Works as expected on 0.7.5. Haven't had a chance with .6 or .7 yet.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0xf1f24]

goroutine 8019 [running]:
panic(0x25e3540, 0xc4200120a0)
    /opt/go/src/runtime/panic.go:500 +0x1a1
github.com/hashicorp/terraform/terraform.(*EvalApplyProvisioners).apply(0xc4215d9230, 0x431cc20, 0xc4200725b0, 0x0, 0x0)
    /opt/gopath/src/github.com/hashicorp/terraform/terraform/eval_apply.go:278 +0x5b4
github.com/hashicorp/terraform/terraform.(*EvalApplyProvisioners).Eval(0xc4215d9230, 0x431cc20, 0xc4200725b0, 0x2, 0x2, 0xc420ee5d90, 0x9)
    /opt/gopath/src/github.com/hashicorp/terraform/terraform/eval_apply.go:179 +0x151
github.com/hashicorp/terraform/terraform.EvalRaw(0x4302480, 0xc4215d9230, 0x431cc20, 0xc4200725b0, 0x29882a0, 0x0, 0x0, 0x0)
    /opt/gopath/src/github.com/hashicorp/terraform/terraform/eval.go:53 +0x17f
github.com/hashicorp/terraform/terraform.(*EvalSequence).Eval(0xc4206ae5e0, 0x431cc20, 0xc4200725b0, 0x2, 0x2, 0xc420ee5a20, 0x9)
    /opt/gopath/src/github.com/hashicorp/terraform/terraform/eval_sequence.go:10 +0x6c
github.com/hashicorp/terraform/terraform.EvalRaw(0x4302d80, 0xc4206ae5e0, 0x431cc20, 0xc4200725b0, 0x4302d80, 0xc4206ae5e0, 0xc4215d9320, 0x26)
    /opt/gopath/src/github.com/hashicorp/terraform/terraform/eval.go:53 +0x17f
github.com/hashicorp/terraform/terraform.Eval(0x4302d80, 0xc4206ae5e0, 0x431cc20, 0xc4200725b0, 0xc4206ae5e0, 0x4302d80, 0xc4206ae5e0, 0xc41ff54e44)
    /opt/gopath/src/github.com/hashicorp/terraform/terraform/eval.go:34 +0x53
github.com/hashicorp/terraform/terraform.(*Graph).walk.func1(0x2a939e0, 0xc4204ba260, 0x0, 0x0)
    /opt/gopath/src/github.com/hashicorp/terraform/terraform/graph.go:249 +0xa0f
github.com/hashicorp/terraform/dag.(*AcyclicGraph).Walk.func3(0xc4215768b0, 0xc4209a2840, 0xc4215768c0, 0xc421431890, 0xc4215768d0, 0x2a939e0, 0xc4204ba260, 0xc421392960, 0xc42132c420)
    /opt/gopath/src/github.com/hashicorp/terraform/dag/dag.go:238 +0x251
created by github.com/hashicorp/terraform/dag.(*AcyclicGraph).Walk
    /opt/gopath/src/github.com/hashicorp/terraform/dag/dag.go:247 +0x618

It's a 14M crash log if anyone's interested in seeing the whole thing.

[mathieuherbert]

It's the same for me.
I also downgraded to Terraform 0.7.7 and the code apply well

[mitchellh]

Hi folks, I'm picking this up now. If anyone can get a reproduction case that'd be a great help. I'm going to try with the config above, too and will update anyone here.

[mitchellh]

Got a minimal repro:

main.tf

module "child" {
    source = "./child"
}

child/main.tf

resource "null_resource" "foo" {
  provisioner "local-exec" {
    command = "echo foo"
  }
}

Shocked we don't have unit test coverage of this.

This was caused by new experimental features being tested in the background of Terraform that are actually crashing. Your actual infrastructure was unaffected and actually likely came up properly (your state is probably complete and fine!), but obviously the crash is not what we want to see. I'm going to look at this bug specifically but also protecting against crashes.

[mitchellh]

Fix is in the referenced PR. We'll rush out a 0.7.9 for this this week.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.