vSphere: Terraform apply command hangs for 5 minutes

[Shruti29]

When I run terraform apply on my vSphere setup, it hangs for 5 minutes before actually starting the resource creation.
However terraform plan returns output immediately.

Unfortunately, I don't see any debug logs for this as well.
It hangs for 5 minutes and the resource creation logs are collected as it continues with the resource creation.

Need some help on this. I found the issue #2059 quite similar to what I am facing.

[rbriancollins]

Have you looked at the vCenter during this time? At what point during the 5 minutes does vCenter show activity related to the new VM creation (assuming you are creating a VM)?

[jen20]

Hi @Shruti29! We added continued progress reports to Terraform 0.6.15 which should indicate where this problem lies. If you run with that version, does it report that the resource is "still creating" during the five minute pause?

[Shruti29]

Sorry for the delay in reply.

@rbriancollins It is at the very end of 5 minutes that it shows VM creation activity on the vCenter.

@jen20
I spoke with a colleague of mine who suggested me that it waits for the network resource and IPs. Ideally, this shouldn't be a problem but since I don't have DNS set up in my dev environment I had to comment out this portion of code to get it spawning the resource without the wait time.

if _, ok := d.GetOk("network_interface.0.ipv4_address"); !ok {
        if v, ok := d.GetOk("boot_delay"); ok {
            stateConf := &resource.StateChangeConf{
                Pending:    []string{"pending"},
                Target:     []string{"active"},
                Refresh:    waitForNetworkingActive(client, vm.datacenter, vm.Path()),
                Timeout:    600 * time.Second,
                Delay:      time.Duration(v.(int)) * time.Second,
                MinTimeout: 2 * time.Second,
            }

            _, err := stateConf.WaitForState()
            if err != nil {
                return err
            }
        }
    }

After commenting this out, the resource spawns immediately.

I believe this is not required if one has a working DNS setup.

[jen20]

Hi @Shruti29! Great work narrowing down the cause of this. Perhaps @markpeek, @frapposelli or @chrislovecnm have an idea of how to mitigate the effects of this?

[chrislovecnm]

Thanks @jen20 -

@benlangfeld was looking at the same problem on Friday as he has reported this bug as well: #6174

This is a duplicate, as I think we have three open.

I as mentioned a couple of the API comiters as well, on that issue, since we are unable to find an example of how exactly we would do the Wait properly. I can open an issue with govmomi, but I don't have time till tomorrow. If someone else can get to it first that would be awesome.

Thanks for your help @Shruti29

[chrislovecnm]

There is a pull request in that I hope will help vmware/govmomi#501

@dougm thoughts?

[chrislovecnm]

@dougm I assume that we could replace the vm.WaitForIP method with WaitForNetIP?

[dougm]

@chrislovecnm yes, you can replace WaitForIP with WaitForNetIP

[chrislovecnm]

Can anyone on here test this PR within this code base?

[chrislovecnm]

@dkalleg do you have a moment to see if the PR helps? This is big problem that we have right now.

[dkalleg]

I haven't seen any 5 minutes delay problem where no activity is seen in vSphere.. I have seen a possible bug that I made a note to look into at some point around creating a vm from a template on some network. In that scenario i was seeing the vm coming up but the network would always show up in a "Not connected" state, where I had to go manually hit the Connect check box in the UI before it would try getting an IP on that network. At that point I believe it would be spinning in the block that @Shruti29 pointed out, so thats something to look into.

Shruti, can we see the terraform .tf file your using? Also can you double check whether you're seeing any action in the vSphere UI during this 5 minute gap (make sure your refreshing the window)? If your truely getting stuck in the code block you listed earlier, then the VM should already exist and you should see that reflected in the vSphere UI. You can also look at the webconsole by clicking on the little image of the guest OS ui; in my template case I would see the ubuntu console telling me something like 'waiting for a network connection to try to acquire an IP via DHCP... sleeping 120 seconds.'

[chrislovecnm]

@Shruti29 tag you're it 😄

@dkalleg much thanks.

[chrislovecnm]

@dkalleg issue: #6174 has another example TF file.@benlangfeld can you pipe in at all? @dkalleg is getting very familiar with the code, and needs more information.

[benlangfeld]

I have not experienced this issue. I have also not had an opportunity to test any further changes yet; I will do ASAP.

[chrislovecnm]

Anyone that can help testing #6479 would get a free cookie ;) We are trying to get up to date API before we tweak network stuff.

If you have questions please ask!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.