Google Cloud: Update instances on instance group when instance_template is changed

[premist]

On resource google_compute_instance_group_manager, you can specify instance_template resource for the instance group. When I modify google_compute_instance_template resource, associated google_compute_instance_group_manager is updated but its instances are not updated. I had to manually delete VM instances so Instance Group can recreate instances with new instance template. I can think two possible approach to this:

1. Recreate instances after updating instance-groups

By using gcloud compute instance-groups managed recreate-instances after updating an instance group, you can trigger recreation of the instances. This is probably the easiest way to solve a problem, but instance group will have no active instances for a while, which is not ideal on production environment.

2. Use rolling update

By using gcloud alpha compute rolling-updates start, you can gracefully retire old instances and create new instances. It has several flags that can be used to control the instance count and interval between each batch updates. We've used this command before we investigate Terraform, and it worked pretty well even though it's still in alpha state.

[lwander]

Good catch on the update issue. I will implement approach 1 since we'd rather not support the Alpha API as it's subject to change.

[lwander]

As you pointed out, it's bad in prod to restart all the instances at once. So I added the update_strategy field where you can specify NONE and then rely on running the alpha command by hand until that makes its way into the stable v1 API, where Terraform will give you the option to do that automatically.

[premist]

Thank you @lwander, it sounds like a good idea! Since we're testing Terraform on our new staging environment, I'll try out your PR locally to see if it works well.

[sparkprime]

If you make the IGM depend on the template, then changing the template should recreate the IGM (and all its instances). If this doesn't happen this is a core bug.

Of course that's not the behavior you want :)

What I would actually do is create an IGM for each version of your software -- then you can independently scale them / tear them down and implement whatever phased rollout policy you want.

[lwander]

No, instance_template isn't marked as ForceNew, so the IGM shouldn't restart on a change to the template.

[sparkprime]

Let's navel-gaze... In an ideal would one could imagine a declarative deployment agent being able to do 2 things:

1. You update the template, it shuts everything down and restarts it. This is appropriate for local development, staging environment, automated integration tests, load tests, etc.

2. You update the template, you get a rolling update of your fleet that ensures that sufficient instances are always running at any given time and automatically rolls back on failures that it detects (e.g. a spike in HTTP errors, latency spike, not meeting SLO, etc.) Only one update is allowed at a time (new updates fail) but there is an explicit rollback command. This is appropriate for managing a highly available production service.

This IGM rollout stuff gets you some but not all of the way to 2. You actually need application knowledge to do it properly, and I've only seen it done in PAAS providers. What IGM currently supports is definitely useful though, although unfortunately it is exposed imperatively so requires some mapping to expose naturally in Terraform.

However you can, with a bit of work, build (2) on top of (1) by executing multiple Terraform applies from a continuous deployment agent that is aware of your application's monitoring. However you'd need a ForceNew instance_template for that, which would invalidate the current support for (2).

Maybe the right thing to do is like the startup_script -- have one forcenew field and one updateable field.

[lwander]

For anyone reading this in the future:

@sparkprime and I talked this out and agreed to keep update_strategy for the following reasons:

1. A policy of "RESTART" is the same as modifying instance_template designated as ForceNew, but with the added benefit that it does not recreate the entire IGM. This allows a user to implement @sparkprime's rollout approach detailed above.
2. A policy of "NONE" is the same as modifying instance_template designated as not ForceNew.
3. Future policies will all be supported, all without creating extra fields for the same attribute.

[lwander]

Closing since #3892 has been merged

[premist]

Awesome, thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.