aws:ec2:LaunchTemplate update_default_version Fails on AWS Provider

[farvour]

I upgraded the AWS provider from 3.23 to the latest 4.5.1 recently to do a Pulumi 3.x upgrade. Noticed some code that sets update_default_version on this provider fails now when doing a pulumi update on the existing resources.

Expected behavior

No changes should have been seen by the provider and updateDefaultVersion should continue to function as desired.

Current behavior

  aws:ec2:LaunchTemplate (ci-general):
    error: 1 error occurred:
    	* updating urn:pulumi:ci::aws-eks-py::aws:ec2/launchTemplate:LaunchTemplate::ci-general: 1 error occurred:
    	* error modifying Launch Template (lt-<REDACTED>): InvalidParameterValue: Invalid launch template version: either '$Default', '$Latest', or a numeric version are allowed.

also, it doesn't even show up in the list of diffSet, so something tells me the provider is misbehaving internally here.

     pulumi:pulumi:Stack        aws-eks-py-ci   **failed**              1 error
 ~   ├─ aws:ec2:LaunchTemplate  ci-kiam-server  **updating failed**     [diff: -tagsAll~tags]; 1 error
 ~   └─ aws:ec2:LaunchTemplate  ci-general      **updating failed**     [diff: -tagsAll~tags]; 1 error

Steps to reproduce

We are using the Python dialect. Create a pulumi_aws.ec2.LaunchTemplate resource and set the update_default_version attribute to True.

Context (Environment)

Existing resource was present and may or may not affect the outcome of the result. This was a LaunchTemplate whose versioning updating was working just fine prior to upgrading the pulumi-aws provider to 4.5.1 and pulumi to 3.2.1.

This seems like a really old Terraform issue to be honest, that somehow resurfaced itself (regression?) in the newer AWS provider. hashicorp/terraform-provider-aws#4655

Affected feature

N/A

[farvour]

The workaround right now is to just pull the update_default_version field so it's False. Not ideal, but our AutoScaleGroups use $$Latest so it doesn't break anything, just makes the resource inconsistent in the UI for other users who want to examine the current default version of a LaunchTemplate.

[Sodki]

@farvour is this still an issue to you? I'm using update_default_version=True with the latest Pulumi and it's working.

[stack72]

Going to close this out - there's been no response here so it seems all is working?

[farvour]

So,... this feels like it came back again:

      ~ pulumi:providers:aws: (update)
        [id=c0481110-ffc5-47c0-94da-ded7f28d552f]
        [urn=urn:pulumi:orgname.us-gov-west-1.networking::aws-orgname-networking-general::pulumi:providers:aws::us-gov-networking]
      ~ version                  : "5.32.0" => "5.33.0"

    aws:ec2:LaunchTemplate (ec2-ubuntu-22.04-general):
    error: 1 error occurred:
    	* updating urn:pulumi:orgname.us-gov-west-1.networking::aws-orgname-networking-general::aws:ec2/launchTemplate:LaunchTemplate::ec2-ubuntu-22.04-general: 1 error occurred:
    	* updating EC2 Launch Template (lt-0b97625fbfdb72e01): InvalidParameterValue: Invalid launch template version: either '$Default', '$Latest', or a numeric version are allowed.
    	status code: 400, request id: 6c9e1be4-acad-4da9-a591-af92db353476```

[lacaSG]

I'm getting this when I'm trying to add a tag to an otherwise unchanged launch template. If I update a property of the template, it works fine. using update_default_version=True

[murbot]

I'm getting this when I'm trying to add a tag to an otherwise unchanged launch template. If I update a property of the template, it works fine. using update_default_version=True

Just ran into the exact same scenario.

[lukehoban]

I believe this is the same issue tracked in hashicorp/terraform-provider-aws#34867 upstream. Will reopen here as tracking that issue being fixed and adopted into this provider.

[t0yv0]

I can reproduce on latest Pulumi with the following:

import pulumi
import pulumi_aws as aws

config = pulumi.Config()
key_name = config.require("key_name")
tag_name = config.require("tag_name")

ec2_launch_template = aws.ec2.LaunchTemplate(
    "my_launch_template",
    image_id="ami-0c55b159cbfafe1f0",
    instance_type="t2.micro",
    key_name=key_name,
    tags={
        "Name": tag_name
    },
    update_default_version=True)

# Export the ID of the launch template
pulumi.export("launch_template_id", ec2_launch_template.id)
pulumi.export("launch_template_latest_version", ec2_launch_template.latest_version)

#!/usr/bin/env bash

set -euo pipefail

pulumi destroy --yes

# Initial configuration
pulumi config set tag_name tag1
pulumi config set key_name key1
pulumi up --skip-preview --yes

# Change only tag
pulumi config set tag_name tag2
pulumi config set key_name key1
pulumi preview --diff
PULUMI_DEBUG_GRPC="$PWD/repro.json" pulumi up --skip-preview --yes   # version not changing

Observing that the issue happens for programs that specify update_default_version=True during changes that only affect
tags and nothing else. If I do not specify update_default_version=True, or if I change only the key_name, then the
program behaves as expected:

• latest_version is incremented for every change to key_name
• latest_version is not incremented for every change to tag_name (but only works with update_default_version=True)

Debugging the failure leads to this line:

https://github.com/hashicorp/terraform-provider-aws/blob/main/internal/service/ec2/ec2_launch_template.go#L1132

ModifyLaunchTemplateWithContext fails because DefaultVersion incorrectly interpreted a Computed latestVersion as 0.

(dlv) print d.diff.Attributes["latest_version"]
*github.com/hashicorp/terraform-plugin-sdk/v2/terraform.ResourceAttrDiff {
        Old: "1",
        New: "0",
        NewComputed: true,
        NewRemoved: false,
        NewExtra: interface {} nil,
        RequiresNew: false,
        Sensitive: false,
        Type: 0,}

I suspect that the diff customizer is something that sets it to computed, verifying this.
https://github.com/hashicorp/terraform-provider-aws/blob/main/internal/service/ec2/ec2_launch_template.go#L963

[t0yv0]

Curiously, this particular repro does not reproduce once WithPlanResourceChange is applied to the aws_launch_template
resource. This is because this flag removes the impedance mismatch that existed prior, where Terraform does not run the
diff customizers again on Update the second time, it only runs them once as part of preparing the PlanResourceChange
plan. This could be our fix here, verifying.

If this theory is correct though, this repro would not reproduce on the similar program written in TF, and the fix would
not be fixing any manifestations of the issue present in the upstream provider. If something reproduces in TF we could
instead workaround by patching to amend the failing code to pay attention to Computed values and treat them specially.

[t0yv0]

Curiously, this particular repro does not reproduce once WithPlanResourceChange is applied to the aws_launch_template
resource. This is because this flag removes the impedance mismatch that existed prior, where Terraform does not run the
diff customizers again on Update the second time, it only runs them once as part of preparing the PlanResourceChange
plan. This could be our fix here, verifying.

If this theory is correct though, this repro would not reproduce on the similar program written in TF, and the fix would
not be fixing any manifestations of the issue present in the upstream provider. If something reproduces in TF we could
instead workaround by patching to amend the failing code to pay attention to Computed values and treat them specially.

[t0yv0]

I have confirmed that this issue does not currently reproduce in Terraform. To double check I also tried tags under
tags_specifications, but it all seems to work as expected. I read the upstream issue that still open in TF but it was a
little unclear on the repro steps.

Suggest moving forward here with this change that fixes the problem with updating tags:

#3687

The root cause is that CustomizeDiff methods should not be run on Update, but are, cause this to fail under Pulumi.

Tracking broader PlanResourceChange rollout in pulumi/pulumi-terraform-bridge#1785

[t0yv0]

This is now releasing and automation should close this and make a new release. Please let us know if this fix works, and reopen if there's any further issues in this area. Appreciate your patience!