TestJobQueueUpgrade panics under PlanResourceChange

[t0yv0]

The crux of the issue:

  "method": "/pulumirpc.ResourceProvider/Diff",
  "request": {
    "id": "terraform-20231205221555324200000003",
    "urn": "urn:pulumi:test::job-queue::aws:batch/computeEnvironment:ComputeEnvironment::sampleComputeEnvironment",
    "olds": {
      "arn": "arn:aws:batch:us-west-2:616138583583:compute-environment/terraform-20231205221555324200000003",
      "computeResources": {
        "ec2Configuration": {
          "imageIdOverride": "",
          "imageType": "ECS_AL2"
        },

Note that there is a problem with aws:batch/computeEnvironment:ComputeEnvironment resource where the 5.42.0 provider
wrote an object for ec2Configuration when a list is expected by the new version, as is also reflected in the Pulumi
schema.

So when upgrading a stack with ComputeEnvironment from 5.42.0 to the latest provider there is an implicit type mismatch.

There appears to be a problem where PlanResourceChange makes it a louder failure than the current silent failure.

Both PlanResourceChange=true and PlanResourceChange=false run state upgraders (although the current and desired version
is 0), and ironically the resource does not define any state upgraders at all.

Note that the state cty.Value passed to the state upgraders differs:

PlanResourceChange=true

panic: UPGRADING GRPC RAW=cty.TupleVal([]cty.Value{
   cty.ObjectVal(map[string]cty.Value{"allocation_strategy":cty.StringVal(""), "bid_percentage":cty.NumberIntVal(0), "desired_vcpus":cty.NumberIntVal(0),
      "ec2_configuration":cty.ObjectVal(map[string]cty.Value{"image_id_override":cty.StringVal("[]"), "image_type":cty.StringVal("ECS_AL2")}), "ec2_key_pair":cty.StringVal(""), "image_id":cty.StringVal(""), "instance_role":cty.StringVal("arn:aws:iam::616138583583:instance-profile/ecsInstanceRoleInstanceProfile-912858a"), "instance_type":cty.TupleVal([]cty.Value{cty.StringVal("c4.large")}), "max_vcpus":cty.NumberIntVal(16), "min_vcpus":cty.NumberIntVal(0), "security_group_ids":cty.TupleVal([]cty.Value{cty.StringVal("sg-0bb51dcade107992e")}), "spot_iam_fleet_role":cty.StringVal(""), "subnets":cty.TupleVal([]cty.Value{cty.StringVal("subnet-07f3a862bd9c5f1a3")}), "tags":cty.EmptyObjectVal, "type":cty.StringVal("EC2")})}) [recovered]

PlanResourceChange=false

panic: UPGRADING GRPC RAW=cty.ListVal([]cty.Value{
   cty.ObjectVal(map[string]cty.Value{"allocation_strategy":cty.StringVal(""), "bid_percentage":cty.NumberIntVal(0), "desired_vcpus":cty.NumberIntVal(0),
       "ec2_configuration":cty.ListValEmpty(cty.Object(map[string]cty.Type{"image_id_override":cty.String, "image_type":cty.String})), "ec2_key_pair":cty.StringVal(""), "image_id":cty.StringVal(""), "instance_role":cty.StringVal("arn:aws:iam::616138583583:instance-profile/ecsInstanceRoleInstanceProfile-912858a"), "instance_type":cty.SetVal([]cty.Value{cty.StringVal("c4.large")}), "launch_template":cty.ListValEmpty(cty.Object(map[string]cty.Type{"launch_template_id":cty.String, "launch_template_name":cty.String, "version":cty.String})), "max_vcpus":cty.NumberIntVal(16), "min_vcpus":cty.NumberIntVal(0), "placement_group":cty.StringVal(""), "security_group_ids":cty.SetVal([]cty.Value{cty.StringVal("sg-0bb51dcade107992e")}), "spot_iam_fleet_role":cty.StringVal(""), "subnets":cty.SetVal([]cty.Value{cty.StringVal("subnet-07f3a862bd9c5f1a3")}), "tags":cty.MapValEmpty(cty.String), "type":cty.StringVal("EC2")})})

Note that PlanResourceChange=false elegantly bypassed the problem by simply ignoring the data it can't read in
ec2_configuration.

However for PlanResoruceChange=true the TupleVal is hiding an underlying failure of recoverCtyValue:

recoverCtyValue fails:
panic: FAILED: recoverCtyValue failed: Cannot reconcile map map[image_id_override:[] image_type:ECS_AL2] to cty.List(cty.Object(map[string]cty.Type{"image_id_override":cty.String, "image_type":cty.String})) [recovered]

Which makes sense, it failed to read the object into a slot of type list. Then it decided to use approximate methods to
recover the state, which yielded the TupleVal combined with an ec2_configuration that is now an ObjectVal which is not
expected.

Threading this data through resource upgraders trips up JSONMapToStateValue which responds with a very unhelpful error
message: "missing expected [".

There is another issue where the bridge fails to handle the error diagnostics from the state upgraders, and therefore
panics instead of showing this error.

	github.com/pulumi/pulumi-terraform-bridge/v3 v3.83.1-0.20240603141117-4503342dc4c8

[t0yv0]

@VenelinMartinov @iwahbe

[t0yv0]

I think this is an implicit schema change for aws:batch/computeEnvironment:ComputeEnvironment ec2Configuration at Pulumi provider level. This schema change was not present in TF proper. Perhaps there is a bridge mechanism to compensate and normalize the state for this resource?

[iwahbe]

You can override TransformFromState to fix the issue for just this resource.

[t0yv0]

That's right. I'll have a look at that indeed. For the bridge, submitting a quick fix to handle diagnostics in upgrade state.

We can also consider soft-reporting errors from state upgrades, either always, or when desired_version=current_version.

[t0yv0]

https://github.com/pulumi/pulumi-terraform-bridge/pull/2053/files

[VenelinMartinov]

I wonder why the new type checking does not catch this?

[t0yv0]

It's not intended to check state. It would be wrong to do so in fact since the state may have been written at a different provider version. This is like pulumi/pulumi-terraform-bridge#1667 but with an additional plot twist that TF schema did not change (1667 does not apply) but Pulumi schema did.