[Bug]: provider-aws-glue v1.5.0 doesn't create the partition index

[sannaroby]

Is there an existing issue for this?

• I have searched the existing issues

Affected Resource(s)

glue.aws.upbound.io/v1beta1- CatalogTable

Resource MRs required to reproduce the bug

apiVersion: glue.aws.upbound.io/v1beta1
kind: CatalogTable
metadata:
  annotations:
    crossplane.io/external-name: "test_table"
  name: test-table
  namespace: test-namespace
spec:
  deletionPolicy: 'Delete'
  forProvider:
    catalogId: "00000000000"
    databaseName: test_db
    region: eu-west-1
    parameters:
      classification: parquet
      compressionType: GZIP
    partitionIndex:
      - indexName: partition_index_1
        keys:
          - year
          - month
          - day
    partitionKeys:
      - name: year
        type: int
      - name: month
        type: int
      - name: day
        type: int
      - name: hour
        type: int
    storageDescriptor:
      - inputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat
        outputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat
        location: s3://test-db/test-table
        columns:
          - name: column1
            type: string
          - name: column2
            type: string
          - name: originaltimestamp
            type: timestamp
        serDeInfo:
          - name: parquetHiveSerDe
            parameters:
              serialization.format: "1"
              parquet.compression: "GZIP"
            serializationLibrary: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe
        storedAsSubDirectories: false
    tableType: EXTERNAL_TABLE

### Steps to Reproduce

Create a new CatalogTable resource (aws glue standard table) with partition-keys and partition-index defined.

### What happened?

We've update the provider-aws-glue from the v0.41.0 to the v1.5.0 version.
The provider correctly creates the glue table with also the partition columns.
However, the provider doesn't create the "partition index": it is in an infinite loop, continuously updating the table, but without creating the “partion index” defined in the MR.


### Relevant Error Output Snippet

```shell
2024-06-03T10:02:46Z DEBUG	provider-aws	Observing the external resource	{"uid": "ddb17b5a-ee94-4c11-9b5a-a84f79dbadcc", "name": "test-table", "gvk": "glue.aws.upbound.io/v1beta1, Kind=CatalogTable"}
2024-06-03T10:02:47Z	DEBUG	provider-aws	Diff detected	{"uid": "ddb17b5a-ee94-4c11-9b5a-a84f79dbadcc", "name": "test-table", "gvk": "glue.aws.upbound.io/v1beta1, Kind=CatalogTable", "instanceDiff": "*terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{\"partition_index.0.index_name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"partition_index_1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"partition_index.0.keys.#\":*terraform.ResourceAttrDiff{Old:\"0\", New:\"3\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"partition_index.0.keys.0\":*terraform.ResourceAttrDiff{Old:\"\", New:\"year\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"partition_index.0.keys.1\":*terraform.ResourceAttrDiff{Old:\"\", New:\"month\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"partition_index.0.keys.2\":*terraform.ResourceAttrDiff{Old:\"\", New:\"day\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, RawConfig:cty.NilVal, RawState:cty.NilVal, RawPlan:cty.NilVal, Meta:map[string]interface {}(nil)}"}
2024-06-03T10:02:47.220387785Z 2024-06-03T10:02:47Z	DEBUG	provider-aws	Successfully requested update of external resource	{"controller": "managed/glue.aws.upbound.io/v1beta1, kind=catalogtable", "request": {"name":"test-table"}, "uid": "ddb17b5a-ee94-4c11-9b5a-a84f79dbadcc", "version": "679787377", "external-name": "test_table", "requeue-after": "2024-06-04T09:04:47Z"}
2024-06-03T10:02:47.220396307Z 2024-06-03T10:02:47Z	DEBUG	provider-aws	Async update starting...	{"trackerUID": "ddb17b5a-ee94-4c11-9b5a-a84f79dbadcc", "resourceName": "test-table", "gvk": "gl
ue.aws.upbound.io/v1beta1, Kind=CatalogTable", "tfID": "000000000000:test_db:test_table"}
2024-06-03T10:02:47.220400237Z 2024-06-03T10:02:47Z	DEBUG	provider-aws	Updating the external resource	{"uid": "ddb17b5a-ee94-4c11-9b5a-a84f79dbadcc", "name": "test-table", "gvk": "glue.aws.upbound.io/v1beta1, Kind=CatalogTable"}
2024-06-03T10:02:47Z	DEBUG	provider-aws	Async update ended.	{"trackerUID": "ddb17b5a-ee94-4c11-9b5a-a84f79dbadcc", "resourceName": "test-table", "gvk": "glue.aws.upbound.io/v1beta1, Kind=CatalogTable", "error": null, "tfID": "000000000000:test_db:test_table"}
2024-06-03T10:02:47Z	DEBUG	provider-aws	Reconcile request has been requeued.	{"gvk": "glue.aws.upbound.io/v1beta1, Kind=CatalogTable", "name": "test-table", "rateLimiterName": "", "when": "0s"}
2024-06-03T10:02:47Z	DEBUG	provider-aws	Reconciling	{"controller": "managed/glue.aws.upbound.io/v1beta1, kind=catalogtable", "request": {"name":"test-table"}}
2024-06-03T10:02:47Z	DEBUG	provider-aws	Connecting to the service provider	{"uid": "ddb17b5a-ee94-4c11-9b5a-a84f79dbadcc", "name": "test-table", "gvk": "glue.aws.upbound.io/v1beta1, Kind=CatalogTable"}
2024-06-03T10:02:47Z	DEBUG	provider-aws	Checking cache entry	{"cacheKey": "8df36442-50b2-44c6-8b67-54b315d60045:1:eu-west-1:IRSA:211686c177c059daa3f131e7e0b487f269d5f4734d485f2b4d256956973f8d12:/var/run/secrets/eks.amazonaws.com/serviceaccount/token:arn:aws:iam::000000000000:role/glue-admin-role", "pc": "aws.upbound.io/v1beta1, Kind=ProviderConfig"}
2024-06-03T10:02:47Z	DEBUG	provider-aws	Cache hit	{"cacheKey": "8df36442-50b2-44c6-8b67-54b315d60045:1:eu-west-1:IRSA:211686c177c059daa3f131e7e0b487f269d5f4734d485f2b4d256956973f8d12:/var/run/secrets/eks.amazonaws.com/serviceaccount/token:arn:aws:iam::000000000000:role/glue-admin-role", "pc": "aws.upbound.io/v1beta1, Kind=ProviderConfig"}
2024-06-03T10:02:47Z	DEBUG	provider-aws	Observing the external resource	{"uid": "ddb17b5a-ee94-4c11-9b5a-a84f79dbadcc", "name": "test-table", "gvk": "glue.aws.upbound.io/v1beta1, Kind=CatalogTable"}
2024-06-03T10:02:48Z	DEBUG	provider-aws	Diff detected	{"uid": "ddb17b5a-ee94-4c11-9b5a-a84f79dbadcc", "name": "test-table", "gvk": "glue.aws.upbound.io/v1beta1, Kind=CatalogTable", "instanceDiff": "*terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{\"partition_index.0.index_name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"partition_index_1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"partition_index.0.keys.#\":*terraform.ResourceAttrDiff{Old:\"0\", New:\"3\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"partition_index.0.keys.0\":*terraform.ResourceAttrDiff{Old:\"\", New:\"year\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"partition_index.0.keys.1\":*terraform.ResourceAttrDiff{Old:\"\", New:\"month\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"partition_index.0.keys.2\":*terraform.ResourceAttrDiff{Old:\"\", New:\"day\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, RawConfig:cty.NilVal, RawState:cty.NilVal, RawPlan:cty.NilVal, Meta:map[string]interface {}(nil)}"}
2024-06-03T10:02:48.041955235Z 2024-06-03T10:02:48Z	DEBUG	provider-aws	Successfully requested update of external resource	{"controller": "managed/glue.aws.upbound.io/v1beta1, kind=catalogtable", "request": {"name":"test-table"}, "uid": "ddb17b5a-ee94-4c11-9b5a-a84f79dbadcc", "version": "679787403", "external-name": "test_table", "requeue-after": "2024-06-04T11:14:10Z"}
2024-06-03T10:02:48.042779553Z 2024-06-03T10:02:48Z	DEBUG	provider-aws	Async update starting...	{"trackerUID": "ddb17b5a-ee94-4c11-9b5a-a84f79dbadcc", "resourceName": "test-table", "gvk": "gl
ue.aws.upbound.io/v1beta1, Kind=CatalogTable", "tfID": "000000000000:test_db:test_table"}
2024-06-03T10:02:48.042827355Z 2024-06-03T10:02:48Z	DEBUG	provider-aws	Updating the external resource	{"uid": "ddb17b5a-ee94-4c11-9b5a-a84f79dbadcc", "name": "test-table", "gvk": "glue.aws.upbound.io/v1beta1, Kind=CatalogTable"}
2024-06-03T10:02:48Z	DEBUG	provider-aws	Async update ended.	{"trackerUID": "ddb17b5a-ee94-4c11-9b5a-a84f79dbadcc", "resourceName": "test-table", "gvk": "glue.aws.upbound.io/v1beta1, Kind=CatalogTable", "error": null, "tfID": "000000000000:test_db:test_table"}
2024-06-03T10:02:48Z	DEBUG	provider-aws	Reconcile request has been requeued.	{"gvk": "glue.aws.upbound.io/v1beta1, Kind=CatalogTable", "name": "test-table", "rateLimiterName": "", "when": "0s"}
2024-06-03T10:02:48Z	DEBUG	provider-aws	Reconciling	{"controller": "managed/glue.aws.upbound.io/v1beta1, kind=catalogtable", "request": {"name":"test-table"}}
2024-06-03T10:02:48Z	DEBUG	provider-aws	Connecting to the service provider	{"uid": "ddb17b5a-ee94-4c11-9b5a-a84f79dbadcc", "name": "test-table", "gvk": "glue.aws.upbound.io/v1beta1, Kind=CatalogTable"}
2024-06-03T10:02:48Z	DEBUG	provider-aws	Checking cache entry	{"cacheKey": "8df36442-50b2-44c6-8b67-54b315d60045:1:eu-west-1:IRSA:211686c177c059daa3f131e7e0b487f269d5f4734d485f2b4d256956973f8d12:/var/run/secrets/eks.amazonaws.com/serviceaccount/token:arn:aws:iam::000000000000:role/glue-admin-role", "pc": "aws.upbound.io/v1beta1, Kind=ProviderConfig"}
2024-06-03T10:02:48Z	DEBUG	provider-aws	Cache hit	{"cacheKey": "8df36442-50b2-44c6-8b67-54b315d60045:1:eu-west-1:IRSA:211686c177c059daa3f131e7e0b487f269d5f4734d485f2b4d256956973f8d12:/var/run/secrets/eks.amazonaws.com/serviceaccount/token:arn:aws:iam::000000000000:role/glue-admin-role", "pc": "aws.upbound.io/v1beta1, Kind=ProviderConfig"}
2024-06-03T10:02:48Z	DEBUG	provider-aws	Observing the external resource	{"uid": "ddb17b5a-ee94-4c11-9b5a-a84f79dbadcc", "name": "test-table", "gvk": "glue.aws.upbound.io/v1beta1, Kind=CatalogTable"}
2024-06-03T10:02:48Z	DEBUG	provider-aws	Diff detected	{"uid": "ddb17b5a-ee94-4c11-9b5a-a84f79dbadcc", "name": "test-table", "gvk": "glue.aws.upbound.io/v1beta1, Kind=CatalogTable", "instanceDiff": "*terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{\"partition_index.0.index_name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"partition_index_1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"partition_index.0.keys.#\":*terraform.ResourceAttrDiff{Old:\"0\", New:\"3\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"partition_index.0.keys.0\":*terraform.ResourceAttrDiff{Old:\"\", New:\"year\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"partition_index.0.keys.1\":*terraform.ResourceAttrDiff{Old:\"\", New:\"month\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"partition_index.0.keys.2\":*terraform.ResourceAttrDiff{Old:\"\", New:\"day\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, RawConfig:cty.NilVal, RawState:cty.NilVal, RawPlan:cty.NilVal, Meta:map[string]interface {}(nil)}"}

Crossplane Version

1.16.0

Provider Version

1.5.0

Kubernetes Version

No response

Kubernetes Distribution

EKS

Additional Info

In the v0.41.0 the resource was created correctly.
I also tried with provider version 1.1.0 and found the same kind of problem.

[turkenf]

Hi @sannaroby,

Thank you for bringing up this. Did you get the same error when you tried to create the resource in provider v1.5.0?

[sannaroby]

Hi @turkenf.
I've the same problem (it is in an infinite loop, continuously updating the table, but without creating the “partion index” defined in the MR) with the versions: 0.46.0, 1.1.0, 1.4.0 and 1.5.0.
Instead, with version 0.41.0 the table is created correctly.

[turkenf]

@sannaroby, thank you for your response, I can reproduce the issue in provider v1.7.0.

[github-actions]

This provider repo does not have enough maintainers to address every issue. Since there has been no activity in the last 90 days it is now marked as stale. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.

[sannaroby]

/fresh