aws-cdk-eks: stack fails and corrupts after upgrade to v2.35 due to EKS logging config change

[juweeks]

Describe the bug

Upgrading to v2.35 (from v2.33) and deploying (just the upgrade- no other code difference) to an existing EKS cluster pushes what appears to be a change in logging configuration to the stack.

Not only does the stack fail to deploy, but the nested stack fails to rollback, staying in UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS state and failing the parent stack rollback. The only way to remediate is to delete the OnEventHandler and IsCompleteHandler resources, thereby corrupting the entire thing.

Expected Behavior

a stack that doesn't blow up

Current Behavior

cdk diff on EKS stack after upgrading to v2.35:

Resources
[~] Custom::AWSCDK-EKS-Cluster eks/cluster/gggg/Resource/Resource eksclustergggg4CAC641D 
 └─ [~] Config
     └─ [~] .logging:
         └─ [~] .clusterLogging:
             └─ @@ -8,5 +8,9 @@
                [ ]       "controllerManager",
                [ ]       "scheduler"
                [ ]     ]
                [+]   },
                [+]   {
                [+]     "enabled": false,
                [+]     "types": []
                [ ]   }
                [ ] ]

This results in a deploy failure in the ClusterResourceProviderNestedStack in the OnEventHandler custom resource:

{
    "errorType": "InvalidParameterException",
    "errorMessage": "No changes needed for the logging config provided",
    "code": "InvalidParameterException",
    "message": "No changes needed for the logging config provided",
    "time": "2022-08-08T23:14:41.167Z",
    "requestId": "29aa9de7-7bd6-4f3e-9db3-54777d84a714",
    "statusCode": 400,
    "retryable": false,
    "retryDelay": 28.276972511621913,
    "stack": [
        "InvalidParameterException: No changes needed for the logging config provided",
        "    at Object.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:52:27)",
        "    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/rest_json.js:49:8)",
        "    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)",
        "    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)",
        "    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)",
        "    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)",
        "    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)",
        "    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10",
        "    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)",
        "    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12)"
    ]
}

Full diff for reference:

Resources
[~] Custom::AWSCDK-EKS-Cluster eks/cluster/gggg/Resource/Resource eksclustergggg4CAC641D 
 └─ [~] Config
     └─ [~] .logging:
         └─ [~] .clusterLogging:
             └─ @@ -8,5 +8,9 @@
                [ ]       "controllerManager",
                [ ]       "scheduler"
                [ ]     ]
                [+]   },
                [+]   {
                [+]     "enabled": false,
                [+]     "types": []
                [ ]   }
                [ ] ]
[~] AWS::CloudFormation::Stack @aws-cdk--aws-eks.ClusterResourceProvider.NestedStack/@aws-cdk--aws-eks.ClusterResourceProvider.NestedStackResource awscdkawseksClusterResourceProviderNestedStackawscdkawseksClusterResourceProviderNestedStackResource9827C454 
 ├─ [~] NestedTemplate
 │   └─ [~] .Resources:
 │       ├─ [~] .CDKMetadata:
 │       │   └─ [~] .Properties:
 │       │       └─ [~] .Analytics:
 │       │           ├─ [-] v2:deflate64:H4sIAAAAAAAA/1VPy04DIRT9lu6ZqykbXbYmujHNpE3cEgrXEYeBhgsqIfy7MK0LV+dxn2cLnMP9Rn7ToPQ8WHOGckCKqE9RqplZuZy1FFZmDMJ5jeIS/E8WckIX4dCMsetdl6+9ibVVohAXkggjwa4DIw77pGaMe0l4Wwrl6d2tM28YyHjHnpNTsZNW+OOVGblAOXqL3V5x9Nao3OWVVYZqC+WEKgUT80vw6dKr/4xa2fpLyzUZNzGVKPpFBCSfgkKCFuTL6Jagn7m5lY05fnh3x+EBHjefZMwQkotmQThe8RfRyLD6QgEAAA==
 │       │           └─ [+] v2:deflate64:H4sIAAAAAAAA/1VPy04DIRT9lu6Zq+nERJetiW5MM2kTt4TCdaTDQMOFVkL4d2FaF67O4z7PGvoneFyJK3VSTZ3RR8g7pIDqEIScmBHzUQluRELPrVPIz979JC5GtAF21Ria3jT50ZpYXcUz9VwQYSDYNGDUwzbKCcNWEN6XQn79ssvMJ3rSzrK3aGVopBb+eGFazJD3zmCzFxyc0TI1eWOFoVxDPqCMXof07l08t+o/oxS2/FJzjdqOTEYKbuYeyUUvkaAGuWhVE7Qzd7ewIYVvZx96eIaX1Ym07ny0Qc8I+xv+AqDKDaBCAQAA
 │       ├─ [~] .IsCompleteHandler7073F4DA:
 │       │   ├─ [~] .Metadata:
 │       │   │   └─ [~] .aws:asset:path:
 │       │   │       ├─ [-] asset.04a62144012b2cd32474867b1533a36eb328db26f983a17ff81189234aff4705
 │       │   │       └─ [+] asset.4ac7c7f6f1e1700aa9ace7e1f38c92b9245129096921bcdc5ff2b1748d57c29d
 │       │   └─ [~] .Properties:
 │       │       └─ [~] .Code:
 │       │           └─ [~] .S3Key:
 │       │               ├─ [-] 04a62144012b2cd32474867b1533a36eb328db26f983a17ff81189234aff4705.zip
 │       │               └─ [+] 4ac7c7f6f1e1700aa9ace7e1f38c92b9245129096921bcdc5ff2b1748d57c29d.zip
 │       └─ [~] .OnEventHandler42BEBAE0:
 │           ├─ [~] .Metadata:
 │           │   └─ [~] .aws:asset:path:
 │           │       ├─ [-] asset.04a62144012b2cd32474867b1533a36eb328db26f983a17ff81189234aff4705
 │           │       └─ [+] asset.4ac7c7f6f1e1700aa9ace7e1f38c92b9245129096921bcdc5ff2b1748d57c29d
 │           └─ [~] .Properties:
 │               └─ [~] .Code:
 │                   └─ [~] .S3Key:
 │                       ├─ [-] 04a62144012b2cd32474867b1533a36eb328db26f983a17ff81189234aff4705.zip
 │                       └─ [+] 4ac7c7f6f1e1700aa9ace7e1f38c92b9245129096921bcdc5ff2b1748d57c29d.zip
 └─ [~] TemplateURL
     └─ [~] .Fn::Join:
         └─ @@ -5,6 +5,6 @@
            [ ]     {
            [ ]       "Ref": "AWS::URLSuffix"
            [ ]     },
            [-]     "/cdk-hnb659fds-assets-ACCOUNT-us-east-1/0786f62ccf83e5ac76dc61ff94d274b13763f161e7a55aaea426b86951b77e81.json"
            [+]     "/cdk-hnb659fds-assets-ACCOUNT-us-east-1/a4c76fa1256e9f6a80e010b5f0639632c344bded01dd736325bda4eb9d5c045a.json"
            [ ]   ]
            [ ] ]
[~] AWS::CloudFormation::Stack @aws-cdk--aws-eks.KubectlProvider.NestedStack/@aws-cdk--aws-eks.KubectlProvider.NestedStackResource awscdkawseksKubectlProviderNestedStackawscdkawseksKubectlProviderNestedStackResourceA7AEBA6B 
 ├─ [~] NestedTemplate
 │   └─ [~] .Resources:
 │       ├─ [~] .AwsCliLayerF44AAF94:
 │       │   ├─ [~] .Metadata:
 │       │   │   └─ [~] .aws:asset:path:
 │       │   │       ├─ [-] asset.ba6af152ce7366ad6f9019e5c6719704a92411a2eb6f3bc1cb39dea393ece19e.zip
 │       │   │       └─ [+] asset.fc2a3ca6356f84129511297b00cd5cef3c7b67abdf955eb5efa75c711da7399f.zip
 │       │   └─ [~] .Properties:
 │       │       └─ [~] .Content:
 │       │           └─ [~] .S3Key:
 │       │               ├─ [-] ba6af152ce7366ad6f9019e5c6719704a92411a2eb6f3bc1cb39dea393ece19e.zip
 │       │               └─ [+] fc2a3ca6356f84129511297b00cd5cef3c7b67abdf955eb5efa75c711da7399f.zip
 │       └─ [~] .CDKMetadata:
 │           └─ [~] .Properties:
 │               └─ [~] .Analytics:
 │                   ├─ [-] v2:deflate64:H4sIAAAAAAAA/02Pu47CMBBFv4XeGRBpoASkbaCIQKK1HGdgZ+PYkscGRZH/nThhEdV9THHPrKEsYbVQTy500xaGahjGIAdsGY6xRh1M5d2DGvTCqK5uFAw/0epAzorDzX77k+rRX9HzmJMg1cFwdgbzadLKGdJ9jrNLgkupmDEw7LKMGfZRtxj2ijGJqbwEdSd7f69Lk0dkhtSGYPfkg6FpWLQz7j/2VCahIwfXSY/sotfI8HknY73bJKo+/Dq7LGED28UfExU+2kAdwnnWFy09WdcqAQAA
 │                   └─ [+] v2:deflate64:H4sIAAAAAAAA/02PTYvCMBCGf4v3dFwsgh5V8LJ7KBW8hjSddcemCWQSpZT8d5vWXfb0fszhfWYD5RY+VurJhW67wlAD4xTkiB3DZ2xQB1N596AWvTCqb1oF4zlaHchZcfq2//2XGtBf0fOUkyDVw1g7g/k0a+UM6SHHxSXBpVTMGBgOWaYMx6g7DEfFmMRcXoK6kb2916XJIzJDakNwePLJ0DwsugX3F3suk9CRg+ulR3bRa2T4eydjvdskqiH8OLsuYQf71Z2JCh9toB6hXvQFupW1TSoBAAA=
 └─ [~] TemplateURL
     └─ [~] .Fn::Join:
         └─ @@ -5,6 +5,6 @@
            [ ]     {
            [ ]       "Ref": "AWS::URLSuffix"
            [ ]     },
            [-]     "/cdk-hnb659fds-assets-ACCOUNT-us-east-1/d52fdb7e93bf3a10795401b23fc961cc2a4e16bd05842beef2b11c3f7b8bc369.json"
            [+]     "/cdk-hnb659fds-assets-ACCOUNT-us-east-1/34d4db1389512a45b874c10e5b3846c106f23f5e42b06dec51e3421cb0b4a692.json"
            [ ]   ]
            [ ] ]

Reproduction Steps

1. existing EKS stack on v2.33
2. cdk diff -> no changes
3. upgrade to v2.35
4. cdk diff -> see changes
5. cdk deploy stack on v2.35
6. 💥

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.35.0 (build 5c23578)

Framework Version

2.35.0

Node.js Version

v16.15.1

OS

mac 12.4

Language

Python

Language Version

3.8.9

Other information

No response

[juweeks]

possible fix from an AWS consultant:

1. Go to root eks stack (not nested one)
2. Stack actions -> continue rollback
3. Advanced options -> ignore eks resource
4. Wait for root and nested stacks to go back to rollback complete stage
5. Roll eks version back to 2.33

[TCBeekley]

I am facing the same error messages when upgrading > 2.34 in Typescript. This includes 2.37.0.

The fix that @juweeks Posted worked for me, but I went back to 2.34.0, and everything ran fine again.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[Hazzard17h]

If stuck with No changes needed for the logging config provided error, manually disable at least one cluster logging type, and deploy the stack again.