Concurrent executions cannot be set above 20 due to API throttling on StartPipelineExecution which causes Invoke Customizations SFN to fail

[andyscott1986]

AFT Version:

1.12.2

Terraform Version & Provider Versions

terraform version

1.7.2

terraform providers

hashicorp/aws v5.64.0

Bug Description
When maximum_concurrent_customizations is set to above 20 the invoke customizations SFN fails on the Execute Pipelines stage due to API Throttling on the StartPipelineExecution API.

To Reproduce
Steps to reproduce the behaviour:

1. Set the variable "maximum_concurrent_customizations" to anything over 20.
2. Trigger Invoke customizations SFN.
3. SFN fails at execute pipeline stage due to API throttling on the StartPipeline API.

Expected behavior
Be able to set max concurrent executions above 20 but with in other service limits and quotas - 40 or 50

Related Logs
{
"Error": "ClientError",
"Cause": "{"errorMessage": "An error occurred (ThrottlingException) when calling the StartPipelineExecution operation (reached max retries: 4): Rate exceeded", "errorType": "ClientError", "requestId": "1c26ec87-accb-4d79-a5e4-d15399050e74", "stackTrace": [" File \"/var/task/aft_customizations_execute_pipeline.py\", line 38, in lambda_handler\n execute_pipeline(session, str(account_id))\n", " File \"/opt/python/lib/python3.11/site-packages/aft_common/codepipeline.py\", line 78, in execute_pipeline\n response = client.start_pipeline_execution(name=name)\n", " File \"/opt/python/lib/python3.11/site-packages/botocore/client.py\", line 534, in _api_call\n return self._make_api_call(operation_name, kwargs)\n", " File \"/opt/python/lib/python3.11/site-packages/botocore/client.py\", line 976, in _make_api_call\n raise error_class(parsed_response, operation_name)\n"]}"
}

Additional context
Large AWS organization with over 600 accounts in AFT takes over 6 hours to deploy updates to all accounts through the different environments in wave deployments.

AWS Support:

I would like to inform that for API “StartPipelineExecution” an account is allowed to make one StartPipelineExecution API in every 10 seconds whereas 20 is the burst rate. Meaning, when the value for burst is collected to count 20 then only the account can make 20 calls next second. But for the burst value to be 20, with the rate of 1 per every 10 seconds, it takes 200 seconds to collect 20 value in burst. But if we are frequently making the request, the burst will never reach to 20. Hence, at any point account wont be able to make more than 20 call per second.

Hence as a solution, if we need more than 50 calls to be made then lambda should have a logic to invoke 20 calls and then stop for next 200 seconds so the burst value of 20 be available again and then make another 20 calls next second.