builder/amazon-ebs: RequestLimitExceeded error during instance shutdown

[erickt]

Good afternoon,

We're using a combination of packer 0.10.0, spinnaker and Datadog on AWS, which is generating a large number of AWS requests. We are hitting similar error messages as the bugs #50 and #709 because Packer doesn't appear to implement the Amazon recommended exponential backoff:

...
==> amazon-ebs: Stopping the source instance...
==> amazon-ebs: Waiting for the instance to stop...
==> amazon-ebs: Error waiting for instance to stop: RequestLimitExceeded: Request limit exceeded.
==> amazon-ebs:     status code: 503, request id:
==> amazon-ebs: Terminating the source AWS instance...
==> amazon-ebs: No AMIs to cleanup
==> amazon-ebs: Deleting temporary security group...
==> amazon-ebs: Deleting temporary keypair...
Build 'amazon-ebs' errored: Error waiting for instance to stop: RequestLimitExceeded: Request limit exceeded.
    status code: 503, request id: 
...

I saw that terraform did implement exponential backoff in hashicorp/terraform#1787. Could something like this be replicated in packer?

Thanks!

[mwhooker]

Hmm, the aws sdk we use already implements retrying when the request is throttled. However, it looks like we set the value to 11 https://github.com/mitchellh/packer/blob/master/builder/amazon/common/access_config.go#L37

If you feel comfortable, you can try removing WithMaxRetries and seeing if you still get the issue.

Removing it will let the service set its own default max retries.

If removing it doesn't help, you can try setting it to a big number like 100.

Feel free to send a PR with whatever ends up working out the best

[erickt]

Thanks for the reference. We will try that out and let you know.

[mwhooker]

Should be fixed in #4083 by setting the polling delay to a higher number.